Hello,
I set up a Deadline farm using Unreal and multiple computers. We got the whole thing running. However, in the automation process, we’re hitting quite a big knot : since the Worker loses communication with the UE instance on the worker, when it’s compiling shaders for the first time, it will fail the job after 3 to 4 minutes.
E.G. :
=======================================================
Error
=======================================================
Error: A connection was not established with an unreal process
at Deadline.Plugins.PluginWrapper.RenderTasks(Task task, String& outMessage, AbortLevel& abortLevel)
=======================================================
Type
=======================================================
RenderPluginException
[…]
2024-03-04 13:38:19: 0: STDOUT: [2024.03.04-18.38.17:818][ 0]LogDeviceProfileManager: [000006D5B9A83000][000006D5B9AD2800 66] HoloLens,
2024-03-04 13:38:19: 0: STDOUT: [2024.03.04-18.38.17:818][ 0]LogDeviceProfileManager: [000006D5B9A83200][000006D5B9AD5000 66] MagicLeap_Vulkan,
2024-03-04 13:38:19: 0: STDOUT: [2024.03.04-18.38.17:818][ 0]LogDeviceProfileManager: Active device profile: [000006D5BAEB3000][000006D59987A000 66] WindowsEditor
2024-03-04 13:38:19: 0: STDOUT: [2024.03.04-18.38.17:818][ 0]LogCsvProfiler: Display: Metadata set : deviceprofile="WindowsEditor"
2024-03-04 13:38:19: 0: STDOUT: [2024.03.04-18.38.18:642][ 0]LogMaterial: WorldGridMaterial-SM6-Default: [...]
2024-03-04 13:38:19: 0: STDOUT: [2024.03.04-18.38.18:738][ 0]LogMaterial: Display: Missing cached shadermap for WorldGridMaterial in PCD3D_SM6, Default, SM6, Editor (DDC key hash: 4b332486d55e0aed4da33dbefa856c375c18ce9d), compiling. Is special engine material.
2024-03-04 13:38:19: 0: STDOUT: [2024.03.04-18.38.19:002][ 0]LogShaderCompilers: Display: FShaderCompileThreadRunnable::WriteNewTasks() took too long (0.801 seconds, threshold is 0.300 s), will parallelize next time.
2024-03-04 13:38:22: 0: INFO: Waiting on client connection..
2024-03-04 13:38:22: 0: STDOUT: [2024.03.04-18.38.20:540][ 0]LogMaterial: Display: Missing cached shadermap for DefaultDeferredDecalMaterial in PCD3D_SM6, Default, SM6, Editor (DDC key hash: a3cf6dfb7a549ee59341b64c111867ccfd3c6ec9), compiling.
2024-03-04 13:38:22: 0: STDOUT: [2024.03.04-18.38.20:936][ 0]LogMaterial: Display: Missing cached shadermap for DefaultLightFunctionMaterial in PCD3D_SM6, Default, SM6, Editor (DDC key hash: 6f0036e32c97984c3d22c471669ae4a4c054dcd1), compiling.
2024-03-04 13:38:25: 0: INFO: Waiting on client connection..
[...] Same line repeating for 2 minutes [...]
2024-03-04 13:40:18: 0: STDOUT: [2024.03.04-18.40.16:747][ 0]LogDerivedDataCache: /AppData/Local/UnrealEngine/Common/DerivedDataCache: Maintenance finished in +00:00:04.760 and deleted 0 files with total size 0 MiB and 0 empty folders. Scanned 2227 files in 3206 folders with total size 129 MiB.
2024-03-04 13:40:22: 0: INFO: Waiting on client connection..
2024-03-04 13:40:24: 0: INFO: Waiting on client connection..
2024-03-04 13:40:30: 0: INFO: Waiting on client connection..
2024-03-04 13:40:33: 0: INFO: Waiting on client connection..
2024-03-04 13:40:37: 0: INFO: Waiting on client connection..
2024-03-04 13:40:39: 0: INFO: Waiting on client connection..
2024-03-04 13:40:43: 0: INFO: Waiting on client connection..
2024-03-04 13:40:48: 0: INFO: Waiting on client connection..
2024-03-04 13:40:50: 0: INFO: Waiting on client connection..
2024-03-04 13:40:54: 0: INFO: Waiting on client connection..
2024-03-04 13:40:58: 0: INFO: Waiting on client connection..
2024-03-04 13:41:00: 0: INFO: Waiting on client connection..
2024-03-04 13:41:05: 0: INFO: Waiting on client connection..
2024-03-04 13:41:08: 0: INFO: Waiting on client connection..
2024-03-04 13:41:11: 0: Done executing plugin command of type 'Render Task'
=======================================================
Details
=======================================================
Date: 03/04/2024 13:41:17
Frames: 0
Elapsed Time: 00:00:03:20
Job Submit Date: 03/04/2024 13:20:51
Job User: axelh
Average RAM Usage: 9214022656 (27%)
Peak RAM Usage: 9979756544 (30%)
Average CPU Usage: 81%
Peak CPU Usage: 100%
Used CPU Clocks (x10^6 cycles): 4507441
Total CPU Clocks (x10^6 cycles): 5564741
=======================================================
Worker Information
=======================================================
Worker Name: MF-PC-151
Version: v10.3.0.9 Release (fd97c242e)
Operating System: Windows 10 Professionnel
Running As Service: No
So far we tried changing the deadline preset in UE to have 0 or 6000 seconds in the timeout section, auto timeout on and off, going on Monitor to change the max timeout to 2 hours, etc.
Workaround so far is to let the project compile the shaders on the workers, before launching the job. But, in the case of automation, it’s really not ideal and would mean we have to do that on every new project.
Any ideas on what could be done to resolve this situation?
Thanks!