Render Job Begins Fine and Then Slowly Every Jobs Fails

Deadline
1

I’m working on setting up both a local farm for quick jobs and a farm on AWS for larger projects on tighter Deadlines. Currently, I’m demoing between two Windows Workers nodes, submitting a 3dsMax file that with the Redshift renderer to a Repository on a failure beefy Dell R410 CentOS server.

It begins working just fine, each node trades off and does a frame, up until about 8 frames in, and then one node starts crashing quickly (sub 1-minute) and the other node has what I think is actually a worse failure mode, it just hangs in the same render bucket and will gladly sit doing nothing for 2 hours unless you requeue the frame.

All I get from the output error it below, what’s the issue? Is it on the Redshift side of things or on Deadlines end? Thanks.

2020/02/28 09:10:55 ERR: [02928] [15888] An unexpected exception has occurred in the network renderer and it is terminating.
2020/02/28 09:10:55 ERR: [02928] [18488] 09:10:55 [ERROR]    ====================================================================================================== ASSERT FAILED  File GPUComputing_CUDA.cpp Line 4020  StreamSynchronize() failed (CUDA_ERROR_ILLEGAL_ADDRESS). This is possibly due to a GPU crash (device 0). Please re-render this scene with the 'Debug Capture' option enabled (in the Redshift 'System' tab) and, once you get the crash again, send the developers the log file html and bin files located in C:\ProgramData\Redshift\Log/Log.Latest.0. Thanks! ====================================================================================================== 
)
   at Deadline.Plugins.PluginWrapper.RenderTasks(String taskId, Int32 startFrame, Int32 endFrame, String& outMessage, AbortLevel& abortLevel)

=======================================================
Type
=======================================================
RenderPluginException

=======================================================
Stack Trace
=======================================================
   at Deadline.Plugins.SandboxedPlugin.d(DeadlineMessage bay)
   at Deadline.Plugins.SandboxedPlugin.RenderTask(String taskId, Int32 startFrame, Int32 endFrame)
   at Deadline.Slaves.SlaveRenderThread.c(TaskLogWriter adv)

Apparently I can’t paste in the error and can’t upload a TXT file, so it’s attached as a zip.

DeadlineRenderTaskErrorNotes.txt.zip (6.9 KB)