Cinema S26 Crashing with Deadline

gavin_grant · September 9, 2022, 8:12am

Good morning all

I would like to report an on going issue that i am experiencing with C4D S26 and Octane 2021.1.5-R4 and Deadline 10.1.23.6 - At the end of saving a frame, at any point during the job ( seems very random!) I get a c4d crash with unhandled exception error.

The C4D file has has a set of alembics and lighting and a few lights, nothing heavy at all and uses around 3.5GB of vram

Here is the error log i receive

=======================================================
Error
=======================================================
Error: FailRenderException : RenderTask: Unexpected exception (Monitored managed process "Cinema4DProcess" has exited or been terminated.)
   at Deadline.Plugins.DeadlinePlugin.FailRender(String message) (Python.Runtime.PythonException)
  File "C:\ProgramData\Thinkbox\Deadline10\workers\DHS-22-Big-Bertha\plugins\631aef90c5c71d882cd0e64f\Cinema4DBatch.py", line 67, in RenderTasks
    self.MyCinema4DController.RenderTasks()
  File "C:\ProgramData\Thinkbox\Deadline10\workers\DHS-22-Big-Bertha\plugins\631aef90c5c71d882cd0e64f\Cinema4DBatch.py", line 790, in RenderTasks
    self.Plugin.LogInfo( self.PollUntilComplete( False ) )
  File "C:\ProgramData\Thinkbox\Deadline10\workers\DHS-22-Big-Bertha\plugins\631aef90c5c71d882cd0e64f\Cinema4DBatch.py", line 959, in PollUntilComplete
    self.Plugin.FailRender( "RenderTask: Unexpected exception (%s)" % e.Message )
   at Python.Runtime.Dispatcher.Dispatch(ArrayList args)
   at __FranticX_GenericDelegate0Dispatcher.Invoke()
   at Deadline.Plugins.DeadlinePlugin.RenderTasks()
   at Deadline.Plugins.DeadlinePlugin.DoRenderTasks()
   at Deadline.Plugins.PluginWrapper.RenderTasks(Task task, String& outMessage, AbortLevel& abortLevel)
   at Deadline.Plugins.PluginWrapper.RenderTasks(Task task, String& outMessage, AbortLevel& abortLevel)

=======================================================
Type
=======================================================
RenderPluginException

=======================================================
Stack Trace
=======================================================
   at Deadline.Plugins.SandboxedPlugin.d(DeadlineMessage bgj, CancellationToken bgk)
   at Deadline.Plugins.SandboxedPlugin.RenderTask(Task task, CancellationToken cancellationToken)
   at Deadline.Slaves.SlaveRenderThread.c(TaskLogWriter ajq, CancellationToken ajr)

=======================================================
Log
=======================================================
2022-09-09 08:53:48:  0: Start Job timeout is disabled.
2022-09-09 08:53:48:  0: Task timeout is disabled.
2022-09-09 08:53:48:  0: Plugin rendering frame(s): 16-20
2022-09-09 08:53:48:  0: Executing plugin command of type 'Render Task'
2022-09-09 08:53:48:  0: INFO: Render Tasks called
2022-09-09 08:53:48:  0: INFO: Pre Build Script
2022-09-09 08:53:48:  0: INFO: Validating the path: 'F:\Projects\Monuments_CharacterStudies\Poseidon\Test_renders\220808_posidon_test_renders\Pose_c_stop_looking_at\'
2022-09-09 08:53:48:  0: INFO: Validating the path: 'C:\ProgramData\Thinkbox\Deadline10\workers\DHS-22-Big-Bertha\jobsData\631aef90c5c71d882cd0e64f\c4dOutput_tempc4AWA0'
2022-09-09 08:53:48:  0: INFO: Rendering main output to local drive, will copy files and folders to final location after render is complete
2022-09-09 08:53:48:  0: INFO: Validating the path: 'F:\Projects\Monuments_CharacterStudies\Atlas\C4D\Atlas_v2\'
2022-09-09 08:53:48:  0: INFO: Validating the path: 'C:\ProgramData\Thinkbox\Deadline10\workers\DHS-22-Big-Bertha\jobsData\631aef90c5c71d882cd0e64f\c4dOutputMP_tempxmpLI0'
2022-09-09 08:53:48:  0: INFO: Rendering multipass output to local drive, will copy files and folders to final location after render is complete
2022-09-09 08:53:48:  0: INFO: 
2022-09-09 08:53:48:  0: STDOUT: Running Script: C:\ProgramData\Thinkbox\Deadline10\workers\DHS-22-Big-Bertha\jobsData\631aef90c5c71d882cd0e64f\thread0_temp8OJ4q0\c4d_Batch_Script.py
2022-09-09 08:53:48:  0: STDOUT: There is no file name specified for the rendering image.
2022-09-09 08:53:48:  0: STDOUT: Do you want to continue without saving?
2022-09-09 08:53:48:  0: STDOUT: straightAlpha:0
2022-09-09 08:53:48:  0: STDOUT: *** pBufType:1 pColorSpace:1
2022-09-09 08:53:48:  0: STDOUT: Rendering frame 16 at <Fri Sep  9 08:53:48 2022>
2022-09-09 08:53:48:  0: STDOUT:  
2022-09-09 08:53:48:  0: STDOUT: FRAME:16 fps:30
2022-09-09 08:53:49:  0: STDOUT: NetRender enabled. Slave count is 0.
2022-09-09 08:53:49:  0: STDOUT: --------------------------- EXPORT LOG ---------------------------
2022-09-09 08:54:14:  0: STDOUT: Export materials time= 25921.223 ms
2022-09-09 08:54:14:  0: STDOUT: Collect objects time= 1.664 ms
2022-09-09 08:54:15:  0: STDOUT: Mesh creation time = 114.417 ms.
2022-09-09 08:54:21:  0: STDOUT:  MB:0/0  ST/MOV:0/42  Nodes:162  Tris:230k  DispTris:0  Hairs:0  Meshes:47
2022-09-09 08:54:21:  0: STDOUT:  Textures Grey8/16:0/0  Rgb32/64:10/1
2022-09-09 08:54:21:  0: STDOUT: VRAM used/free/max:3.189Gb/5.394Gb/11Gb  Out-of-core used:0Kb  RAM used:17.005Gb total:127.615Gb  OpenGL free/total:0/0
2022-09-09 08:54:43:  0: STDOUT: mblurTM=0sec.  createTM=0.043sec.  updateTM=0.537sec. renderTM:22.062sec.   totalTM:54.142sec. 
2022-09-09 08:54:43:  0: STDOUT: 0-PassID:0 Pass:Beauty
2022-09-09 08:54:43:  0: STDOUT: 1-PassID:43 Pass:Denoised beauty
2022-09-09 08:54:43:  0: STDOUT: 2-PassID:7 Pass:Reflection
2022-09-09 08:54:43:  0: STDOUT: 3-PassID:15 Pass:Subsurface scattering
2022-09-09 08:54:43:  0: STDOUT: 4-PassID:32 Pass:Shadow
2022-09-09 08:54:43:  0: STDOUT: 5-PassID:23 Pass:Light ID 1
2022-09-09 08:54:43:  0: STDOUT: 6-PassID:24 Pass:Light ID 2
2022-09-09 08:54:43:  0: STDOUT: 7-PassID:25 Pass:Light ID 3
2022-09-09 08:54:43:  0: STDOUT: 8-PassID:26 Pass:Light ID 4
2022-09-09 08:54:43:  0: STDOUT: 9-PassID:27 Pass:Light ID 5
2022-09-09 08:54:43:  0: STDOUT: 10-PassID:1010 Pass:Ambient occlusion
2022-09-09 08:54:43:  0: STDOUT: 11-PassID:2001 Pass:Crypto material node name
2022-09-09 08:54:43:  0: STDOUT: Tonemapping the all passes tm:421.944
2022-09-09 08:54:43:  0: STDOUT: Displaying passes in tm=88.009
2022-09-09 08:54:48:  0: STDOUT: Passes saved in:4.923sec. 
2022-09-09 08:54:51:  0: STDOUT: Error: application crashed
2022-09-09 08:54:51:  0: STDOUT: C4DUnhandledExceptionFilter: writing exception info
2022-09-09 08:54:51:  0: STDOUT: C4DUnhandledExceptionFilter: writing call stacks
2022-09-09 08:54:51:  0: WARNING: Monitored managed process Cinema4DProcess is no longer running
2022-09-09 08:54:52:  0: Done executing plugin command of type 'Render Task'

=======================================================
Details
=======================================================
Date: 09/09/2022 08:54:57
Frames: 16-20
Elapsed Time: 00:00:01:09
Job Submit Date: 09/09/2022 08:47:31
Job User: gavin.grant
Average RAM Usage: 18129575936 (14%)
Peak RAM Usage: 18471604224 (14%)
Average CPU Usage: 16%
Peak CPU Usage: 69%
Used CPU Clocks (x10^6 cycles): 289656
Total CPU Clocks (x10^6 cycles): 1810349

=======================================================
Worker Information
=======================================================
Worker Name: DHS-22-Big-Bertha
Version: v10.1.23.6 Release (773a6289d)
Operating System: Windows 10 Pro
Running As Service: No
Machine User: Big-Bertha
IP Address: 
MAC Address: 
CPU Architecture: x64
CPUs: 16
CPU Usage: 7%
Memory Usage: 17.0 GB / 127.6 GB (13%)
Free Disk Space: 657.145 GB 
Video Card: NVIDIA GeForce RTX 2080 Ti

Thus far, i have updated the repository from 10.1.16.5 - Uninstalled the workers from each node and re-installed. Though maybe i need to fully wipe the repo and start from scratch ?

Prior to updating the repo on one node i :

Uninstalled c4d & re-installed
Tested with moving all assets from one file to another and trying that file
Using a project that has all assets in one file ( previously lights and materials were Xref’d to a master file
Updated windows, system restored windows, updated and rolled back gpu drivers

Please let me know if i need to provide any further information

anthonygelatka · September 9, 2022, 8:33am

Couple of things that stand out for me

This looks like it could be nothing, as maybe it’s reading the c4d settings as opposed to the octane settings, looks like the path is set to F:\ … earlier in the log

2022-09-09 08:53:48:  0: STDOUT: There is no file name specified for the rendering image.
2022-09-09 08:53:48:  0: STDOUT: Do you want to continue without saving?

Is this Octane NetRender? does disabling this before submission help?

2022-09-09 08:53:49:  0: STDOUT: NetRender enabled. Slave count is 0

Are you monitoring the box running the job? How many GPU’s is it running? (With a name like Bertha it must be a lovely machine)

Have you created a ticket with Otoy? Have you tried the export to standalone and render?

gavin_grant · September 9, 2022, 8:58am

Hey there thanks for the input!

F:\ is a server storage on our LAN
I have also tried rendering to local C:\ with the same errors
I will try and disable the octane net render, i totally forgot that was left on!

The machine has x 8 2080ti’s it is indeed a lovely machine, when it work ^^

Additionally i have tried this across two additional render nodes, both with the same c4d install as well as octane release.

I have not opened a ticket as of yet, i will do that also

gavin_grant · September 9, 2022, 12:01pm

Hello all, small update in regards to all this.
We believe it was an Ip conflict, Something on the network / LAN - The ip on the repo machine has been set to static and config on our server has been told to expect the machine as static also instead of randomly assigning it an IP - This has also been done for the render node.

For three hours i have not received any errors in Deadline Where as before i was getting crashes very frequently, which links up with my randomness of my crashes where there was no continual trend with the rate of my crashes.

Now the hunt begins to find out what was causing the conflict, fun at a large studio