task progress 100% still in green color

Deadline
1

Hi,

We have a problem when rendering, the task progress shows its already 100% but it wont complete the task. Task progress still in green color even though reach 100% for like 5 hours, and it wont start the next task, seems like it stuck there until user manually set it to completed (Ctrl + M ).

Here is the slave task log:
2017-06-23 12:17:36: 0: STDOUT: [2017/Jun/23|12:17:35] V-Ray: Denoising complete in 216.75 s
2017-06-23 12:17:36: 0: STDOUT: [2017/Jun/23|12:17:36] V-Ray: Number of raycasts: 41767000443 (2400.16 per pixel)
2017-06-23 12:17:36: 0: STDOUT: [2017/Jun/23|12:17:36] V-Ray: Camera rays: 123416828 (7.09 per pixel)
2017-06-23 12:17:36: 0: STDOUT: [2017/Jun/23|12:17:36] V-Ray: Shadow rays: 1646416610 (94.61 per pixel)
2017-06-23 12:17:36: 0: STDOUT: [2017/Jun/23|12:17:36] V-Ray: GI rays: 333417069 (19.16 per pixel)
2017-06-23 12:17:36: 0: STDOUT: [2017/Jun/23|12:17:36] V-Ray: Reflection rays: 172799268 (9.93 per pixel)
2017-06-23 12:17:36: 0: STDOUT: [2017/Jun/23|12:17:36] V-Ray: Refraction rays: 0 (0.00 per pixel)
2017-06-23 12:17:36: 0: STDOUT: [2017/Jun/23|12:17:36] V-Ray: Unshaded rays: 39541346191 (2272.26 per pixel)
2017-06-23 12:17:36: 0: STDOUT: [2017/Jun/23|12:17:36] V-Ray: Light cache utilization: 99.41%
2017-06-23 12:17:36: 0: STDOUT: [2017/Jun/23|12:17:36] V-Ray: Premultiplied light cache utilization: 59.57%
2017-06-23 12:17:36: 0: STDOUT: [2017/Jun/23|12:17:36] V-Ray: Number of light evaluations: 1114048419 (64.02 per pixel)
2017-06-23 12:17:36: 0: STDOUT: [2017/Jun/23|12:17:36] V-Ray: Number of intersectable primitives: 196681
2017-06-23 12:17:36: 0: STDOUT: [2017/Jun/23|12:17:36] V-Ray: SD triangles: 196680
2017-06-23 12:17:36: 0: STDOUT: [2017/Jun/23|12:17:36] V-Ray: MB triangles: 0
2017-06-23 12:17:36: 0: STDOUT: [2017/Jun/23|12:17:36] V-Ray: Static primitives: 0
2017-06-23 12:17:36: 0: STDOUT: [2017/Jun/23|12:17:36] V-Ray: Moving primitives: 0
2017-06-23 12:17:36: 0: STDOUT: [2017/Jun/23|12:17:36] V-Ray: Infinite primitives: 1
2017-06-23 12:17:36: 0: STDOUT: [2017/Jun/23|12:17:36] V-Ray: Static hair segments: 0
2017-06-23 12:17:36: 0: STDOUT: [2017/Jun/23|12:17:36] V-Ray: Moving hair segments: 0
2017-06-23 12:17:51: 0: STDOUT: [2017/Jun/23|12:17:51] V-Ray: Successfully written image file “O:/Output/Active/P2297A_NipponIndia_POSM/3D/OutputTemp/testRender/weatherBond/beauty/WeatherbondBlobby_POS_beauty.0005.exr”
2017-06-23 12:17:51: 0: STDOUT: [2017/Jun/23|12:17:51] V-Ray: Total frame time 1h 26m 10.1s (5170.1 s)
2017-06-23 12:17:51: 0: STDOUT: [2017/Jun/23|12:17:51] V-Ray: Cleaning up bitmap manager
2017-06-23 12:17:51: 0: STDOUT: [2017/Jun/23|12:17:51] V-Ray: Cleaning up texture cache
2017-06-23 12:17:51: 0: STDOUT: [2017/Jun/23|12:17:51] V-Ray: Total sequence time 1h 26m 10.2s (5170.2 s)
2017-06-23 12:17:51: 0: STDOUT: [2017/Jun/23|12:17:51] V-Ray: Render complete
2017-06-23 12:17:51: 0: STDOUT: [2017/Jun/23|12:17:51] V-Ray: ========================
2017-06-23 12:17:51: 0: STDOUT: [2017/Jun/23|12:17:51] V-Ray: Clearing exporter memory…
2017-06-23 12:17:51: 0: STDOUT: [2017/Jun/23|12:17:51] V-Ray: Total time clearing exporter memory 0h 0m 0.0s (0.0 s)
2017-06-23 12:17:51: 0: STDOUT: Scene C:\Users\ren3d\AppData\Local\Thinkbox\Deadline8\slave\3d08\jobsData\594c82251b13fd1628929f36\thread0_tempXfY3W0\WeatherbondBlobby_POS_render_MASTER.ma completed.
2017-06-23 12:35:43: Skipping thermal shutdown check because it is not required at this time
2017-06-23 12:46:08: Skipping thermal shutdown check because it is not required at this time
2017-06-23 14:40:02: Skipping thermal shutdown check because it is not required at this time
2017-06-23 15:11:23: Skipping thermal shutdown check because it is not required at this time
2017-06-23 15:32:16: Skipping thermal shutdown check because it is not required at this time
2017-06-23 16:03:28: Skipping thermal shutdown check because it is not required at this time
2017-06-23 16:24:15: Skipping thermal shutdown check because it is not required at this time
2017-06-23 17:26:26: Skipping thermal shutdown check because it is not required at this time
2017-06-23 17:36:51: Skipping thermal shutdown check because it is not required at this time
2017-06-23 17:47:12: Skipping thermal shutdown check because it is not required at this time

Keep repeating the same command - ‘Skipping thermal shutdown check because it is not required at this time’ without move on to the next task.

2

Hi, I’m having a similar problem with a Houdini job, what’s the solution?

3

Hey,

What Deadline version are you on (ie. 10.0.18.1)?

The progress is determined manually by us through monitoring what the render application is spitting out to our logs but that doesn’t mean the job is “complete” when it reaches 100%. However, looking at your log, it seems like even though the render has completed, the underlying render process isn’t exiting which why we the job hasn’t moved over to “Complete”. Is the output correctly rendered?

Cheers

4

Hi

Thanks for your replied. I’m rendering via Deadline 10.0.16.6. Once the task is completed, the frames are good. For stall tasks, I would re-queue them or suspend and resubmit the job. When I resubmit the job, Deadline would render output 100+ frame before slowing down and stop outputting frames with tasks process at 0% and in green color. I change frame range to 1 frame per task and I got the same issue, but without “Skipping thermal shutdown check because it is not required at this time”. The job is a Houdini 16.5.473.

5

We have certainly seen render processes that won’t exit in the past.

The “Skipping thermal shutdown check because it is not required at this time” always runs at a specific interval, and so it’ll be the only output you’ll see if the render process has locked up.

Now, if you’re able to get this lock up to happen reliably, I’d suggest running through the render troubleshooting guide here:
docs.thinkboxsoftware.com/produ … m-deadline

Even running the application in question in non-batch mode is a good way to make sure it’s not a plugin-problem as the process runs through a different Deadline plugins in most cases (Nuke is the exception).

That testing should help you get to a spot where you can strip the project of plugins settings that may be causing the lock-up. I know I’ve run through that process many times in the past to find that a certain plugin has caused issues.