Hello!
I have this posted on the NewTek forums as well but just in case I thought I would post this here too as we would love to get this problem solved sooner than later.
We’ve run into a bit of a snag here at the studio. We are using clean installs of LW 11.6.3 Build 2737 and Deadline 6.2 as our network rendering controller.
Everything works fine but occasionally a render will error out. Usually this happens on longer renders, think 30mins/frame upwards to 2-3hrs/frame, but can sometimes happen on shorter renders. The problem comes after a frame is done rendering and goes to save the image. Here is a snippet of log from one of our failed renders as it was finishing up a frame -
.........
0: STDOUT: Image Reconstruction : 96%.
0: STDOUT: Image Reconstruction : 96%.
0: STDOUT: Image Reconstruction : 97%.
0: STDOUT: Image Reconstruction : 97%.
0: STDOUT: Image Reconstruction : 97%.
0: STDOUT: Image Reconstruction : 98%.
0: STDOUT: Image Reconstruction : 98%.
0: STDOUT: Image Reconstruction : 98%.
0: STDOUT: Image Reconstruction : 99%.
0: STDOUT: Image Reconstruction : 99%.
0: STDOUT: Writing RGB image to Z:\_Projects\_zFarmTest\Frames\[REDACTED]\Server_Room029_V011_Render_Test_V002_008.exr.
0: WARNING: Monitored managed process Lightwave0 is no longer running
=======================================================
Details
=======================================================
Date: 07/01/2014 21:32:06
Frames: 8
Elapsed Time: 00:03:39:38
Job Submit Date: 07/01/2014 17:52:20
Job User: [redacted]
Average RAM Usage: 3028160256 (15%)
Peak RAM Usage: 3045830656 (15%)
Average CPU Usage: 99%
Peak CPU Usage: 100%So for whatever reason, the image gets to 99% (after rendering for 3hrs 39mins) and can’t write out the image file as it thinks LWSN is no longer running? To make sure it was not a connection issue (think node timeout), I created a batch that would write a small .txt file to the server every 5 mins, to make sure the mapped drives weren’t disconnected on these longer renders. Those all would consistently write to the server and this random network render error persisted.
Here is a snippet of log from the same scene that DID successfully render. Same node, same scene, some settings reduced to slightly reduce render time (AA, MB, etc). (Render settings / render time seems irrelevant to this problem though)
..........
0: STDOUT: Image Reconstruction : 96%.
0: STDOUT: Image Reconstruction : 97%.
0: STDOUT: Image Reconstruction : 97%.
0: STDOUT: Image Reconstruction : 97%.
0: STDOUT: Image Reconstruction : 98%.
0: STDOUT: Image Reconstruction : 98%.
0: STDOUT: Image Reconstruction : 98%.
0: STDOUT: Image Reconstruction : 99%.
0: STDOUT: Image Reconstruction : 99%.
0: STDOUT: Writing RGB image to Z:\_Projects\_zFarmTest\Frames\[REDACTED]\Server_Room029_V011_Render_Test_V003_003.exr.
0: STDOUT: Frame completed.
0: STDOUT: Last Frame Rendered: 3.
0: STDOUT: Rendering Time: 2h 24m 1s (8641.5 seconds).
0: STDOUT: Freeing segment buffers.
0: STDOUT: Freeing frame buffers.
0: STDOUT: LightWave command: wait.
0: INFO: Received response: Ready
0: INFO: Finished Lightwave Rendering Phase
=======================================================
Details
=======================================================
Date: 08/04/2014 23:04:15
Frames: 3
Job Submit Date: 08/04/2014 18:14:20
Job User: [redacted]
Average RAM Usage: 0 (0%)
Peak RAM Usage: 0 (0%)
Average CPU Usage: 0%
Peak CPU Usage: 0%There aren’t any plugin loading errors, and I’ve tried running scenes in both -2 mode and -3 on the farm and they both experience the random error of not being able to write the image due to Lightwave0 no longer running. As noted earlier, this has happened on scenes that can take a couple minutes a frame, or a couple hours a frame, it appears to be completely random.
This issue also happened when we were running BNR5 and LW 11.5 so it seems to be a random LWSN issue?
Hope someone can help as it’s very frustrating when we wait HOURS per frame for some renders only to have random nodes fail when they go to write the image. LW is the only app having this problem at the moment.