AWS Thinkbox Discussion Forums

Failed Vray DR jobs being marked as complete

Hi,

When Vray DR jobs crash or fail, they are getting marked as complete by Deadline.

This means that the dependancy job will start and render the image anyway (in our case, the Vray DR job was the irradiance map/light cache file)

It’s not always apparent when this has happened, I just notice massively long render times for the tile tasks.

Running 9.0.6.1, but it was doing the same on 9.0.1.1

Slave log below:

[code]

Log

2017-08-16 08:01:03: 0: Loading Job’s Plugin timeout is Disabled
2017-08-16 08:01:03: 0: INFO: Executing plugin script ‘C:\Users\richard\AppData\Local\Thinkbox\Deadline9\slave\w2\plugins\5993e58eec85f4310875a513\3dsCmd.py’
2017-08-16 08:01:03: 0: INFO: 3dsCmd Plugin Initializing…
2017-08-16 08:01:03: 0: INFO: Slave Running as Service: False
2017-08-16 08:01:03: 0: INFO: About: 3dsCmd Plugin for Deadline
2017-08-16 08:01:03: 0: INFO: Render Job As User disabled, running as current user ‘richard’
2017-08-16 08:01:03: 0: INFO: The job’s environment will be merged with the current environment before rendering
2017-08-16 08:01:03: 0: Start Job timeout is disabled.
2017-08-16 08:01:03: 0: Task timeout is disabled.
2017-08-16 08:01:03: 0: Loaded job: KBK_S01_CAM_BAK_[FD01] (Pre) (5993e58eec85f4310875a513)
2017-08-16 08:01:03: 0: Successfully mapped X: to //p/Assets
2017-08-16 08:01:03: 0: Successfully mapped Y: to //p/Resources
2017-08-16 08:01:03: 0: Successfully mapped Z: to //p/Projects
2017-08-16 08:01:03: 0: INFO: Start Job called - starting up 3dsCmd plugin
2017-08-16 08:01:03: 0: INFO: V-Ray DBR: Delaying load of 3dsmaxcmd until RenderTasks phase
2017-08-16 08:01:03: 0: Plugin rendering frame(s): 0
2017-08-16 08:01:03: 0: INFO: Render Tasks called
2017-08-16 08:01:03: 0: INFO: Releasing DBR job as MASTER has now been elected: W2
2017-08-16 08:01:03: 0: INFO: Sys Env Var PATH: C:\Program Files\Thinkbox\Deadline9\bin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Program Files\Common Files\Autodesk Shared;C:\Program Files (x86)\Skype\Phone;C:\Program Files (x86)\QuickTime\QTSystem;C:\Program Files (x86)\Autodesk\Backburner;C:\Users\richard\AppData\Local\Microsoft\WindowsApps;
2017-08-16 08:01:03: 0: INFO: Sys Env Var PATH length: 419
2017-08-16 08:01:03: 0: INFO: Backburner Path(s) Found in PATH: ‘C:\Program Files (x86)\Autodesk\Backburner’
2017-08-16 08:01:03: 0: INFO: Backburner server.exe version: 2016.0.0.1727
2017-08-16 08:01:03: 0: INFO: Rendering with 3dsmaxcmd version: 2016
2017-08-16 08:01:03: 0: INFO: Not enforcing a build of 3dsmax because version 2014 and later is 64 bit only
2017-08-16 08:01:03: 0: INFO: Slave 3dsmaxcmd.exe version: 18.0.873.0
2017-08-16 08:01:03: 0: INFO: Slave 3dsmaxcmd.exe description: 3ds Max 2016 base install
2017-08-16 08:01:03: 0: INFO: Slave 3dsmax.exe version: 18.0.873.0
2017-08-16 08:01:03: 0: INFO: Slave 3dsmax.exe description: 3ds Max 2016 base install
2017-08-16 08:01:03: 0: INFO: Submitted from 3dsmax.exe version: 18.8.739.0
2017-08-16 08:01:03: 0: INFO: Submitted from 3dsmax.exe description: 3ds Max 2016 + servicepack_sp4
2017-08-16 08:01:03: 0: WARNING: Slave’s 3dsmax version is NOT the same as the 3dsmax version that was used to submit this job! Unexpected results may occur!
2017-08-16 08:01:03: 0: INFO: Creating output directories if necessary…
2017-08-16 08:01:03: 0: INFO: V-Ray DBR: Configuring Distributed Render Job…
2017-08-16 08:01:03: 0: INFO: V-Ray DBR: Dynamic Start: True
2017-08-16 08:01:03: 0: INFO: V-Ray DBR: Plugin Config Settings to be applied to local file: vray_dr.cfg
2017-08-16 08:01:03: 0: INFO: V-Ray DBR: Port Range: 20204
2017-08-16 08:01:03: 0: INFO: V-Ray DBR: Use Local Machine: True
2017-08-16 08:01:03: 0: INFO: V-Ray DBR: Transfer Missing Assets: True
2017-08-16 08:01:03: 0: INFO: V-Ray DBR: Use Cached Assets: False
2017-08-16 08:01:03: 0: INFO: V-Ray DBR: Cache Limit Type: None
2017-08-16 08:01:03: 0: INFO: V-Ray DBR: Cache Limit: 100
2017-08-16 08:01:03: 0: INFO: V-Ray DBR: Starting distributed render immediately
2017-08-16 08:01:03: 0: INFO: V-Ray DBR: Use IP Address instead of Hostname: True
2017-08-16 08:01:03: 0: INFO: V-Ray DBR: Commencing distributed render with the following machines:
2017-08-16 08:01:03: 0: INFO: V-Ray DBR: 10.3.222.102
2017-08-16 08:01:03: 0: INFO: V-Ray DBR: 10.3.222.112
2017-08-16 08:01:03: 0: INFO: V-Ray DBR: 10.3.222.107
2017-08-16 08:01:03: 0: INFO: V-Ray DBR: 10.3.222.108
2017-08-16 08:01:03: 0: INFO: Language code string: ENU
2017-08-16 08:01:03: 0: INFO: Language sub directory: en-US
2017-08-16 08:01:03: 0: INFO: 3dsmax user profile path: C:\Users\richard\AppData\Local\Autodesk\3dsmax\2016 - 64bit\ENU
2017-08-16 08:01:03: 0: INFO: 3dsmax plugcfg directory: C:\Users\richard\AppData\Local\Autodesk\3dsmax\2016 - 64bit\ENU\en-US\plugcfg
2017-08-16 08:01:03: 0: INFO: V-Ray DBR: Backing up original config file to: C:\Users\richard\AppData\Local\Thinkbox\Deadline9\slave\w2\jobsData\5993e58eec85f4310875a513\vray_dr.cfg
2017-08-16 08:01:03: 0: INFO: V-Ray DBR: Deleting original config file: C:\Users\richard\AppData\Local\Autodesk\3dsmax\2016 - 64bit\ENU\en-US\plugcfg\vray_dr.cfg
2017-08-16 08:01:03: 0: INFO: V-Ray DBR: Config file created: C:\Users\richard\AppData\Local\Autodesk\3dsmax\2016 - 64bit\ENU\en-US\plugcfg\vray_dr.cfg
2017-08-16 08:01:03: 0: INFO: V-Ray DBR: Waiting 10 seconds to give V-Ray Spawner time to initialize on other machines
2017-08-16 08:01:13: 0: INFO: V-Ray DBR: Ready to go, moving on to distributed render
2017-08-16 08:01:13: 0: INFO: Rendering to local drive, will copy files and folders to final location after render is complete
2017-08-16 08:01:13: 0: INFO: V-Ray DBR: Rendering frame 0
2017-08-16 08:01:13: 0: INFO: Scene file to render: “C:\Users\richard\AppData\Local\Thinkbox\Deadline9\slave\w2\jobsData\5993e58eec85f4310875a513\KBK_S01_CAM_BAK_[FD01].max”
2017-08-16 08:01:13: 0: INFO: Starting monitored managed process CmdProcess
2017-08-16 08:01:13: 0: INFO: Stdout Redirection Enabled: True
2017-08-16 08:01:13: 0: INFO: Stdout Handling Enabled: True
2017-08-16 08:01:13: 0: INFO: Popup Handling Enabled: True
2017-08-16 08:01:13: 0: INFO: QT Popup Handling Enabled: True
2017-08-16 08:01:13: 0: INFO: WindowsForms10.Window.8.app.* Popup Handling Enabled: False
2017-08-16 08:01:13: 0: INFO: Using Process Tree: True
2017-08-16 08:01:13: 0: INFO: Hiding DOS Window: True
2017-08-16 08:01:13: 0: INFO: Creating New Console: False
2017-08-16 08:01:13: 0: INFO: Running as user: richard
2017-08-16 08:01:13: 0: INFO: Executable: “C:\Program Files\Autodesk\3ds Max 2016\3dsmaxcmd.exe”
2017-08-16 08:01:13: 0: INFO: Argument: “C:\Users\richard\AppData\Local\Thinkbox\Deadline9\slave\w2\jobsData\5993e58eec85f4310875a513\KBK_S01_CAM_BAK_[FD01].max” -v:4 -start:0 -end:0 -cam:“CAM -RENAME-ME-001” -w:3000 -h:4000 -pixelAspect:1.0 -atmospherics:true -renderHidden:false -effects:true -useAreaLights:false -displacements:true -force2Sided:false -videoColorCheck:false -superBlack:false -renderFields:false -fieldOrder:Odd -skipRenderedFrames:false -outputName:“C:\Users\richard\AppData\Local\Thinkbox\Deadline9\slave\w2\jobsData\5993e58eec85f4310875a513\3dsOutput_tempGZg5R0\SET_01_BAKER_TAPIOCA_CAM_A.exr” -renderElements:false -useAdvLight:true -computeAdvLight:false -gammaCorrection:true -continueOnError -rfw:1 -videopostJob:0
2017-08-16 08:01:13: 0: INFO: Full Command: “C:\Program Files\Autodesk\3ds Max 2016\3dsmaxcmd.exe” “C:\Users\richard\AppData\Local\Thinkbox\Deadline9\slave\w2\jobsData\5993e58eec85f4310875a513\KBK_S01_CAM_BAK_[FD01].max” -v:4 -start:0 -end:0 -cam:“CAM -RENAME-ME-001” -w:3000 -h:4000 -pixelAspect:1.0 -atmospherics:true -renderHidden:false -effects:true -useAreaLights:false -displacements:true -force2Sided:false -videoColorCheck:false -superBlack:false -renderFields:false -fieldOrder:Odd -skipRenderedFrames:false -outputName:“C:\Users\richard\AppData\Local\Thinkbox\Deadline9\slave\w2\jobsData\5993e58eec85f4310875a513\3dsOutput_tempGZg5R0\SET_01_BAKER_TAPIOCA_CAM_A.exr” -renderElements:false -useAdvLight:true -computeAdvLight:false -gammaCorrection:true -continueOnError -rfw:1 -videopostJob:0
2017-08-16 08:01:13: 0: INFO: Startup Directory: “C:\Program Files\Autodesk\3ds Max 2016”
2017-08-16 08:01:13: 0: INFO: Process Priority: BelowNormal
2017-08-16 08:01:13: 0: INFO: Process Affinity: default
2017-08-16 08:01:13: 0: INFO: Process is now running
2017-08-16 08:01:14: 0: STDOUT: 16/08/2017 08:01:14; 1 frames initialized
2017-08-16 08:01:14: 0: STDOUT: 16/08/2017 08:01:14;
2017-08-16 08:01:14: 0: STDOUT: Max install location: C:\Program Files\Autodesk\3ds Max 2016
2017-08-16 08:01:14: 0: STDOUT: 16/08/2017 08:01:14; Max file being rendered: C:\Users\richard\AppData\Local\Thinkbox\Deadline9\slave\w2\jobsData\5993e58eec85f4310875a513\KBK_S01_CAM_BAK_[FD01].max
2017-08-16 08:01:14: 0: STDOUT: 16/08/2017 08:01:14; Renderer: V-Ray Adv 3.60.01
2017-08-16 08:01:20: 0: INFO: Ignoring popup “MAXScript Debugger”
2017-08-16 08:01:32: 0: STDOUT: 16/08/2017 08:01:32; MENTAL RAY LOG: MI 0.0 8 MB progr: reading startup file “rayrc”
2017-08-16 08:01:32: 0: STDOUT: 16/08/2017 08:01:32; MENTAL RAY LOG: MI 0.0 8 MB progr: parsing file rayrc
2017-08-16 08:01:32: 0: STDOUT: 16/08/2017 08:01:32; MENTAL RAY LOG: LINK 0.0 8 MB progr: loaded shader library base.dll (.\shaders_standard\mentalray\shaders/base.dll)
2017-08-16 08:01:32: 0: STDOUT: 16/08/2017 08:01:32; MENTAL RAY LOG: LINK 0.0 8 MB progr: loaded shader library contour.dll (.\shaders_standard\mentalray\shaders/contour.dll)
2017-08-16 08:01:32: 0: STDOUT: 16/08/2017 08:01:32; MENTAL RAY LOG: LINK 0.0 8 MB progr: loaded shader library physics.dll (.\shaders_standard\mentalray\shaders/physics.dll)
2017-08-16 08:01:32: 0: STDOUT: 16/08/2017 08:01:32; MENTAL RAY LOG: LINK 0.0 8 MB progr: loaded shader library lume.dll (.\shaders_standard\mentalray\shaders/lume.dll)
2017-08-16 08:01:32: 0: STDOUT: 16/08/2017 08:01:32; MENTAL RAY LOG: LINK 0.0 8 MB progr: loaded shader library subsurface.dll (.\shaders_standard\mentalray\shaders/subsurface.dll)
2017-08-16 08:01:32: 0: STDOUT: 16/08/2017 08:01:32; MENTAL RAY LOG: LINK 0.0 8 MB progr: loaded shader library 3dsmaxshaders.dll (.\shaders_standard\mentalray\shaders/3dsmaxshaders.dll)
2017-08-16 08:01:32: 0: STDOUT: 16/08/2017 08:01:32; MENTAL RAY LOG: LINK 0.0 8 MB progr: loaded shader library 3dsmaxhair.dll (.\shaders_standard\mentalray\shaders/3dsmaxhair.dll)
2017-08-16 08:01:32: 0: STDOUT: 16/08/2017 08:01:32; MENTAL RAY LOG: LINK 0.0 8 MB progr: loaded shader library paint.dll (.\shaders_standard\mentalray\shaders/paint.dll)
2017-08-16 08:01:32: 0: STDOUT: 16/08/2017 08:01:32; MENTAL RAY LOG: LINK 0.0 8 MB progr: loaded shader library architectural.dll (.\shaders_standard\mentalray\shaders/architectural.dll)
2017-08-16 08:01:32: 0: STDOUT: 16/08/2017 08:01:32; MENTAL RAY LOG: LINK 0.0 8 MB progr: loaded shader library production.dll (.\shaders_standard\mentalray\shaders/production.dll)
2017-08-16 08:01:32: 0: STDOUT: 16/08/2017 08:01:32; MENTAL RAY LOG: LINK 0.0 8 MB progr: loaded shader library adskshaders.dll (.\shaders_standard\mentalray\shaders/adskshaders.dll)
2017-08-16 08:01:32: 0: STDOUT: 16/08/2017 08:01:32; MENTAL RAY LOG: LINK 0.0 8 MB progr: loaded shader library SubstanceShader.dll (.\shaders_standard\mentalray\shaders/SubstanceShader.dll)
2017-08-16 08:01:32: 0: STDOUT: 16/08/2017 08:01:32; MENTAL RAY LOG: LINK 0.0 8 MB progr: loaded shader library VectorMapShader.dll (.\shaders_standard\mentalray\shaders/VectorMapShader.dll)
2017-08-16 08:01:32: 0: STDOUT: 16/08/2017 08:01:32; MENTAL RAY LOG: LINK 0.0 11 MB progr: loaded shader library FShader3.dll (./FShader3.dll)
2017-08-16 08:01:32: 0: STDOUT: 16/08/2017 08:01:32; MENTAL RAY LOG: LINK 0.0 11 MB progr: loaded shader library PointCloudShader.dll (./PointCloudShader.dll)
2017-08-16 08:01:32: 0: STDOUT: 16/08/2017 08:01:32; MENTAL RAY LOG: LINK 0.0 11 MB progr: loaded shader library RCShader.dll (./RCShader.dll)
2017-08-16 08:01:32: 0: STDOUT: 16/08/2017 08:01:32; MENTAL RAY LOG: MI 0.0 11 MB progr: parsing file autodeskBase.mi
2017-08-16 08:01:32: 0: STDOUT: 16/08/2017 08:01:32; MENTAL RAY LOG: MI 0.0 11 MB progr: parsing file autodeskPointCloudBaseShader.mi
2017-08-16 08:01:32: 0: STDOUT: 16/08/2017 08:01:32; MENTAL RAY LOG: MI 0.0 11 MB progr: parsing file autodeskPointCloudShader.mi
2017-08-16 08:01:32: 0: STDOUT: 16/08/2017 08:01:32; MENTAL RAY LOG: MI 0.0 11 MB progr: parsing file FShader3.mi
2017-08-16 08:01:32: 0: STDOUT: 16/08/2017 08:01:32; MENTAL RAY LOG: MI 0.0 11 MB progr: parsing file RCShader.mi
2017-08-16 08:01:46: 0: STDOUT: 16/08/2017 08:01:46; An unexpected exception has occurred in the network renderer and it is terminating.
2017-08-16 08:01:46: 0: STDOUT: 16/08/2017 08:01:46; Error rendering frame 0: An unexpected exception has occurred in the network renderer and it is terminating.
2017-08-16 08:01:47: 0: STDOUT: 16/08/2017 08:01:47; Job Completed with Error(s) - see above
2017-08-16 08:01:47: 0: STDOUT: 16/08/2017 08:01:47; Scene C:\Users\richard\AppData\Local\Thinkbox\Deadline9\slave\w2\jobsData\5993e58eec85f4310875a513\KBK_S01_CAM_BAK_[FD01].max completed.
2017-08-16 08:01:47: 0: STDOUT: 16/08/2017 08:01:47; Max is down
2017-08-16 08:01:47: 0: STDOUT: 16/08/2017 08:01:47; Error occured while rendering job.
2017-08-16 08:01:47: 0: INFO: Moving output files and folders from C:\Users\richard\AppData\Local\Thinkbox\Deadline9\slave\w2\jobsData\5993e58eec85f4310875a513\3dsOutput_tempGZg5R0 to \p\Proj\2017\H\K_PHASE_2\STILLS\RENDERS\SET_01_BAKER_TAPIOCA\CAMEO_A\MAINPOOL
2017-08-16 08:01:47: 0: INFO: V-Ray DBR: Restoring backup config file: C:\Users\richard\AppData\Local\Thinkbox\Deadline9\slave\w2\jobsData\5993e58eec85f4310875a513\vray_dr.cfg to original location: C:\Users\richard\AppData\Local\Autodesk\3dsmax\2016 - 64bit\ENU\en-US\plugcfg\vray_dr.cfg
2017-08-16 08:01:47: 0: INFO: V-Ray DBR: Marking other incomplete tasks as complete

=======================================================
Details

Date: 08/16/2017 08:01:48
Frames: 0
Job Submit Date: 08/16/2017 07:26:23
Job User: matt
Average RAM Usage: 926345600 (2%)
Peak RAM Usage: 1646780416 (3%)
Average CPU Usage: 4%
Peak CPU Usage: 6%
Used CPU Clocks (x10^6 cycles): 44711
Total CPU Clocks (x10^6 cycles): 1117765

=======================================================
Slave Information

Slave Name: W2
Version: v9.0.1.1 Release (16886394d)
Operating System: Windows 10 Pro
Running As Service: No
Machine User: richard
IP Address: 10.3.222.2
MAC Address: 4C:CC:6A:63:9C:5A
CPU Architecture: x64
CPUs: 16
CPU Usage: 20%
Memory Usage: 15.8 GB / 63.9 GB (24%)
Free Disk Space: 1.215 TB (313.299 GB on C:, 931.178 GB on D:)
Video Card: LogMeIn Mirror Driver[/code]

Any ideas on a fix?

Thanks.

It looks like we just have to add a stdout handler to catch the errors. I would think “An unexpected exception has occurred in the network renderer and it is terminating.” is enough to go on here. :slight_smile:

I’ll make a dev issue for the integration team to take care of.

Any idea how we might be able to reproduce the problem? I’m not 100% what’ll happen when we re-start one of DR machines. The master would probably be the worst case and I’m not sure how well everyone would recover.

I would like to know what causes this error in the first place?
But yeah, catch the error via regex and then fail out the entire job, which will cause the spawner based slaves to get requeud clean.

It just seems to be random when it crashes. Restarting the job and everything works like it should.

I usually put it down to xrefs not loading in time (or scenes that aren’t properly cleaned/bitmap relinks etc) that cause the initial crash.

Quick update to this forum thread. So, the underlying issue is one of a change back in 3ds Max 2016 by ADSK, where 3dsmaxcmd now encodes StdOut to utf-16 instead of utf-8, which results in us being unable to ‘parse’ the StdOut and hence, our regex handler to identify any lines containing “Error” would go un-detected. This has now all been fixed with some other lovely little enhancements to both our main plugin architecture and the 3dsCmd plugin and should all appear in the next SP of DL9. That would be Service Pack 7 or the next build of DL 10 public preview.

Also, you should always run the same 3dsMax SP version between “submitted” machine and “slave” machine doing the rendering. All kinds of bad things generally happen otherwise. (see originally posted log for more details).

Privacy | Site terms | Cookie preferences