Can't submit DBR job to specific nodes

GrahamMac · October 2, 2023, 3:29pm

Hi,
I have a working custom submitter for sending 3dsmax jobs to deadline.
(3dsMax 2021, Deadline 10.3.0.10, Vray5.10.02)
It can send normal and DBR jobs without a problem.

I have just added a feature to select which nodes to whitelist/blacklist for a job. This also works fine, but only for non DBR jobs.
When I submit a DBR job the white/black list is empty in the job parameters, in the monitor, and the job renders with whatever nodes are available.

These are the SMTDSettings I’m setting for a DBR job:

SMTDSettings.DBR = true
SMTDSettings.DBRServers = 3 --this can be any number the user chooses
SMTDSettings.LimitEnabled = false
SMTDSettings.MachineLimit = 0
SMTDSettings.SlavesBlackList = false
SMTDSettings.SlavesList = #(“node01”, “node02”, “node03”)

Perhaps there are other SMTDSettings I need to set?

Any assistance much appreciated!!
Thanks
Graham

GrahamMac · October 3, 2023, 8:49am

I thought I found the solution!!
It worked at first, then didn’t!!:

–This works with the DBR it seems
SMTDSettings.MastersBlackList = true --or false
SMTDSettings.MastersList = #(“node01”, “node02”, “node03”)

–I also set these back so they don’t interfere:
SMTDSettings.SlavesBlackList = true
SMTDSettings.SlavesList = #()

My first render test worked perfectly: I submitted the DBR job to only use nodes 14,15,16 and it only used these nodes.

Then I repeated the test, (sent render from Max again) but this time it used 14,15,19 !!
I noticed the “allow list workers” in the job’s parameters, (which used to list 14,15,16) is now blank.
I suspended the job and started it again. Then it used 19,25,28 !!

maybe a deadline bug??!!
Thanks
Graham

Justin_B · October 9, 2023, 3:20pm

In your successive tests, what does the machine list look like in the submission params?

By the sound of it the jobs that are being created don’t have a machine list set on them, so whatever machines are idle are getting used.

Try submitting a DBR job with an allow/deny list using our existing submitter and compare the created job - I think there’s some nuance we might be missing here.

GrahamMac · October 10, 2023, 8:39am

Hi Justin, thanks for the reply!

I tried the stock submitter and am seeing similar behavior to my submitter. It’s really random, sometimes it renders with the correct nodes, sometimes it replaces one node with another, sometimes it uses all different nodes.

My test:
Using “submit Max to deadline” (the one with ALL the settings!):

I cleared out all nodes in the limits tab.
In the render tab under “V-ray/mental ray/corona DBR” I added Nodes 25,26,27,28.
Set to “Allow list”
Set “workers” to 4
Ticked the Off-Load checkbox.
Then I click Submit.

Node27 was chosen to lead the DBR job, so here is it’s log:

2023-10-10 09:01:46:  Scheduler Thread - Job's Limit Groups: 
2023-10-10 09:01:46:  0: Loading Job's Plugin timeout is Disabled
2023-10-10 09:01:46:  0: SandboxedPlugin: Render Job As User disabled, running as current user 'render'
2023-10-10 09:01:49:  Synchronization time for job files: 45.710 ms
2023-10-10 09:01:49:  Synchronizing Plugin 3dsCmd from \\yoda\deadlinerepository10\plugins\3dsCmd took: 0 seconds
2023-10-10 09:01:49:  0: Executing plugin command of type 'Initialize Plugin'
2023-10-10 09:01:49:  0: INFO: Executing plugin script 'C:\ProgramData\Thinkbox\Deadline10\workers\node27\plugins\652504e9937b530f1260211b\3dsCmd.py'
2023-10-10 09:01:49:  0: INFO: Plugin execution sandbox using Python version 3
2023-10-10 09:01:49:  0: INFO: 3dsCmd Plugin Initializing...
2023-10-10 09:01:49:  0: INFO: Worker Running as Service: False
2023-10-10 09:01:49:  0: INFO: About: 3dsCmd Plugin for Deadline
2023-10-10 09:01:49:  0: INFO: The job's environment will be merged with the current environment before rendering
2023-10-10 09:01:49:  0: Done executing plugin command of type 'Initialize Plugin'
2023-10-10 09:01:49:  0: Start Job timeout is disabled.
2023-10-10 09:01:49:  0: Task timeout is disabled.
2023-10-10 09:01:49:  0: Loaded job: Vray scene Max2021_BFLC (652504e9937b530f1260211b)
2023-10-10 09:01:49:  0: Executing plugin command of type 'Start Job'
2023-10-10 09:01:49:  0: DEBUG: S3BackedCache Client is not installed.
2023-10-10 09:01:49:  0: INFO: Executing global asset transfer preload script 'C:\ProgramData\Thinkbox\Deadline10\workers\node27\plugins\652504e9937b530f1260211b\GlobalAssetTransferPreLoad.py'
2023-10-10 09:01:49:  0: INFO: Looking for legacy (pre-10.0.26) AWS Portal File Transfer...
2023-10-10 09:01:49:  0: INFO: Looking for legacy (pre-10.0.26) File Transfer controller in C:/Program Files/Thinkbox/S3BackedCache/bin/task.py...
2023-10-10 09:01:49:  0: INFO: Could not find legacy (pre-10.0.26) AWS Portal File Transfer.
2023-10-10 09:01:49:  0: INFO: Legacy (pre-10.0.26) AWS Portal File Transfer is not installed on the system.
2023-10-10 09:01:49:  0: INFO: Start Job called - starting up 3dsCmd plugin
2023-10-10 09:01:49:  0: INFO: V-Ray DBR: Delaying load of 3dsmaxcmd until RenderTasks phase
2023-10-10 09:01:49:  0: Done executing plugin command of type 'Start Job'
2023-10-10 09:01:49:  0: Plugin rendering frame(s): 0
2023-10-10 09:01:50:  0: Executing plugin command of type 'Render Task'
2023-10-10 09:01:50:  0: INFO: Render Tasks called
2023-10-10 09:01:50:  0: INFO: Releasing DBR job as MASTER has now been elected: node27
2023-10-10 09:01:50:  0: INFO: Setting Process Environment Variable NW_ROOT_PATH to C:\ProgramData\Thinkbox\Deadline10\workers\node27\jobsData\652504e9937b530f1260211b\0_temp7zgMR0
2023-10-10 09:01:50:  0: INFO: Rendering with 3ds Max Cmd Version: 2021
2023-10-10 09:01:50:  0: INFO: Not enforcing a build of 3ds Max because version 2014 and later is 64 bit only
2023-10-10 09:01:50:  0: INFO: Worker 3ds Max Cmd Version: 23.3.6.3814
2023-10-10 09:01:50:  0: INFO: Worker 3ds Max Version: 23.3.6.3814
2023-10-10 09:01:50:  0: INFO: Submitted from 3ds Max Version: 23.3.14.3938
2023-10-10 09:01:50:  0: WARNING: Worker's 3ds Max version is NOT the same as the 3ds Max version that was used to submit this job! Unexpected results may occur!
2023-10-10 09:01:50:  0: INFO: Secure mode enabled: False
2023-10-10 09:01:50:  0: INFO: Creating output directories if necessary...
2023-10-10 09:01:50:  0: INFO: V-Ray DBR: Configuring Distributed Render Job...
2023-10-10 09:01:50:  0: INFO: V-Ray DBR: Dynamic Start: True
2023-10-10 09:01:50:  0: INFO: V-Ray DBR: Plugin Config Settings to be applied to local file: vray_dr.cfg
2023-10-10 09:01:50:  0: INFO: V-Ray DBR: Port Range: 20204
2023-10-10 09:01:50:  0: INFO: V-Ray DBR: Use Local Machine: True
2023-10-10 09:01:50:  0: INFO: V-Ray DBR: Transfer Missing Assets: False
2023-10-10 09:01:50:  0: INFO: V-Ray DBR: Use Cached Assets: False
2023-10-10 09:01:50:  0: INFO: V-Ray DBR: Cache Limit Type: None
2023-10-10 09:01:50:  0: INFO: V-Ray DBR: Cache Limit: 100
2023-10-10 09:01:50:  0: INFO: V-Ray DBR: Starting distributed render immediately
2023-10-10 09:01:50:  0: INFO: V-Ray DBR: Use IP Address instead of Hostname: False
2023-10-10 09:01:50:  0: INFO: V-Ray DBR: Commencing distributed render with the following machines:
2023-10-10 09:01:50:  0: INFO: V-Ray DBR:   node25
2023-10-10 09:01:50:  0: INFO: V-Ray DBR:   node26
2023-10-10 09:01:50:  0: INFO: Language code string: ENU
2023-10-10 09:01:50:  0: INFO: Language sub directory: en-US
2023-10-10 09:01:50:  0: INFO: 3dsmax user profile path: C:\Users\render\AppData\Local\Autodesk\3dsMax\2021 - 64bit\ENU
2023-10-10 09:01:50:  0: INFO: 3dsmax plugcfg directory: C:\Users\render\AppData\Local\Autodesk\3dsMax\2021 - 64bit\ENU\en-US\plugcfg
2023-10-10 09:01:50:  0: INFO: V-Ray DBR: Backing up original config file to: C:\ProgramData\Thinkbox\Deadline10\workers\node27\jobsData\652504e9937b530f1260211b\vray_dr.cfg
2023-10-10 09:01:50:  0: INFO: V-Ray DBR: Deleting original config file: C:\Users\render\AppData\Local\Autodesk\3dsMax\2021 - 64bit\ENU\en-US\plugcfg\vray_dr.cfg
2023-10-10 09:01:50:  0: INFO: V-Ray DBR: Config file created: C:\Users\render\AppData\Local\Autodesk\3dsMax\2021 - 64bit\ENU\en-US\plugcfg\vray_dr.cfg
2023-10-10 09:01:50:  0: INFO: V-Ray DBR: Waiting 10 seconds to give V-Ray Spawner time to initialize on other machines
2023-10-10 09:02:00:  0: INFO: V-Ray DBR: Ready to go, moving on to distributed render
2023-10-10 09:02:00:  0: INFO: Rendering to local drive, will copy files and folders to final location after render is complete
2023-10-10 09:02:00:  0: INFO: V-Ray DBR: Rendering frame 0
2023-10-10 09:02:00:  0: INFO: Scene file to render: "C:\ProgramData\Thinkbox\Deadline10\workers\node27\jobsData\652504e9937b530f1260211b\Vray scene Max2021_BFLC.max"
2023-10-10 09:02:00:  0: INFO: Starting monitored managed process CmdProcess
2023-10-10 09:02:00:  0: INFO: Stdout Redirection Enabled: True
2023-10-10 09:02:00:  0: INFO: Stdout Handling Enabled: True
2023-10-10 09:02:00:  0: INFO: Popup Handling Enabled: True
2023-10-10 09:02:00:  0: INFO: QT Popup Handling Enabled: True
2023-10-10 09:02:00:  0: INFO: WindowsForms10.Window.8.app.* Popup Handling Enabled: True
2023-10-10 09:02:00:  0: INFO: Using Process Tree: True
2023-10-10 09:02:00:  0: INFO: Hiding DOS Window: True
2023-10-10 09:02:00:  0: INFO: Creating New Console: False
2023-10-10 09:02:00:  0: INFO: Running as user: render
2023-10-10 09:02:00:  0: INFO: Executable: "C:\Program Files\Autodesk\3ds Max 2021\3dsmaxcmd.exe"
2023-10-10 09:02:00:  0: INFO: Argument: "C:\ProgramData\Thinkbox\Deadline10\workers\node27\jobsData\652504e9937b530f1260211b\Vray scene Max2021_BFLC.max" -v:4 -start:0 -end:0 -cam:"Camera002" -w:6000 -h:3375 -pixelAspect:1.0 -atmospherics:true -renderHidden:false -effects:true -useAreaLights:false -displacements:true -force2Sided:false -videoColorCheck:false -superBlack:false -renderFields:false -fieldOrder:Odd -skipRenderedFrames:false -outputName:"C:\ProgramData\Thinkbox\Deadline10\workers\node27\jobsData\652504e9937b530f1260211b\3dsOutput_tempjyW2U0\D1_F00_Camera002_Opt01_.jpg" -renderElements:true -useAdvLight:true -computeAdvLight:false -continueOnError -rfw:0 -videopostJob:0
2023-10-10 09:02:00:  0: INFO: Full Command: "C:\Program Files\Autodesk\3ds Max 2021\3dsmaxcmd.exe" "C:\ProgramData\Thinkbox\Deadline10\workers\node27\jobsData\652504e9937b530f1260211b\Vray scene Max2021_BFLC.max" -v:4 -start:0 -end:0 -cam:"Camera002" -w:6000 -h:3375 -pixelAspect:1.0 -atmospherics:true -renderHidden:false -effects:true -useAreaLights:false -displacements:true -force2Sided:false -videoColorCheck:false -superBlack:false -renderFields:false -fieldOrder:Odd -skipRenderedFrames:false -outputName:"C:\ProgramData\Thinkbox\Deadline10\workers\node27\jobsData\652504e9937b530f1260211b\3dsOutput_tempjyW2U0\D1_F00_Camera002_Opt01_.jpg" -renderElements:true -useAdvLight:true -computeAdvLight:false -continueOnError -rfw:0 -videopostJob:0
2023-10-10 09:02:00:  0: INFO: Startup Directory: "C:\Program Files\Autodesk\3ds Max 2021"
2023-10-10 09:02:00:  0: INFO: Process Priority: BelowNormal
2023-10-10 09:02:00:  0: INFO: Process Affinity: default
2023-10-10 09:02:00:  0: INFO: Process is now running
2023-10-10 09:02:00:  0: STDOUT: 10/10/2023 09:02:00;  1 frames initialized
2023-10-10 09:02:00:  0: STDOUT: 10/10/2023 09:02:00;  Max install location: C:\Program Files\Autodesk\3ds Max 2021\
2023-10-10 09:02:00:  0: STDOUT: 10/10/2023 09:02:00;  Max executable: 3dsmax.exe
2023-10-10 09:02:00:  0: STDOUT: 10/10/2023 09:02:00;  Max file being rendered: C:\ProgramData\Thinkbox\Deadline10\workers\node27\jobsData\652504e9937b530f1260211b\Vray scene Max2021_BFLC.max
2023-10-10 09:02:00:  0: STDOUT: 10/10/2023 09:02:00;  Renderer: V-Ray 5, update 1.2
2023-10-10 09:02:01:  0: STDOUT: 10/10/2023 09:02:01;  Python version: 3
2023-10-10 09:02:01:  0: STDOUT: 10/10/2023 09:02:01;  Product version: 3ds Max 2021.3.6 Security Fix (23.3.6.3814)
2023-10-10 09:02:01:  0: STDOUT: 10/10/2023 09:02:01;  Data collection and use is 'ON'. Change your participation anytime in the Help menu of 3ds Max.
2023-10-10 09:02:12:  Skipping thermal shutdown check because it is not required at this time
2023-10-10 09:02:12:  0: STDOUT: 10/10/2023 09:02:11;  Autodesk 3ds Max Security Tools is active via configuration settings
2023-10-10 09:02:12:  0: STDOUT: 10/10/2023 09:02:11;  ALC Security Tool 1.1 loaded.  (c:\program files\autodesk\3ds max 2021\applicationplugins\securitytools\contents\scripts\ALC_SecurityTool.ms)
2023-10-10 09:02:12:  0: STDOUT: 10/10/2023 09:02:11;  ALC2 Security Tool 1.1 loaded.  (c:\program files\autodesk\3ds max 2021\applicationplugins\securitytools\contents\scripts\ALC2_SecurityTool.ms)
2023-10-10 09:02:12:  0: STDOUT: 10/10/2023 09:02:11;  CRP Security Tool 1.0 loaded.  (c:\program files\autodesk\3ds max 2021\applicationplugins\securitytools\contents\scripts\CRP_SecurityTool.ms)
2023-10-10 09:02:12:  0: STDOUT: 10/10/2023 09:02:11;  ADSL Security Tool 1.0 loaded.  (c:\program files\autodesk\3ds max 2021\applicationplugins\securitytools\contents\scripts\ADSL_SecurityTool.ms)
2023-10-10 09:02:12:  0: STDOUT: 10/10/2023 09:02:11;  PhysXPluginMfx Security Tool 1.0 loaded.  (c:\program files\autodesk\3ds max 2021\applicationplugins\securitytools\contents\scripts\PhysXPluginMfx_SecurityTool.ms)
2023-10-10 09:02:12:  0: STDOUT: 10/10/2023 09:02:11;  PhysXPluginMfx2 Security Tool 1.0 loaded.  (c:\program files\autodesk\3ds max 2021\applicationplugins\securitytools\contents\scripts\PhysXPluginMfx2_SecurityTool.ms)
2023-10-10 09:02:26:  0: STDOUT: 10/10/2023 09:02:26;  Initializing Python version: 3.7.6
2023-10-10 09:02:28:  0: STDOUT: 10/10/2023 09:02:27;  Max is ready
2023-10-10 09:02:28:  0: STDOUT: 10/10/2023 09:02:28;  Frame 0 assigned
2023-10-10 09:02:28:  0: STDOUT: 10/10/2023 09:02:28;  [V-Ray] ============================================================
2023-10-10 09:02:28:  0: STDOUT: 10/10/2023 09:02:28;  [V-Ray] Console created, V-Ray 5.10.02 for x64 from Apr 19 2021, 11:34:41
2023-10-10 09:02:28:  0: STDOUT: 10/10/2023 09:02:28;  [V-Ray] ============================================================
2023-10-10 09:02:28:  0: STDOUT: 10/10/2023 09:02:28;  [V-Ray] Compiled with Microsoft Visual C++ compiler, version 19.12
2023-10-10 09:02:28:  0: STDOUT: 10/10/2023 09:02:28;  [V-Ray] Host is 3ds Max, version 23
2023-10-10 09:02:28:  0: STDOUT: 10/10/2023 09:02:28;  [V-Ray] V-Ray DLL version is 5.10.02
2023-10-10 09:02:30:  0: INFO: V-Ray DBR: Updating cfg file for distributed render with the following machines:
2023-10-10 09:02:30:  0: INFO: V-Ray DBR:   node25
2023-10-10 09:02:30:  0: INFO: V-Ray DBR:   node26
2023-10-10 09:02:30:  0: INFO: V-Ray DBR:   node14
2023-10-10 09:02:50:  0: STDOUT: 10/10/2023 09:02:50;  BEGIN FRAME 0; TASK 1 OF 1; (C:\ProgramData\Thinkbox\Deadline10\workers\node27\jobsData\652504e9937b530f1260211b\3dsOutput_tempjyW2U0\D1_F00_Camera002_Opt01_0000.jpg);
2023-10-10 09:06:43:  0: STDOUT: 10/10/2023 09:06:42;  END FRAME 0; TASK 1 OF 1; (C:\ProgramData\Thinkbox\Deadline10\workers\node27\jobsData\652504e9937b530f1260211b\3dsOutput_tempjyW2U0\D1_F00_Camera002_Opt01_0000.jpg); 233s; 0 faces; MEM 15266 MB;
2023-10-10 09:06:43:  0: STDOUT: 10/10/2023 09:06:42;  Job Complete - Results in C:\ProgramData\Thinkbox\Deadline10\workers\node27\jobsData\652504e9937b530f1260211b\3dsOutput_tempjyW2U0\
2023-10-10 09:06:43:  0: STDOUT: 10/10/2023 09:06:43;  Job Completed Successfully
2023-10-10 09:06:43:  0: STDOUT: 10/10/2023 09:06:43;  Scene C:\ProgramData\Thinkbox\Deadline10\workers\node27\jobsData\652504e9937b530f1260211b\Vray scene Max2021_BFLC.max completed.

As you can see, under: “Commencing distributed render with the following machines:”
It only lists 25 and 26 (27 is leading, so clearly doesn’t need to be here, but no 28!)

Later on under: “Updating cfg file for distributed render with the following machines:”
It lists nodes 25,26,14

The nodes which actually rendered were indeed: 14,25,26,27

Separately, in case this helps, I noticed that after submitting any DBR job (with specified nodes) from any submitter, once the job starts (or is completed) the “Allow list workers” in job’s properties in the Monitor, is always cleared out.
If I submit a suspended job and before resuming job, I check “Allow list workers”, I always see the correct nodes listed there.

Hopefully this info helps!

Cheers,
Graham

Justin_B · October 12, 2023, 2:46pm

That bit about the machine list being different is a surprise - does that happen if you use the V-Ray Swarm submitter as well? It’s the one under Submit->3D->Vray Swarm.

Anyways, tracking the logic to see what’s going on here:

The bit of that log here:

2023-10-10 09:01:50:  0: INFO: V-Ray DBR: Commencing distributed render with the following machines:
2023-10-10 09:01:50:  0: INFO: V-Ray DBR:   node25
2023-10-10 09:01:50:  0: INFO: V-Ray DBR:   node26

Is driven by GetDBRMachines() on line 859 from 3dscmd.py (the application plugin), called on line 208 of the same file.

What it does is pull the list of workers that have checked out tasks from the job, and uses that list to update the V-Ray DBR config file. It’s not pulling from the machine list, the assumption is that the machine list is only made of idle machines (enforced by the submitter) and thus they’d all be able to pick up tasks from the job.

This part here:

2023-10-10 09:02:30:  0: INFO: V-Ray DBR: Updating cfg file for distributed render with the following machines:
2023-10-10 09:02:30:  0: INFO: V-Ray DBR:   node25
2023-10-10 09:02:30:  0: INFO: V-Ray DBR:   node26
2023-10-10 09:02:30:  0: INFO: V-Ray DBR:   node14

Is driven by the same function call, just a little afterwards. Likely to keep the list of machines that should be added to the DBR config fresh.

After that the plugin hangs out waiting for a ‘render complete’ response from the DBR system, and the job is marked completed.

What this means is there’s some other function clearing out the machine list - have you got any event plugins that modify the list? Check the job history to see if there’s an entry where the machine limit is getting updated, it should include what’s making the change (user or script, etc). You can get the history from the job’s context menu.

Let us know what you see!

GrahamMac · October 26, 2023, 5:57am

Thank you again, Justin for the reply and suggestions.
I will test all this as soon as I can clear my other deadlines (no pun intended ) and post back here ASAP!