Node won't start concurrent tasks

oborgstrom · February 13, 2020, 2:31pm

Hi,

I’m creating a custom plugin which runs an exe with some arguments. I’m setting the concurrent tasks to 4 and I’m setting the Concurrent Task Parameter as follows:

[ConcurrentTasks]
Type=label
Label=ConcurrentTasks
Category=About Plugin
CategoryOrder=-1
Index=0
Default=True
Description=Not configurable

See below General properties under job details in the monitor. (Beteween typo in monitor)

Property	Value
Comment
Department	ARD
Pool	ard
Secondary Pool
Group	etabs
Batch Name
Priority	100
Concurrent Tasks	4
Limit Tasks to Slave’s Task Limit	No
On Job Complete	Do nothing
Job Is Protected	No
Re-synchronize Auxiliary Files Beteween Tasks	No
Reload Plugin Between Tasks	No
Enforce Sequential Rendering	No
Job Is Interruptible	No
Progress cut-off for Job Interruption	100
Remaining Time Threshold for Job Interruption	0
Suppress Event Plugins	No

I’m submitting the job with 4 Commands but still it only runs one task at a time per node.
Does anyone know what I’m missing?

Using: “Deadline Client Version: 10.0.24.4 Release”

Thanks in advance

kavi · February 13, 2020, 11:03pm

Hey @oborgstrom

Whats the status of the other tasks in the task panel? Are they in “Waiting to start”? And the remaining tasks eventually complete or they are stuck in waiting status? If possible send over your job logs.

There was a bug in concurrent tasks feature where the worker would skip over some tasks and leave them in the “Waiting to start state” . This was fixed from Deadline 10.0.27.3.

oborgstrom · February 14, 2020, 9:19am

Hi @kavi

The other tasks are in queue and rendered and eventually completed one after another. As if I only allowed one concurrent task.

Here’s the log for the first and second tasks

Log

2020-02-13 14:07:50: 0: Plugin will be reloaded because a new job has been loaded.
2020-02-13 14:07:50: 0: Loading Job’s Plugin timeout is Disabled
2020-02-13 14:07:50: 0: Loaded plugin ETABSWorker
2020-02-13 14:07:50: 0: INFO: Executing plugin script ‘C:\Users\svc-render\AppData\Local\Thinkbox\Deadline10\slave\LONREN209\plugins\5e4558222b736353882778b3\ETABSWorker.py’
2020-02-13 14:07:50: 0: INFO: About: CSi ETABS Plugin for Deadline
2020-02-13 14:07:50: 0: INFO: Render Job As User disabled, running as current user ‘svc-render’
2020-02-13 14:07:50: 0: INFO: The job’s environment will be merged with the current environment before rendering
2020-02-13 14:07:50: 0: Start Job timeout is disabled.
2020-02-13 14:07:50: 0: Task timeout is disabled.
2020-02-13 14:07:50: 0: Loaded job: ETABS_test (5e4558222b736353882778b3)
2020-02-13 14:07:50: 0: INFO: Executing global asset transfer preload script ‘C:\Users\svc-render\AppData\Local\Thinkbox\Deadline10\slave\LONREN209\plugins\5e4558222b736353882778b3\GlobalAssetTransferPreLoad.py’
2020-02-13 14:07:50: 0: INFO: Looking for AWS Portal File Transfer…
2020-02-13 14:07:50: 0: INFO: Looking for File Transfer controller in C:/Program Files/Thinkbox/S3BackedCache/bin/task.py…
2020-02-13 14:07:50: 0: INFO: Could not find AWS Portal File Transfer.
2020-02-13 14:07:50: 0: INFO: AWS Portal File Transfer is not installed on the system.
2020-02-13 14:07:50: 0: Plugin rendering frame(s): 0
2020-02-13 14:07:50: 0: INFO: Stdout Redirection Enabled: True
2020-02-13 14:07:50: 0: INFO: Stdout Handling Enabled: False
2020-02-13 14:07:50: 0: INFO: Popup Handling Enabled: False
2020-02-13 14:07:50: 0: INFO: Using Process Tree: True
2020-02-13 14:07:50: 0: INFO: Hiding DOS Window: True
2020-02-13 14:07:50: 0: INFO: Creating New Console: False
2020-02-13 14:07:50: 0: INFO: Running as user: svc-render
2020-02-13 14:07:50: 0: INFO: Executable: “O:\Public\Transfer\00_Distributed_Comptuing\02_ETABS_Analysis\00_ETABS_Files\Hydra.Deadline.ETABSWorker.exe”
2020-02-13 14:07:50: 0: INFO: Checking line: O:\Public\Transfer\00_Distributed_Comptuing\02_ETABS_Analysis\01_Testrun\181106_Tulip_V148_Final_Load_Combos.EDB O:\Public\Transfer\00_Distributed_Comptuing\02_ETABS_Analysis\01_Testrun\Instructions\Instruction000.txt 000
2020-02-13 14:07:50: 0: INFO: Argument: O:\Public\Transfer\00_Distributed_Comptuing\02_ETABS_Analysis\01_Testrun\181106_Tulip_V148_Final_Load_Combos.EDB O:\Public\Transfer\00_Distributed_Comptuing\02_ETABS_Analysis\01_Testrun\Instructions\Instruction000.txt 000
2020-02-13 14:07:50: 0: INFO: Full Command: “O:\Public\Transfer\00_Distributed_Comptuing\02_ETABS_Analysis\00_ETABS_Files\Hydra.Deadline.ETABSWorker.exe” O:\Public\Transfer\00_Distributed_Comptuing\02_ETABS_Analysis\01_Testrun\181106_Tulip_V148_Final_Load_Combos.EDB O:\Public\Transfer\00_Distributed_Comptuing\02_ETABS_Analysis\01_Testrun\Instructions\Instruction000.txt 000
2020-02-13 14:07:50: 0: INFO: Startup Directory: “O:\Public\Transfer\00_Distributed_Comptuing\02_ETABS_Analysis\00_ETABS_Files”
2020-02-13 14:07:50: 0: INFO: Process Priority: BelowNormal
2020-02-13 14:07:50: 0: INFO: Process Affinity: default
2020-02-13 14:07:50: 0: INFO: Process is now running
2020-02-13 14:08:04: 0: STDOUT: Cannot open file
2020-02-13 14:08:04: 0: STDOUT: BufferFileIn::BufferFileIn():
2020-02-13 14:08:04: 0: STDOUT: fileName = C:\ETABS_Temp_Parallel_Analysis000\181106_Tulip_V148_Final_Load_Combos000.Y_
2020-02-13 14:11:13: 0: INFO: Process exit code: 0
2020-02-13 14:11:13: 0: Render time for frame(s): 3.390 m
2020-02-13 14:11:13: 0: Total time for task: 3.414 m
2020-02-13 14:11:14: 0: Saving task log…

Details

Date: 02/13/2020 14:11:14
Frames: 0
Job Submit Date: 02/13/2020 14:07:30
Job User: oborgstrom
Average RAM Usage: 24134608 (1%)
Peak RAM Usage: 30760960 (1%)
Average CPU Usage: 0%
Peak CPU Usage: 0%
Used CPU Clocks (x10^6 cycles): 0
Total CPU Clocks (x10^6 cycles): 20397146

Slave Information

Slave Name: LONREN209
Version: v10.0.24.4 Release (cda40613f)
Operating System: Windows 10 Enterprise
Running As Service: No
Machine User: svc-render
IP Address: 10.11.6.77
MAC Address: 0C:C4:7A:52:0F:AA
CPU Architecture: x64
CPUs: 40
CPU Usage: 2%
Memory Usage: 6.2 GB / 63.9 GB (9%)
Free Disk Space: 621.642 GB
Video Card: Microsoft Basic Display Adapter

Log

2020-02-13 14:11:17: 0: Plugin already loaded: ETABSWorker
2020-02-13 14:11:17: 0: Start Job timeout is disabled.
2020-02-13 14:11:17: 0: Task timeout is disabled.
2020-02-13 14:11:17: 0: Job already loaded: ETABS_test
2020-02-13 14:11:17: 0: Plugin rendering frame(s): 1
2020-02-13 14:11:17: 0: INFO: Stdout Redirection Enabled: True
2020-02-13 14:11:17: 0: INFO: Stdout Handling Enabled: False
2020-02-13 14:11:17: 0: INFO: Popup Handling Enabled: False
2020-02-13 14:11:17: 0: INFO: Using Process Tree: True
2020-02-13 14:11:17: 0: INFO: Hiding DOS Window: True
2020-02-13 14:11:17: 0: INFO: Creating New Console: False
2020-02-13 14:11:17: 0: INFO: Running as user: svc-render
2020-02-13 14:11:17: 0: INFO: Executable: “O:\Public\Transfer\00_Distributed_Comptuing\02_ETABS_Analysis\00_ETABS_Files\Hydra.Deadline.ETABSWorker.exe”
2020-02-13 14:11:17: 0: INFO: Checking line: O:\Public\Transfer\00_Distributed_Comptuing\02_ETABS_Analysis\01_Testrun\181106_Tulip_V148_Final_Load_Combos.EDB O:\Public\Transfer\00_Distributed_Comptuing\02_ETABS_Analysis\01_Testrun\Instructions\Instruction001.txt 001
2020-02-13 14:11:17: 0: INFO: Argument: O:\Public\Transfer\00_Distributed_Comptuing\02_ETABS_Analysis\01_Testrun\181106_Tulip_V148_Final_Load_Combos.EDB O:\Public\Transfer\00_Distributed_Comptuing\02_ETABS_Analysis\01_Testrun\Instructions\Instruction001.txt 001
2020-02-13 14:11:17: 0: INFO: Full Command: “O:\Public\Transfer\00_Distributed_Comptuing\02_ETABS_Analysis\00_ETABS_Files\Hydra.Deadline.ETABSWorker.exe” O:\Public\Transfer\00_Distributed_Comptuing\02_ETABS_Analysis\01_Testrun\181106_Tulip_V148_Final_Load_Combos.EDB O:\Public\Transfer\00_Distributed_Comptuing\02_ETABS_Analysis\01_Testrun\Instructions\Instruction001.txt 001
2020-02-13 14:11:17: 0: INFO: Startup Directory: “O:\Public\Transfer\00_Distributed_Comptuing\02_ETABS_Analysis\00_ETABS_Files”
2020-02-13 14:11:17: 0: INFO: Process Priority: BelowNormal
2020-02-13 14:11:17: 0: INFO: Process Affinity: default
2020-02-13 14:11:17: 0: INFO: Process is now running
2020-02-13 14:11:33: 0: STDOUT: Cannot open file
2020-02-13 14:11:33: 0: STDOUT: BufferFileIn::BufferFileIn():
2020-02-13 14:11:33: 0: STDOUT: fileName = C:\ETABS_Temp_Parallel_Aalysis001\181106_Tulip_V148_Final_Load_Combos001.Y_
2020-02-13 14:13:06: 0: INFO: Process exit code: 0
2020-02-13 14:13:06: 0: Render time for frame(s): 1.817 m
2020-02-13 14:13:06: 0: Total time for task: 1.824 m
2020-02-13 14:13:07: 0: Saving task log…

Details

Date: 02/13/2020 14:13:07
Frames: 1
Job Submit Date: 02/13/2020 14:07:30
Job User: oborgstrom
Average RAM Usage: 23921950 (1%)
Peak RAM Usage: 31043584 (1%)
Average CPU Usage: 0%
Peak CPU Usage: 0%
Used CPU Clocks (x10^6 cycles): 0
Total CPU Clocks (x10^6 cycles): 11069336

Slave Information

Slave Name: LONREN209
Version: v10.0.24.4 Release (cda40613f)
Operating System: Windows 10 Enterprise
Running As Service: No
Machine User: svc-render
IP Address: 10.11.6.77
MAC Address: 0C:C4:7A:52:0F:AA
CPU Architecture: x64
CPUs: 40
CPU Usage: 2%
Memory Usage: 6.1 GB / 63.9 GB (9%)
Free Disk Space: 621.889 GB
Video Card: Microsoft Basic Display Adapter

Also find attached my plugin files and the files that are created as I run the job.
ETABSWorkerPlugin.zip (1.4 KB)
job_files.zip (914 Bytes)

Thank you again in advance
Oscar

oborgstrom · February 14, 2020, 2:42pm

Hi again @kavi

I managed to get it working. Seems like the Node was caching and older version of the plugin and didn’t reload my updates.

Oscar