Hi
I want to submit e.g. 500 jobs.
I do not want to call the deadlinecommand 500 times as it is veeery slow (25 min and more).
I have seen that you can use job files and that deadlinecommand supports loading multiple files.
But I do not want to create 500 job files.
Is there any way to use one job file with 500 jobs?
Or how would you solve this issue?
If it is not possible via commandline, are there python modules to submit jobs?
Depends on the kind of jobs. I’d probably try to wrap in in one job with 500 tasks. This isn’t viable for every batch of jobs though.
Possibly submitting a job that submits those 500 jobs would br an option.
Otherwise I’d just live with the longer submission time if 500 jobs i guess, can’t be that horrible.
I’d probably try to wrap in in one job with 500 tasks
Each job has 100 frames.
And If I use one job, I loose the batch name and user name for the jobs
can’t be that horrible.
It takes 25 minutes.
(CPU is Ryzen 3700 with 3.66Ghz, so that should not be the bottleneck. Deadline without TLS/SSH. Does not matter if connection is direct or via RCS proxy)
Anyway, thanks.
Ist that for eauch job? That’s indeed horrible. For all 500 personally I’d be ready to live with that if I didn’t have to do it multiple times a day. Doesn’t mean that it can’t be improved.
I don’t think there’s a way to sobmit jobs other then those you already know of.
What kinds of jobs are you submitting? How often do you expect to have that many jobs to submit? Are you submitting auxiliary files, if yes is there a way to avoid that?
Im defenitely neither network nor database engineer but maybe you are bottlenecked by how fast you can write to the database by either your network or the hosts performance. I assume your database does live on a SSD?
Ist that for eauch job?
No, for all jobs. 25-35 minutes.
I assume your database does live on a SSD?
Yes. And I even get 250MB/s on that share from a different machine.
I assume the traffic/file write is not the issue.
deadlinecommand takes 3-5 seconds each time to start and send the job.
I assume that app is just slow… (And so far I do not see any option to enable a log file/verbose messages to know what’s taking that much time)
Deadlinecommand should be very fast submitting a job. Are you submitting the job with the Auxiliary files with the job? It may also count on the time depending on the scene.
I did run a a test job submission with a CommandLine plugin as a batch script which prints the time before and after running the job submission command:
time /t
"C:\Program Files\Thinkbox\Deadline10\bin\deadlinecommand.exe" "651afb84926f1ff702dd9a40_jobInfo.job" "651afb84926f1ff702dd9a40_pluginInfo.job"
time /t
The above works in windows, it didn’t take a second to submit a job to Deadline.
Can you try submitting a normal commandline job using the below jobinfo and plugininfo files.
jobinfo.job
UserName=user
Region=
Frames=1-10
ChunkSize=2
Denylist=
OverrideTaskExtraInfoNames=False
MachineName=YVR28-T4QRK6TUW
Plugin=CommandLine
EventOptIns=
plugininfo.job
SingleFramesOnly=False
Executable=C:\Windows\System32\cmd.exe
Arguments=ping www.google.come
StartupDirectory=
ShellExecute=True
Shell=cmd
This should show the submission time with a simple commandline job. Also if the deadlinecommand is taking time, it should show in pulling the -pools from the Database.
deadlinecommand.exe -pools
2.8s for a submission.
1s for -pools
TIME: 19:51:23,07
C:\Windows>“C:\Program Files\Thinkbox\Deadline10\bin\deadlinecommand” -SubmitCommandLineJob -executable \test-fileserver\Projects_RenderTests\batchBenchmark\render.bat -frames 1-100 -chunksize 5 -prop BatchName=test -prop UserName=myuser
Submitting to Repository: R:/DeadlineRepository10
Submission Contains No Auxiliary Files.
Result=Success
JobID=651b112d021b5bb9bb6aa1ce
The job was submitted successfully.TIME: 19:51:25,85
C:\Windows>“C:\Program Files\Thinkbox\Deadline10\bin\deadlinecommand” -pools
noneTIME: 19:51:26,85
Just to add a datapoint: I’m experiencing the same as times as @Heide of 2.5-4 seconds to submit and just under one second for -pools.