AWS Thinkbox Discussion Forums

Submit jobs through mongo

Is it possible to create a job by submitting documents to mongo?
I have the standalone python api setup but that is rather slow.
When I tried submitting about 10,000 jobs at once it took 15 minutes to finish creating the jobs.

Technically it’s possible, but not supported. Any direct interaction with the database isn’t supported, as we can change the schema without warning.

You may have a better time with SubmitMultipleJobs, but 10,000 is a lot of jobs to submit at once.

Just wanted to point out that 15 minutes * 60 seconds is 900 seconds. It implies that you had over 11 jobs per second submission rate, which is unlikely, even with the SubmitMultipleJobs approach. I would have expected you to report several hours… :wink:

We can submit 10’s of thousands of automated jobs per day with our current pipeline using celery.
It takes less than a minute to submit this many tasks there so this feels really slow in comparison.

Ok, I decided to run some tests to see how fast the Multi Job Submission runs. I generated a text file to submit 1000 jobs. Note that all the jobs used the SceneFile= option to reference an already saved scene file for the 3D application, so no auxiliary files had to be copied to the Repository storage, thus ensuring the fastest performance. Each job was set to render a single Task, but later I tested 100 Tasks per Job and it made no difference.

Executing this from the deadlinecommand on Windows took exactly 50 seconds - in the first 23 seconds deadlinecommand did not respond, I assume it was parsing the control file. The other 37 seconds it reported the job submissions in a rather fast scrolling report. So 10,000 should submit in 500 seconds, which is of course about 8.3 minutes. With more complex cases I can believe it could take 15 minutes.

The above test was performed using a Direct Connection to the Repository.

When I connected the client to the Remote Connection Server, the time to submit 1000 Jobs went up to 247 seconds… Not optimal.

Not that it helps you get more speed, it just shows that your numbers are plausible.

Privacy | Site terms | Cookie preferences