Heya,
I have a piece of python code when the postjobscript is set via python code causes the final frame of the render as well as the post job script itself to ‘never’ be rendered (at least I haven’t found a way for them to render even when the jobs prio is higher than anything else on the farm).
For context, the job is set to a pool that only has disabled workers in it, the python code is then ran and it enables one of the workers in this pool, sets the postscript job and then activated the worker. The worker then runs the job but stops at the last frame of the job before diverting to any other task it can. The intention is that when the post job script is ran, it will disable the worker once more and this works but only if the post job script is manually set. If the job is then manually failed, its status is listed as corrupted. Can anyone help out with this?
Thanks for reaching out. The Python code you are running to change the job’s task list might be making your job corrupted, this forums post tell about the cause of corruption. It is not recommended to change the job tasks at the render time.
What run that Python script on the job? like, is it a prejob script or OnJobSubmitted?
I see you are using pools and have disabled Workers, can’t you use groups instead?
Cheers for the reply. The correct plugin is there so not sure why it would be using different procedures specifically for the last frame and only when the inputs are made by code and not by hand. Cheers for the insight though.
Hi @zainali,
Thanks for the reply. For the record, my code does not assign of change jobs at all, that is left up to deadline to do.
The exception I think your alluding to here would be adding the postJobScript but I don’t understand how that would be any different to adding it via the modify job properties (which does work fine). That being said, all of this occurs before any task is being rendered on the job. Also, the job does not corrupt until is is manually failed when the last two tasks don’t run.
The reason this (kind of) has to be run at on the job as because I don’t currently want to interface with Maya and was hoping to keep everything within Deadline for the time being. So a job doesn’t know it needs to run the code until I tell it to. It runs via the right click menu on a job that is in queue.
I haven’t really explored using groups as for my use they didn’t seem too different from pools. The code is intended to enable and disable parent and child workers depending on the resource needs of an artist. Child workers are parents that are split in two with CPU and GPU affinity and so if a child is running but the parent is needed. the children need to be disabled for the parent to be enabled to then be able to run on the job. Parent enabling and child disabling is ran via a post task script on the children jobs that disable the child upon completion of the task and enable the parent if all children are disabled (definitely a render time change but this isn’t implemented yet so we’ll see). This process is reversed with the job completion script on the parents job that re-enables the children and disables the parent. Rather then enabling disabling would you recommend moving workers in and out of groups, something else entirely, or is there a better way to achieve all of this?
We know setting the postjob scripts via the API can corrupt jobs, even while setting the script in the Monitor works without issue. The ‘is this job corrupted check’ doesn’t run until the job state changes. So if the job corrupts after it gets marked failed it’s likely the job was corrupted earlier. Instead, set the postjob script in the job creation.
Given this info, setting the script on job creation isn’t workable. Instead, you could have that right-click script duplicate the existing job, add the script and submit the new job. Then delete the old script-less job. That’s also not a great solution, so a little more context on why splitting the Workers needs to be done might be helpful. I assume you’ve looked at concurrent tasks and not had those work out?
For the difference between groups and pools; in short pools are for job priority (both start with P ) and groups are for hardware and software limitations. In long, this blog post breaks down the three ways to manage/limit jobs. So you’d have job pools to make sure the most important jobs get done first, and groups to separate the machines with and without GPUs. For this setup you could put the parent and child Workers into their own groups.
I think the issue you’re having is due to setting the post-task script via the API and corrupting the job, and disabling a Worker that’s currently running will have it fail the task it’s working on.
If I could get a job archive where you’re seeing this behaviour I’d be able to better confirm what’s going on. If you’re not comfortable putting that online, you can send that file in via the ticket system.
As for a better solution, Deadline’s really not built to do this sort of dynamic enabling/disabling of Workers while they’re active. Is it just Maya+Redshift getting run on this farm? What makes a job appropriate for the child Workers versus the parent workers?
Hi @Justin_B and thank you so much for the response.
Great to know about that, just curious but is there a way to run Monitor scripts with the API similar to remote worker commands (a remote repository command)?
I am completely ignorant on many aspects of deadline and no one here has really touched this sort of stuff so I’m mostly going in blind. If I get desperate I will look into into the job copying but obviously would like to keep this as simple as I can and work within what deadline is meant to do (aware of the irony as I try to force Deadline to do things it’s not intended for!). The splitting is essentially to allow our workers to do both more computational intensive tasks as well as less intensive tasks without getting caught up on those smaller tasks. I suspect this current route was taken as a bit of a band aid fix as no-one has properly investigated concurrent tasks, pools, pool priorities, and secondary pools and currently we lack the knowledge to design such a system right now.
Thanks, thats pretty much my understanding now as well. As previously mentioned these are not being used effectively right now with basically groups being used for location (workers render to separate servers which take time to sync so people generally render with their local machines to their local server using these groups), and pools being a bit of a mess. I don’t think using groups for parent and children would work because whats preventing both the parent and child from taking on jobs and overloading the worker? That is what the enabling and disabling is intended to do, but how do I ensure only the parent OR the child to do jobs based on the needs of the farm?
Will for sure look into sending in an archive if there isn’t a better way but I think we know what the issue is anyhow. Jobs are from maya, redshift, houdini, arnold, nuke.
When I was talking about a better solution that would be to having the farm (more specifically the more powerful workers) efficiently handle more resource intensive tasks as well as less resource intensive tasks when needed. I honestly would suspect that the approach itself is wrong which tends to happen when ignorant people come up with a plan lol. If you have any suggestions for guidance on a better pool design that would be awesome. We basically have 3 tiers, a couple 256GB ram machines that can be split into 2x128GB or 4x64GB or a mixture of the two (I know RAM isn’t split like this but we werent sure how else to “give” larger jobs access to more ram), a couple 128GB machines that can be split into two 64GB machines and then a few 64 GB machines. Ideally, when larger jobs come up in the queue it will ensure that the children are free and the 128GB workers to take the job. The splitting is achieved with GPU and CPU affinities to ensure they’re not overlapping on hardware, and at the moment we manually enable and disable the workers as needed.
I’d look into concurrent tasks. Truthfully, set a machine to have the big worker only and modify one of your existing jobs to have 2 concurrent tasks and see how it goes. You might be just fine without this extra layer of dynamic enabling/disabling fun.
Deadline doesn’t have a mechanism for this to be truthful. I’d do the test with two concurrent tasks before thinking harder on building around it.
What you’d love to have is slot based scheduling, where you can say how many slots a job needs and how many slots a Worker’s got to work with. I think Son of Renderman has that? I’m majorly mis-remembering. Deadline in just doesn’t have tooling to account for that.
What you could do is make 3 groups to match your tiers and submit jobs to the appropriately sized group. However that doesn’t account for big machines doing small jobs, which would work. But Workers won’t dequeue tasks outside of the group.
A couple folks have taken a crack at this in the past and I haven’t found a good way to pull this off.
Thank you for replying to my whims. What’s the difference between concurrent tasks and multiple worker instances on the one comp? Ill pass it by the team and see what they say. As you already saw, I’m looking into using a separate python task to draw the workers I want away so Ill see how these approaches turn out.
Cheers,
Stephen
If a job has concurrent tasks set, the Worker will start multiple render sandboxes and work on multiple tasks at a time. You can set an upper limit of concurrent tasks per Worker as well, so you can cap how many tasks your smaller machines can take on.
The question did pertain to the difference between concurrent tasks and multiple worker instances. From my understanding it appears that they are very similar in many aspects. Anyhow, I appreciate the replies.
) and groups are for hardware and software limitations.