AWS Thinkbox Discussion Forums

Slaves not being shut down despite limit setting

And another Slave issue :wink:
Not sure if this is a bug, actually. We do have 20 Deadline Slave licenses right now and 22 machines powered on.
Two of the Slaves just stay powered on in ‘Idle’ state despite that limit.
PLUS: in the Limits setup i defined an overall ‘deadline’ limit of 20 for Slaves which is assigned to the currently rendering job. Shouldn’t either (or both) of that cause the Slaves to be powered down? Or am i misunderstanding something basic here?

Cheers,
Holger

UPDATE:
Not only are those machines not being shut down. But actually one of them was even powered back on through WakeOnLan.
Also i saw that in the ‘License’ column of the Slaves Panel it actually shows the message that the maximum number of licenses has been reached. Wouldn’t this be a reason for it to shut down the machine? Or is actually the reason this specific Slave can’t obtain a license the reason why it can’t be shut down as it can’t be controlled by Deadline due to a missing license?
I also think i found out that when you assign a predefined Limit (in this case the one from my previous post) when the job is already rendering that the limit stubs are only assigned to Slaves once they start the next task. It seems it’s not doing this while a task on a machine is already in progress. And maybe that was the reason it didn’t shut down the machine or powered it back on as it thought there are still stubs available?
For now, setting the machine limit of the job to 20 solved the issue.

Cheers,
Holger

Since slaves only require a license, it is possible for them to run without one (they’ll just never pick up any jobs). So a slave won’t shut itself down if it can’t get a license.

That being said, this shouldn’t prevent power management from shutting the slave down, since that is simply based on how long the slave has been idle. I have to ask, but are these 2 machines that didn’t shut down in the power management group? Can you post the Pulse log from the session where it should have been shutting down these slaves?

Regarding limits, the slaves only check the new limit maximum between tasks. In other words, they won’t cancel their current task to bring the stub count down to the the new maximum. They’ll wait until they finish their current task before dropping the job.

Cheers,
Ryan

Wouldn’t it make sense to implement a check for this? I’m trying a workaround right now where i set up a Deadline license limit in the Limits settings that is now mandatory to be set by all the artists when submitting a job.
But i’m wondering if it wouldn’t make much more sense to implement this directly as i imagine Pulse has methods of finding out how many licenses are available and automatically deal with that. With the workaround we’ll always have to adjust it when new licenses are being added.

I know you were going to ask this :wink: So i made sure and double-checked that they are indeed in the group.

Attached. The affected machines’ names are cell-ws-13 and cell-ws-17.

Cheers,
Holger
deadlinepulse-cell-temp-01-2014-10-22-0000.zip (1020 KB)

Thanks for the log! According to the log, Idle Shutdown is disabled for the “all” group:

That would explain why the slaves aren’t shut down.

For the licensing, why not simply put as many slaves in your power management group as the number of licenses you have? Then you shouldn’t have to worry about creating a Limit for it.

Cheers,
Ryan

Bummer! Wonder how that happened. Was absolutely sure that this was set. Sorry for that mistake.

Regarding the licensing, the way our setup works this will actually quite a bit of maintenance overhead for me as the machines will change from time to time - actually sometimes few times per week. So every time this happens we’d have to adjust the group members of the “all” group. That doesn’t seem practical to me. May i ask why you’re hesitant regarding an implementation into Deadline directly?

Cheers,
Holger

Pulse currently doesn’t have a means for checking how many Deadline licenses are available. It’s something that can be added to the wishlist, but it’s unlikely to be implemented in the short term, since it will involve updating some core licensing code.

Enforcing a limit group is probably your best solution for now. To make things easier, you could write an event plugin that handles the OnJobSubmitted event:
thinkboxsoftware.com/deadlin … eventssdk/

For every job that comes in, you can use the event plugin to automatically add your limit to the job’s list of limits if it isn’t already specified. I’ve attached an example event plugin that you can unzip to \your\repository\custom\events. After that, open the Monitor and select Tools -> Configure Event Plugins while in super user mode. Select the SetJobLimit event from the list on the left, and enter in the Limit name you want to automatically set. If you wanted to specify multiple limits, just separate them with a comma, like this:

limit_01,limit_02,limit_03

After saving the settings, every job will have the limit(s) specified, in addition to any other limits that were set for the job before submission.

Cheers,
Ryan
SetJobLimit.zip (1.02 KB)

hi Ryan,

that indeed sounds like a good solution. I wasn’t aware of the SetJobLimit event yet. Thanks for the script!
My problem is i can’t find it. There’s only a menu called “Configure Events…” but not “Configure Event Plugins”. And in the “Configure Events” there’s no SetJobLimit Event item on the left side, only Draft, Ftrack, Puppet, Salt and Shotgun.

Cheers,
Holger

The SetJobLimit event is a custom event I just whipped up now. I attached it to my previous post. Did you unzip it to \your\repository\custom\events? You should end up with a folder called SetJobLimit in the “events” folder, and inside the SetJobLimit folder should be 3 files (dlinit, param, and py files).

Cheers,
Ryan

Also, sorry, I meant “Configure Events”, not “Configure Event Plugins”.

stil can’t find it :wink:

Strange …

Can you post a screenshot of the SetJobLimit folder in your repository? Also, make sure we can see the full directory path as well.

Thanks!

here you go.

I’m not seeing the SetJobLimit folder here. Where did you unzip it to? Was it the events folder in the custom folder? If so, can you post a screenshot of that?

Thanks!

uh!.. stupid me again!..
Just fixed it and now it works like a charm!
Thanks a lot!
Now i just need to get into Python programming for Deadline to add the option to just browse the existing limits instead of manually typing them.

Cheers,
Holger

Privacy | Site terms | Cookie preferences