Deadline 5 Idle Shutdown not working?

Hi there,

I’m having problems with the Idle Shutdown settings.

I set everything up (as I did in Deadline 4.x) according to the manual, but the slaves just don’t react to the Shutdown commands.

Strangely, everything works as it should when invoked manually (i.e. selecting slaves and hitting shutdown machines) and then waking them on lan this way, but it doesn’t work automatically using the Power Manager.

This is what the Power Management log says:

May 23/11 11:41:07 loocas RAMMSTEIN (STUDIO\loocas): Idle Shutdown: shut down machines ‘Vfx02’ for being idle longer than 1 minutes (RAMMSTEIN)

Pulse is running, of course.

Any thoughts?

Can you check the Launcher log on the machine that is being told to shutdown (ie: Vfx02 in this case) to see if the Launcher received the shutdown command? You can use the timestamp from the power management log to figure out where you should be looking in the launcher log.

Pulse uses the same mechanism to shutdown machines that the Monitor does when doing it manually, so that it’s strange that it would work from one and not the other. Just to confirm, do you have Idle Shutdown in debug mode? That’s the only I can think of that you would get this log message without the slave actually being shutdown.

Cheers,

  • Ryan

Will check the log.

As for the debug mode, nope, I know the debug works without actually shutting down the machines. I double checked the documentation for that and further on I setup everything as I did in Deadline 4 (which worked perfectly fine).

The launcher log on the slaves says:

Why? What does it mean?

That seems to indicate that communication between the Launcher and the Slave running on the same machine is being blocked. I’m not sure why this would happen, especially if it works when done manually from the Monitor…

We’ll see if we can reproduce here.

Hey Lukas,
Since moving from v4 to v5, have you upgraded to Win7 OS or indeed changed your firewall settings? Deadline needs firewall settings to be placed in the “inbound” section, which is a different config to WinXP days…
Also, have you changed to using IPv6 instead of IPv4 recently?
Just wondering if any of these things have changed at your end and hence now affecting Deadline. I’ll see if I can replicate this as well…
Cheers,
Mike

Hey guys,

all the machines have always been Windows 7 pro x64 even when running Deadline 4.x, all firewalls are off, every machine is a part of a domain, everything uses IPv4, the server is a dedicated Windows Server 2008 x64, all works, except for this. :cry:

But the uber-weirdest thing is, one machine, the VFX02, actually shut down when neede! WTF! All other machines I added to the group later do not shut down when instructed.

Just tested it here and it worked fine. That’s interesting that it works on one of your machines, but not the others. That would point to a system configuration issue with those other machines, but since your firewalls are off, I’m not sure what else would block that communication. Do you have any antivirus software installed on the machines?

I am surprised too! :smiley: That’s why I’m posting here. All worked flawlessly fine before on Deadline 4.x

As for antivirus sw, only on the server (running pulse, deadline repository etc…) I have NOD32. But it doesn’t seem to be blocking anything. It shouldn’t, really…

Ok, I just tried to manually shut down all the slaves (manually = via the Monitor) and removed the Power Management group and re-created it from scratch again.

Will test this if it makes any difference.

Ok, another weird behavior. If I turn off the physical machines via the Monitor, why the hell do the machines then report as “Stalled”?!

Ok, I’m honestly starting to think that Deadline 5 is “broken”. :cry:

It just doesn’t work as flawlessly as Deadline 4 used to for me. Some machines start the slaves automatically, some won’t. Then they “actively refuse” to perform remote commands and I have to manually start the Deadline service on some of them. Some are fine, I shall add.

Well, let’s see if they get shutdown after the idle period I newly setup. If they won’t, I don’t know what else to do but to revert back to Deadline 4.x :cry: which’d be a huge waste of money.

Hmm, it seems that manually (from the Monitor) shutting down the slaves and resuming them again while recreating the Power Management group for them all, did the job. The machines are now all offline due to the Power Management rules. I’ll test auto-resume tomorrow (way too late here now). :slight_smile:

Ah, man! The machines just refuse to wake up using the Power Manager. :frowning:

I just had to manually (from the Monitor) wake them up and see if the slaves start properly.

I never had such issues with Deadline 4.x, but hopefully after this mumbo-jumbo all will start working properly again. I certainly don’t want to start paying higher electricity bills, than before…

The slaves won’t start up automatically (as a service) for some reason. I had to manually jump them up right now on all my woken up slaves!

I’ll try to change the user that starts up the services to a local one (had it set to the domain admin), if that makes any difference.

Just to confirm, did the machines wake up, but the slave didn’t start, or did the machines not wake up at all?

Also, when you installed Deadline 5, did you reinstall the Deadline 5 client software on each node? I’m wondering if there is a version mismatch that is causing your headaches…

Cheers,

  • Ryan

I completely deinstalled Deadline 4 (as per recommendation by the manual) and installed Deadline 5 on all machines, incl. the server.

As for the waking up. None of the machines woke up from the Power Management, I had to wake them up manually (via the Monitor), but then the slave service didn’t start, so I went to all of the machines one by one and started the service manually (also I changed the account for controlling the service, if that makes any difference).

Another note, Deadline isn’t sending any e-mails what so ever even though the SMTP server is set up and I filled in all the e-mails.

I’m having tons of problems with Deadline 5, for some reason. :cry:

I’m so sorry to hear that you’re having all these issues. Would you be willing to email your powerManagementSettings.xml and networkSettings.xml file settings to us? We can take a look at your power management and email settings to see if anything stands out. For confidentiality, you can email them directly to support (at) thinkboxsoftware (dot) com.

Cheers,

  • Ryan

Thank you very much, I appretiate your support!

Ok, I finally solved the problem. It was my server’s NICs acting up!

If anyone is interested in more details, I bloged about it here:

http://blog.duber.cz/misc/deadlines-power-management-not-waking-slaves-properly

By the way, thank you very much, Thinkbox support! Without you I wouldn’t have been able to identify the problem in the first place! You rock!