AWS Thinkbox Discussion Forums

Understanding power management commands

Sorry for delay [back from vacation].
Hmm…I think it might be best if you open up a support ticket so one of our support team can remote in and work with you on this to resolve it.
support.thinkboxsoftware.com/

Hello,

Coming back on this issue after several month and our recent 7.2 upgrade.

I finally understood that the shutdown command is sent to the slave and not the pulse !

Is there any way to change this and send the shutdown command via pulse ?

For some reasons, we cannot use the shutdown option, or execute a script on the nodes.
Sometimes the shutdown does not work because of the node system state : over-swaping, kernel panic, hardware issue …
IPMI is the best way to unsure the power state of the nodes.

Thanks,
Pierre

Hi Pierre,
We have this already logged as a feature request/wishlist item for all PM commands (IPMI, PM script, etc) to be re-directed and executed via the machine running the Primary Pulse. Hopefully, we will get to this feature later this year.
In the meantime, could you use a right-click “slave” script in your slave panel, which then sends a command to your primary “Pulse” machine, which then send out the IPMI/script command? Would that work for now? If so, you should be able to use our API to achieve this already in 7.2. LMK, if you want me to try and put something together for you.

Hello Mike,

Is it in the v8 beta already ?

Yes, having the possibility to put a new script in the slave menu could be very useful.
I’m not sure how to do that, any help would be appreciated :stuck_out_tongue:

Cheers,
Pierre

Hi Pierre,

Please find attached a zip file containing 2 scripts - 1 x slave script and 1 x pulse script. Please have a read of the README.md file which explains how to install and configure these IPMI scripts to work in your environment. These scripts haven’t been tested much, so feedback would be good here and we will upload these to our GitHub site for others to use.

IPMI_Scripts.zip (4.99 KB)

Thanks a lot Mike !

First tries, a little bug :
SlaveIPMI.py line 96 :

elif IpmiCommand == "power off": IpmiCommand = "off"
should be

elif command == "power off": IpmiCommand = "off"

I’ll let you know if anything valuable appears.

Cheers,
Pierre

Good catch! Now fixed in our internal version.

Hello again,

going back to this.

I have adapted your scripts for linux :
SlaveIPMI.py
line 164 : CliCommand = "ExecuteScriptNoGui"
instead of

CliCommand = "-ExecuteScript"

Is this correct ?

And then, in PulseIPMI.py

if process.StandardOutput != None: output = process.StandardOutput.ReadToEnd() print "%s" % output
returns nothing, it seem the ipmitool is not called.

Something must be wrong there, but I don’t know where :blush:

Hi,

CliCommand = "ExecuteScriptNoGui"

Sure, that makes perfect sense if you’re running your nodes including Pulse, as headless. I’ll update our GitHub version.

re: PulseIPMI.py not working

Can you try executing the py script manually using this syntax on your Pulse machine (which should have your IPMItool apps installed on it or sym-linked). Also, the Pulse machine needs to have ‘line-of-sight’ to your mgmt vlan (ie: IPMI commands work from this machine currently):

"{local path to}/DeadlineCommand" -ExecuteScriptNoGui "{your_repo}/custom/scripts/Pulse/PulseIPMI.py" {SLAVE_IP} {IPMI_COMMAND}

where:
{local path to} = the path to where Deadline is installed on this machine (might be sym-linked)
{your repo} = your repo path
{SLAVE_IP} = incoming IP address
{IPMI_COMMAND} = command you wish to execute (options are: “on”, “soft”, “off” or “reset”)

and in the PulseIPMI.py script file the following settings are configured correctly:

USERNAME = your username
PASSWORD = your password
IPMIExec = “C:/Program Files (x86)/sourceforge/ipmiutil/ipmiutil.exe” #replace with local path to “…/ipmitool” depending on your hardware being used
DataIP = “10.2.1.” #incoming IP address range
MgmtIP = “10.2.16.” #outgoing mgmt vlan IP address range

If that doesn’t work, try commenting out lines: #68-73 and try again.

If none of the above still doesn’t work, try changing the line (with lines: #68-73 still commented out):

process = ProcessUtils.SpawnProcess( IPMIExec, args, None, ProcessWindowStyle.Hidden, True )

to:

process = ProcessUtils.SpawnProcess( IPMIExec, args )

Hello again,

so now I managed to make the PulseIPMI.py work when using it on the pulse machine :

But when launching it through deadlinemonitor and SlaveIPMI.py, I have this in the launcher log on the pulse machine :

Any clue ?

Hi,

So, it looks very much like I missed an additional argument which is required by our API function here. See attached updated SlaveIPMI.py script for you. Essentially, we need to prefix the command we send remotely to Pulse with “Execute” or “ExecuteNoWait” which explains the argument exception here. For future reference, here is the API docs on the “SlaveUtils.SendRemoteCommand”: docs.thinkboxsoftware.com/produc … cc42c819b1

If you look at the attached file, take a look at the very bottom of the file for the change. Hopefully, this will then work for you?

SlaveIPMI.py.zip (2 KB)

Excellent, it works like a charm now.

Thank you very much for your help, I’ve learned quite a bit in the process :slight_smile:

Cheers,
Pierre

Cool! I’m glad it’s now working for you!

Hi there, coming back to this.
We are on the migration to deadline 8 process, I was wondering if you guys had addressed this issue.

Thanks,
Pierre

Hey Pierre,
Deadline 8.1 (currently in BETA) has added the ability for all “Remote Commands” to be redirected via Pulse. However, we haven’t yet got to the implementation of IPMI commands into the PM controls at the Slave level. I will add your feedback to the internal ticket we already have logged for this feature. So, it might be worth you testing the next 8.1 beta build for this feature and all the other new stuff we have added recently? BETA access for existing customers via email for signed NDA to “beta” [at] “thinkboxsoftware” [dot] com.
Cheers,
Mike

Hello,

I cannot find anything about this subject on the latest 8.1 beta build.

Is there something to test yet ?

Cheers,
Pierre

Hey Pierre! Do you mean the IPMI commands or the Pulse redirect? I do believe the Pulse redirect was added many months ago, but IPMI is still being pushed out by other higher priority features and doesn’t look like it will make it into 8.1 due to unexpected roadmap changes.

Hey Pierre,

Edwin beat me to it. Unfortunately, the IPMI functionality that Mike was referring to has been bumped down the priority list due to higher priority issues coming up. We still have it logged as a feature request though, and we definitely want to see this functionality in a future version of Deadline.

It sounds like you have a working solution at the moment though. I just want to confirm that your current solution will continue to work for you until we can properly support this in Deadline.

Cheers,
Ryan

Hello again,

coming back to this after our migration to deadline 8.
I’m afraid I should have replied to Ryan…

My recent test show that in deadline 8, it is still the same, only the waking up is done directly by pulse, when shutdown and restart are sent to the node launcher

So if I’m correct we can’t plug our IPMI scripts for all the power management tasks except node startup.

Am I correct ?

Cheers,
Pierre

That is correct. We have not forgotten about it though!

Privacy | Site terms | Cookie preferences