In the slave logs, this line appears just before a node decides its in a stalled state:
Info Thread - Could not check if Pulse is running because:
Note: These lines only show up in the .log files saved on the nodes, they dont show up in the monitor ui.
We are seeing a pile of nuke jobs where the out of memory isn’t making its way back to the stdout logs in deadline.
When the following event is run: OnSlaveStalledCallback
is it running on the slave that it occurred on or is it run on the pulse node?
Secondly, is there a slave / repo tool, to get the last slave log used?
Yes, can probably figure this out from the file system, if its running on the actual node that has stalled.
It looks like you have a few issues here. The note about pulse is just a check that a slave does on occasion when trying to see if it should be doing a house cleaning check, and since they are a slave log issue, unconnected to whether a job is running, and as it’s in the slave thread, not the render thread.
I can’t really speak to the nuke out of memory issue, but if you want to get us some logging or reports on that we can take a closer look.
The OnSlaveStalled is run on the machine that marks a slave as stalled.
The only tool in the Monitor to grab a log from the slave is the streaming slave log, but you can connect to a slave to check the application log folder for the most recent files.
As for marking a slave stalled, I don’t see a deadlinecommand option for that, so I don’t think there is an option for that.
