Worker reports Rendering, Task reports "Waiting to Start" - how to get this info via scripting?

jason_d_macleod · February 17, 2021, 3:02am

Hi,

I am trying to write a utility to detect and manage a situation with Workers entering a hanging state. One sign that a machine has entered this state is that the Worker will report being in a “rendering” state, but the Task view lists tasks as being in a “Waiting to Start” state. I am looking for how to retrieve these pieces of information through the Deadline RepositoryUtils scripting interface. The attached image is indicative of what the UI displays when a machine is in a hanging state - machine ETVENC-RN153 is currently reporting that it is rendering, but the task view reveals that it is hanging.

How can I get access to this information in as close to the way that the Monitor UI is accessing it?

Thanks,

Jason

Bobo · February 17, 2021, 5:05am

A task has three properties relevant here:

.TaskStatus is the top level indicator what the Task is doing. It can be “Queued”, “Failed”, “Pending”…
If .TaskStatus is “Rendering”, it means a Worker picked up the Job and the Task, and is in some stage of processing it.

At this point, with .TaskStatus still equal to “Rendering”, the other two properties come into play:

If .WaitingToStart is True and .TaskIsStarted is False, then the Task is “Waiting To Start”
If .WaitingToStart is False and .TaskIsStarted is True, then the Task is “Starting Up”
If .WatingToStart is False and .TaskIsStarted is False, and .TaskStatus is still “Rendering”, then it is actually in the “Rendering” phase.

Then it switches to .TaskStatus of “Completed” if everything went well…

jason_d_macleod · February 18, 2021, 3:07am

So this was trickier than I initially thought - thanks to Justin at Thinkbox Support for pointing me to the CallDeadlineCommand code present in the Houdini Submitter code example that ships with Deadline.

I ended up having to open a connection to the Worker Logs one-by-one and look for specific strings in the log that reveal the hanging behavior. The CallDeadlineCommand example was invaluable.

Thanks,

Jason