bad module import = disabled farm

LaszloSebo · February 16, 2015, 11:45pm

We have some python post-job tasks, and today a bad library import slipped in (missing dll for a dependency).

Sadly, it took out our entire farm due to the python sessions not being sandboxed. Every slave that picked it up threw a c++ runtime error in the slave application itself:

Faulting application name: deadlineslave.exe, version: 6.2.1.49, time stamp: 0x54380381
Faulting module name: ntdll.dll, version: 6.1.7601.18247, time stamp: 0x521eaf24
Exception code: 0xc0000374
Fault offset: 0x00000000000c4102
Faulting process id: 0x15cc
Faulting application start time: 0x01d04a35a64ee6a5
Faulting application path: C:\Program Files\Thinkbox\Deadline6\bin\deadlineslave.exe
Faulting module path: C:\Windows\SYSTEM32\ntdll.dll
Report Id: 2c310e70-b632-11e4-93d9-0cc47a0a744b

Needless to say, python scripting errors being able to take down the entire farm (instead of just erroring the affected tasks) is a major problem

LaszloSebo · February 16, 2015, 11:46pm

Is it possible to handle these popups with the popup handler of deadline btw? The slave gui is hung while the popup is app, so i’m not sure. As soon as we click OK manually, the slave carries on.

rrussell · February 17, 2015, 1:48pm

Sandboxing is coming in Deadline 8.0.

We can’t handle these popups with the popup handler because it only watches the rendering process. In Deadline 8 though, the main slave process maintains a heartbeat with the sandboxed rendering process, so it should be able to detect when it crashes or hangs and handle it appropriately.

Cheers,
Ryan