Python crash case #1

LaszloSebo · August 19, 2013, 9:12pm

Had about 8-10 machines hanging this morning (30% of the deadline farm) with various python related errors.

Based on logs and event viewer entries, i tried to find some pattern, got two types of errors so far (some had no event viewer entry, or i couldnt find them).

Ill log the other case as a separate bug thread.

Last log lines:

2013-08-12 17:13:42: 0: Plugin rendering frame(s): 1076-1090
2013-08-12 17:13:42: 0: INFO: Executing pre task script C:\Users\scanlinevfx\AppData\Local\Thinkbox\Deadline6\slave\LAPRO0318\jobsData\prestart.py

Event viewer has 2 application errors logged for this. First one:

Log Name: Application
Source: .NET Runtime
Date: 8/12/2013 5:13:42 PM
Event ID: 1026
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: lapro0318
Description:
Application: deadlineslave.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.AccessViolationException
Stack:
at Python.Runtime.Runtime.PyType_GenericAlloc(IntPtr, Int32)
at Python.Runtime.CLRObject…ctor(System.Object, IntPtr)
at Python.Runtime.CLRObject.GetInstance(System.Object)
at Deadline.Plugins.ScriptPlugin.a(System.Object)
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
at System.Threading.ThreadHelper.ThreadStart(System.Object)

Event Xml:

1026
2
0
0x80000000000000

3073
Application
lapro0318

Application: deadlineslave.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.AccessViolationException
Stack:
at Python.Runtime.Runtime.PyType_GenericAlloc(IntPtr, Int32)
at Python.Runtime.CLRObject…ctor(System.Object, IntPtr)
at Python.Runtime.CLRObject.GetInstance(System.Object)
at Deadline.Plugins.ScriptPlugin.a(System.Object)
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
at System.Threading.ThreadHelper.ThreadStart(System.Object)

Second one right after the first:

Log Name: Application
Source: Application Error
Date: 8/12/2013 5:13:43 PM
Event ID: 1000
Task Category: (100)
Level: Error
Keywords: Classic
User: N/A
Computer: lapro0318
Description:
Faulting application name: deadlineslave.exe, version: 6.1.0.52215, time stamp: 0x51fa5cf1
Faulting module name: python26.DLL, version: 2.6.7150.1013, time stamp: 0x4e00e564
Exception code: 0xc0000005
Fault offset: 0x00000000000b9509
Faulting process id: 0xb0c
Faulting application start time: 0x01ce92faf578b0bb
Faulting application path: C:\Program Files\Thinkbox\Deadline6\bin\deadlineslave.exe
Faulting module path: C:\Program Files\Thinkbox\Deadline6\bin\python26.DLL
Report Id: 36497ae4-03ad-11e3-aae4-003048c6b711
Event Xml:

1000
2
100
0x80000000000000

3074
Application
lapro0318

deadlineslave.exe
6.1.0.52215
51fa5cf1
python26.DLL
2.6.7150.1013
4e00e564
c0000005
00000000000b9509
b0c
01ce92faf578b0bb
C:\Program Files\Thinkbox\Deadline6\bin\deadlineslave.exe
C:\Program Files\Thinkbox\Deadline6\bin\python26.DLL
36497ae4-03ad-11e3-aae4-003048c6b711

rrussell · August 20, 2013, 2:33pm

We have an update on the Python crashing issue:
viewtopic.php?f=86&t=10028&p=43407#p43407

LaszloSebo · August 22, 2013, 8:39pm

With beta2, we are getting considerably less crashed nodes.

However, im sad to say i just got a python crash again…

Similar event crash to the one in this thread already (with PyType_GenericAlloc):

Log Name: Application
Source: .NET Runtime
Date: 8/22/2013 10:24:33 AM
Event ID: 1026
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: lapro0323
Description:
Application: deadlineslave.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.AccessViolationException
Stack:
at Python.Runtime.Runtime.PyType_GenericAlloc(IntPtr, Int32)
at Python.Runtime.CLRObject…ctor(System.Object, IntPtr)
at Python.Runtime.CLRObject.GetInstance(System.Object)
at Deadline.Plugins.ScriptPlugin.a(System.Object)
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
at System.Threading.ThreadHelper.ThreadStart(System.Object)

Event Xml:

1026
2
0
0x80000000000000

3518
Application
lapro0323

Application: deadlineslave.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.AccessViolationException
Stack:
at Python.Runtime.Runtime.PyType_GenericAlloc(IntPtr, Int32)
at Python.Runtime.CLRObject…ctor(System.Object, IntPtr)
at Python.Runtime.CLRObject.GetInstance(System.Object)
at Deadline.Plugins.ScriptPlugin.a(System.Object)
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
at System.Threading.ThreadHelper.ThreadStart(System.Object)

Running the prestart python script:

2013-08-22 10:24:33: 0: INFO: Starting monitored managed process MayaBatch
2013-08-22 10:24:33: 0: INFO: Stdout Handling Enabled: True
2013-08-22 10:24:33: 0: INFO: Popup Handling Enabled: True
2013-08-22 10:24:33: 0: INFO: Using Process Tree: True
2013-08-22 10:24:33: 0: INFO: Hiding DOS Window: True
2013-08-22 10:24:33: 0: INFO: Creating New Console: False
2013-08-22 10:24:33: 0: INFO: Enforcing 64 bit build of Maya
2013-08-22 10:24:33: 0: INFO: Render Executable: “C:\Program Files\Autodesk\Maya2012\bin\MayaBatch.exe”
2013-08-22 10:24:33: 0: INFO: Render Argument: -prompt -file “\inferno2\projects\common\pipeline\submission_queue\2013_08_22\tmpmuzon1.ma_10_24_25_219\tmpmuzon1.ma” -proj “//inferno2/projects/burn/scenes/SHR_shr_rsrc/images/render3d/bldTriskellionGarageDoors/v0007_jal_turnTable/beautyJPG_hd_sRGB”
2013-08-22 10:24:33: 0: INFO: Startup Directory: “C:\Program Files\Autodesk\Maya2012\bin”
2013-08-22 10:24:33: 0: INFO: Process Priority: BelowNormal
2013-08-22 10:24:33: 0: INFO: Process Affinity: default
2013-08-22 10:24:33: 0: INFO: Process is now running
2013-08-22 10:24:33: 0: Plugin rendering frame(s): 1046-1060
2013-08-22 10:24:33: 0: INFO: Executing pre task script C:\Users\scanlinevfx\AppData\Local\Thinkbox\Deadline6\slave\LAPRO0323\jobsData\prestart.py

The slave was running the latest beta:
2013-08-21 16:36:27: Deadline Slave 6.1 [v6.1.0.52340 R]

LaszloSebo · August 22, 2013, 8:46pm

Found another python crash
Hanging with a white deadline slave application (deadlineslave.exe ticking at 100% cpu usage), after trying to run the MayaBatch.py:

0: Plugin will be reloaded because a new job has been loaded, or one of the job files or plugin files has been modified
0: Loaded plugin: MayaBatch
0: Task timeout is 72000 seconds (Regular Task Timeout)
0: Loaded job: [BURN] Software Render: TST_000_0000_maya_animation_LayoutTest.ma version: v0005 (5216668011a19a1214bdffd9)
0: INFO: Executing plugin script C:\Users\scanlinevfx\AppData\Local\Thinkbox\Deadline6\slave\LAPRO0307\plugins\MayaBatch.py

The gui is still there just says not responding, so there is no corresponding event viewer crash report…

LaszloSebo · August 22, 2013, 8:51pm

I created a crash dump from this latter crash via the task manager, but its like 380 megs… should i upload this somewhere for you guys?

LaszloSebo · August 22, 2013, 8:55pm

I’ve uploaded it to the dropbox link you guys sent me for siggraph materials!

cbond · August 22, 2013, 10:22pm

thanks laszlo, i’ve pointed jon at the folder just now so he can grab it and have it locally

jgaudet · August 22, 2013, 10:54pm

Thanks! I nabbed the dump file off Box, and I’ll poke around. I’ll confer with Ryan tomorrow to see if these could be related/similar to the other problem he already fixed for this build…

Cheers,

Jon

LaszloSebo · August 23, 2013, 4:24pm

Another event viewer application crash (the .net crash report is identical to the one in the first post):

Log Name: Application
Source: Application Error
Date: 8/22/2013 10:24:34 AM
Event ID: 1000
Task Category: (100)
Level: Error
Keywords: Classic
User: N/A
Computer: lapro0323
Description:
Faulting application name: deadlineslave.exe, version: 6.1.0.52340, time stamp: 0x5213a7d2
Faulting module name: python26.DLL, version: 2.6.7150.1013, time stamp: 0x4e00e564
Exception code: 0xc0000005
Fault offset: 0x0000000000104707
Faulting process id: 0x234
Faulting application start time: 0x01ce9ec74129d545
Faulting application path: C:\Program Files\Thinkbox\Deadline6\bin\deadlineslave.exe
Faulting module path: C:\Program Files\Thinkbox\Deadline6\bin\python26.DLL
Report Id: b5d26fef-0b4f-11e3-84d9-003048c69347
Event Xml:

1000
2
100
0x80000000000000

3519
Application
lapro0323

deadlineslave.exe
6.1.0.52340
5213a7d2
python26.DLL
2.6.7150.1013
4e00e564
c0000005
0000000000104707
234
01ce9ec74129d545
C:\Program Files\Thinkbox\Deadline6\bin\deadlineslave.exe
C:\Program Files\Thinkbox\Deadline6\bin\python26.DLL
b5d26fef-0b4f-11e3-84d9-003048c69347

Last lines of the log:

2013-08-22 10:24:33: 0: Got task!
2013-08-22 10:24:33: 0: Plugin will be reloaded because a new job has been loaded, or one of the job files or plugin files has been modified
2013-08-22 10:24:33: Constructor: MayaBatch
2013-08-22 10:24:33: 0: Loaded plugin: MayaBatch
2013-08-22 10:24:33: 0: Task timeout is 72000 seconds (Regular Task Timeout)
2013-08-22 10:24:33: 0: Loaded job: [BURN] VRay Render: SHR_shr_rsrc_maya_modeling_bldTriskellionGarageDoors.ma version: v0007 (5216494b00ff6a195480d844)
2013-08-22 10:24:33: 0: INFO: Executing plugin script C:\Users\scanlinevfx\AppData\Local\Thinkbox\Deadline6\slave\LAPRO0323\plugins\MayaBatch.py
2013-08-22 10:24:33: 0: INFO: About: Maya Batch Plugin for Deadline
2013-08-22 10:24:33: 0: INFO: The job’s environment will be merged with the current environment before rendering
2013-08-22 10:24:33: 0: INFO: Rendering with Maya version 2012.0
2013-08-22 10:24:33: 0: INFO: Starting monitored managed process MayaBatch
2013-08-22 10:24:33: 0: INFO: Stdout Handling Enabled: True
2013-08-22 10:24:33: 0: INFO: Popup Handling Enabled: True
2013-08-22 10:24:33: 0: INFO: Using Process Tree: True
2013-08-22 10:24:33: 0: INFO: Hiding DOS Window: True
2013-08-22 10:24:33: 0: INFO: Creating New Console: False
2013-08-22 10:24:33: 0: INFO: Enforcing 64 bit build of Maya
2013-08-22 10:24:33: 0: INFO: Render Executable: “C:\Program Files\Autodesk\Maya2012\bin\MayaBatch.exe”
2013-08-22 10:24:33: 0: INFO: Render Argument: -prompt -file “\inferno2\projects\common\pipeline\submission_queue\2013_08_22\tmpmuzon1.ma_10_24_25_219\tmpmuzon1.ma” -proj “//inferno2/projects/burn/scenes/SHR_shr_rsrc/images/render3d/bldTriskellionGarageDoors/v0007_jal_turnTable/beautyJPG_hd_sRGB”
2013-08-22 10:24:33: 0: INFO: Startup Directory: “C:\Program Files\Autodesk\Maya2012\bin”
2013-08-22 10:24:33: 0: INFO: Process Priority: BelowNormal
2013-08-22 10:24:33: 0: INFO: Process Affinity: default
2013-08-22 10:24:33: 0: INFO: Process is now running
2013-08-22 10:24:33: 0: Plugin rendering frame(s): 1046-1060
2013-08-22 10:24:33: 0: INFO: Executing pre task script C:\Users\scanlinevfx\AppData\Local\Thinkbox\Deadline6\slave\LAPRO0323\jobsData\prestart.py

full log attached
deadlineslave_LAPRO0323-LAPRO0323-2013-08-22-0000.log (411 KB)