OS: Win 7 Ent N x64
Job: Nuke 6.3v7
Deadline: 5.1.0.47014
Not really sure what is causing it, but running Check for Stalled Slaves in Monitor doesn’t report them and they just sit there dead. These nodes are remote-hosted but the latency is 1ms or less so that shouldn’t be a problem but it is the only difference from these 4 nodes and the rest of them. This error seem to happen somewhat randomly so I don’t have a lot of data to be able to reproduce it. Attached the screenshots and log below. This DOES happen on nodes that aren’t remote-hosted however but it doesn’t seem to happen as often, some of these nodes stay locked for hours or days, not sure what the deal is.
Screenshots:
Monitor.png
Verbose error in Logs:
2012-06-07 12:11:24: 0: Total time for task: 8.251 m
2012-06-07 12:11:25: Scheduler Thread - notification targets…
2012-06-07 12:11:25: Scheduler Thread - submitUser
2012-06-07 12:11:26: Caught unhandled exception: Attempted to read or write protected memory. This is often an indication that other memory is corrupt. (System.AccessViolationException)
at e__NativeCall.Void_Call_1(IntPtr , IntPtr )
at Python.Runtime.Runtime.Decref(IntPtr op)
at Python.Runtime.PythonException.Dispose()
at Python.Runtime.PythonException.Finalize()
2012-06-07 12:11:26: reporting ‘Deadline Slave 5.1 [v5.1.0.47014 R] - AccessViolationException: Attempted to read or write protected memory. This is often an indication that ot’ via email…
2012-06-07 12:11:26: ErrorLog.WriteExceptionLog() >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
Date Stamp
CurrentDate: Thursday, June 07, 2012
CurrentTime: 12:11:26 PM
Exception Details
AccessViolationException – Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
Exception.Data: ( )
Exception.TargetSite: Void Void_Call_1(IntPtr, IntPtr)
Exception.Source: e__NativeCall_Assembly
Exception.StackTrace:
at e__NativeCall.Void_Call_1(IntPtr , IntPtr )
at Python.Runtime.Runtime.Decref(IntPtr op)
at Python.Runtime.PythonException.Dispose()
at Python.Runtime.PythonException.Finalize()
ErrorReporting Watches
“Version”: v5.1.0.47014 R[String]
“currentTaskFrames”: 91-100[String]
“currentJobPool”: 2d[String]
“currentJobSubmitMachineName”: SubmitMachine[String]
“currentJobGroup”: nuke[String]
“currentJobPlugin”: Nuke[String]
“currentJobUser”: submitUser[String]
“currentJobPriority”: 80[Int32]
“currentJobDateTime”: 06/07/2012 11:58:18[DateTime]
“currentJobId”: 000_080_004_474b94ca[String]
Trace2Cache:
0: STDOUT: Crop1: 33.2MB 1920x1080 rgba 100% w 129600
0: STDOUT: Merge26: 33.9MB 1940x1090 rgba 100% w 125350
0: STDOUT: Copy10: 30MB 1920x1080 Starfield,Adam,GlassRef,SpaceshipWindow,WindowHole 100% w 78
0: STDOUT: Merge1: 26.4MB 1940x1090 rgba 100% w 20
0: STDOUT: WindowHole: 187kB 1920x1080 rgba.alpha 100% w 190
0: STDOUT: Colorspace3: 10.9MB 1280x720 rgba 100% w 10817
0: STDOUT: Read8: 8.38MB 1280x720 rgb 100% w 33
0: STDOUT: Read6: 8.66MB 1280x720 rgb 100% w 20
0: STDOUT: Read4: 8.66MB 1280x720 rgb 100% w 20
0: STDOUT: Grade8: 8.05MB 1280x720 rgb 100% w 20
0: STDOUT: HConvolve in Blur in Glow2: 1.06MB 428x242 rgb 100% w 198
0: STDOUT: DownRez in Blur in Glow2: 911kB 429x242 rgb 100% w 2629
0: STDOUT: SpaceshipWindow: 2.48MB 1920x1080 rgba,GlassRef 100% w 31
0: STDOUT: Copy7: 21.8MB 1397x994 rgba.alpha,Adam 100% w 69489
0: STDOUT: Premult6: 17.4MB 1396x992 rgba 100% w 541
0: STDOUT: Grade7: 21.8MB 1450x992 rgba 100% w -1000
0: STDOUT: Copy5: 21.8MB 1450x992 rgba 100% w -1000
0: STDOUT: Keylight in SpillKillPlus4.Keylight1: 16.7MB 1396x992 rgb 100% w 20
0: STDOUT: Read25: 16.7MB 1396x992 rgb 100% w 57040
0: STDOUT: Roto4: 210kB 257x201 rgba.alpha 100% w 7943
0: STDOUT: LensDistortion1: 15.6MB 1736x1048 rgba 100% w 21036
0: STDOUT: Merge24: 17.4MB 1922x1082 rgba 100% w 104438
0: STDOUT: HConvolve in Blur5: 33kB 58x34 rgb 100% w 10810
0: STDOUT: DownRez in Blur5: 33.4kB 59x34 rgb 100% w 5120
0: STDOUT: Read12: 6.14MB 951x536 rgb 100% w 121
0: STDOUT: LensDistortion2: 19.9MB 1734x1046 rgba 100% w 28468
0: STDOUT: Premult7: 22.4MB 1920x1080 rgba 100% w 231111
0: STDOUT: Keylight in SpillKillPlus5.Keylight1: 24.9MB 1920x1080 rgb 100% w 20
0: STDOUT: Grade3: 24.9MB 1920x1080 rgb 100% w 63180
0: STDOUT: Merge10: 25.9MB 1940x1090 rgba 100% w 136250
0: STDOUT: Read23: 14MB 1280x720 rgba 100% w 32450
0: STDOUT: Reformat2: 26MB 1920x1080 rgba 100% w 86400
0: STDOUT: Read22: 11.5MB 1280x720 rgba 100% w 32330
0: STDOUT: Adam1: 2.44MB 1920x1080 rgba 100% w -1000
0: STDOUT: Merge22: 14.8MB 1280x720 rgba 100% w 1320
0: STDOUT: Read7: 8.2MB 1280x720 rgb 100% w 40
0: STDOUT: Read5: 11.4MB 1280x720 rgba 100% w 3163
0: STDOUT: Merge20: 2.85MB 1282x722 rgba.alpha 100% w 4555
0: STDOUT: Premult2: 14.8MB 1280x720 rgba 100% w 1026
0: STDOUT: HConvolve in Blur4: 115kB 258x146 rgba.alpha 100% w 198
0: STDOUT: DownRez in Blur4: 79.5kB 264x152 rgba.alpha 100% w 4660
0: STDOUT: Keyer3: 2.18MB 1280x720 rgba.alpha 100% w 1696
0: STDOUT: HConvolve in Blur in Glow1: 878kB 428x242 rgb 100% w 83
0: STDOUT: DownRez in Blur in Glow1: 674kB 429x242 rgb 100% w 534
0: STDOUT: Read10: 33.2MB 1920x1080 rgba 100% w 135000
0: STDOUT: EdgeExtend.gizmo1.Remove1: 25.2MB 1920x1080 rgba 100% w 232200
0: STDOUT: EdgeExtend.gizmo1.Unpremult4: 23MB 1920x1080 rgba 100% w 21590
0: STDOUT: EdgeExtend.gizmo1.Merge2: 32.3MB 1920x1080 rgb,depth 100% w 20
0: STDOUT: EdgeExtend.gizmo1.Merge1: 31.4MB 1920x1080 rgba 100% w 21560
0: STDOUT: EdgeExtend.gizmo1.Unpremult1: 29.4MB 1920x1080 rgb,depth 100% w 21585
0: STDOUT: Grade17: 33.2MB 1920x1080 rgba 100% w 21
0: STDOUT: HConvolve in Blur in Glow4: 529kB 276x156 rgb 100% w 745
0: STDOUT: DownRez in Blur in Glow4: 535kB 277x157 rgb 100% w 15668
0: STDOUT: Copy7: 21.6MB 1403x994 rgba.alpha,Adam 100% w -1000
0: STDOUT: Premult6: 17.3MB 1402x992 rgba 100% w -1000
0: STDOUT: Keylight in SpillKillPlus4.Keylight1: 16.7MB 1402x992 rgb 100% w -1000
0: STDOUT: Read25: 16.7MB 1402x992 rgb 100% w -1000
0: STDOUT: Roto4: 213kB 258x202 rgba.alpha 100% w -1000
0: STDOUT: LensDistortion1: 15.6MB 1736x1048 rgba 100% w -1000
0: STDOUT: Merge24: 17.4MB 1922x1082 rgba 100% w -1000
0: STDOUT: HConvolve in Blur5: 33kB 58x34 rgb 100% w -1000
0: STDOUT: DownRez in Blur5: 33.4kB 59x34 rgb 100% w -1000
0: STDOUT: Read12: 6.14MB 951x536 rgb 100% w -1000
0: STDOUT: LensDistortion2: 19.9MB 1734x1046 rgba 100% w -1000
0: STDOUT: Premult7: 22.4MB 1920x1080 rgba 100% w -1000
0: STDOUT: Keylight in SpillKillPlus5.Keylight1: 24.9MB 1920x1080 rgb 100% w -1000
0: STDOUT: Grade3: 24.9MB 1920x1080 rgb 100% w -1000
0: STDOUT: Keylight in SpillKillPlus4.Keylight1: output image :5.54MB
0: STDOUT: Keylight in SpillKillPlus4.Keylight1: output image :5.54MB
0: STDOUT: Keylight in SpillKillPlus4.Keylight1: output image :5.54MB
0: STDOUT: Keylight in SpillKillPlus4.Keylight1: output image :5.54MB
0: STDOUT: Keylight in SpillKillPlus5.Keylight1: output image :6.64MB
0: STDOUT: Keylight in SpillKillPlus5.Keylight1: output image :6.64MB
0: STDOUT: Keylight in SpillKillPlus5.Keylight1: output image :6.64MB
0: STDOUT: Keylight in SpillKillPlus5.Keylight1: output image :6.64MB
0: STDOUT: Keylight in SpillKillPlus5.Keylight1: output image :6.64MB
0: STDOUT: Depth Of Field in Depth Of Field1: output image :33.8MB
0: STDOUT: Depth Of Field in Depth Of Field3: output image :33.2MB
0: STDOUT: Depth Of Field in Depth Of Field2: output image :33.2MB
0: STDOUT: Out Of Focus in Out Of Focus1: output image :33.2MB
0: STDOUT: RSMB in RSMB1: output image :33.2MB
0: STDOUT: Keylight in SpillKillPlus4.Keylight1: output image :5.52MB
0: STDOUT: Keylight in SpillKillPlus4.Keylight1: output image :5.52MB
0: STDOUT: Keylight in SpillKillPlus4.Keylight1: output image :5.52MB
0: STDOUT: Keylight in SpillKillPlus4.Keylight1: output image :5.52MB
0: STDOUT: Keylight in SpillKillPlus5.Keylight1: output image :6.64MB
0: STDOUT: Keylight in SpillKillPlus5.Keylight1: output image :6.64MB
0: STDOUT: Keylight in SpillKillPlus5.Keylight1: output image :6.64MB
0: STDOUT: Keylight in SpillKillPlus5.Keylight1: output image :6.64MB
0: STDOUT: Keylight in SpillKillPlus5.Keylight1: output image :6.64MB
0: STDOUT: Depth Of Field in Depth Of Field1: output image :33.8MB
0: STDOUT: Depth Of Field in Depth Of Field3: output image :33.2MB
0: STDOUT: Depth Of Field in Depth Of Field2: output image :33.2MB
0: STDOUT: Out Of Focus in Out Of Focus1: output image :33.2MB
0: INFO: Process exit code: 0
0: Render time for frames: 8.21 m
0: Total time for task: 8.251 m
Scheduler Thread - notification targets…
Scheduler Thread - submitUser
Caught unhandled exception: Attempted to read or write protected memory. This is often an indication that other memory is corrupt. (System.AccessViolationException)
at e__NativeCall.Void_Call_1(IntPtr , IntPtr )
at Python.Runtime.Runtime.Decref(IntPtr op)
at Python.Runtime.PythonException.Dispose()
at Python.Runtime.PythonException.Finalize()
0 Other Running Threads
Process Threads
2620 Wait UserRequest
1460 Wait UserRequest
2540 Running
344 Wait UserRequest
280 Wait ExecutionDelay
2164 Wait ExecutionDelay
1824 Wait UserRequest
2764 Wait UserRequest
452 Wait UserRequest
1356 Wait UserRequest
2312 Wait ExecutionDelay
3020 Wait UserRequest
3004 Wait UserRequest
788 Wait UserRequest
2316 Wait UserRequest
3064 Wait ExecutionDelay
2736 Wait EventPairLow
2676 Wait EventPairLow
324 Wait UserRequest
3228 Wait UserRequest
3856 Wait UserRequest
2888 Wait EventPairLow
916 Wait UserRequest
3676 Wait UserRequest
3448 Wait UserRequest
2308 Wait UserRequest
2544 Wait UserRequest
3696 Wait UserRequest
3332 Wait UserRequest
1720 Wait UserRequest
3884 Wait UserRequest
2880 Wait UserRequest
1076 Wait ExecutionDelay
4040 Wait EventPairLow
496 Wait EventPairLow
Memory and CPU Stats
GC.TotalMemory: 31.539 MB
Environment.WorkingSet: 117.066 MB
ComputerSystem.TotalPhysicalMemory: 47.991 GB
ComputerSystem.FreePhysicalMemory: 46.499 GB
Application Information
Application.ExecutablePath: C:\Apps64\thinkbox\deadline\bin\deadlineslave.exe
Application.CurrentDirectory: C:\Windows\system32
Application.StartupPath: C:\Apps64\thinkbox\deadline\bin
Application.ProductName: Deadline Slave 5.1
Application.ProductVersion: 5.1.0.47014 File.GetLastWriteTime( Application.ExecutablePath ): 4/12/2012 6:38:36 AM
Assembly Information (Executing)
ExecutingAssembly.CodeBase: file:///C:/Apps64/thinkbox/deadline/bin/franticx.DLL
ExecutingAssembly.Location: C:\Apps64\thinkbox\deadline\bin\franticx.dll
ExecutingAssembly.GlobalAssemblyCache: False File.GetLastWriteTime( ExecutingAssembly.Location ): 4/12/2012 6:38:36 AM
Assembly Information (Current)
CurrentAssembly.CodeBase: file:///C:/Apps64/thinkbox/deadline/bin/franticx.DLL
CurrentAssembly.Location: C:\Apps64\thinkbox\deadline\bin\franticx.dll
CurrentAssembly.GlobalAssemblyCache: False File.GetLastWriteTime( CurrentAssembly.Location ): 4/12/2012 6:38:36 AM
Thread Information
CurrentThread.Name:
CurrentThread.Priority: Highest
Process Information
CurrentProcess.Name: deadlineslave
CurrentProcess.MachineName: .
CurrentProcess.MainModule: System.Diagnostics.ProcessModule (deadlineslave.exe)
CurrentProcess.StartDate: Tuesday, June 05, 2012
CurrentProcess.StartTime: 4:10:02 PM
CurrentProcess.UserProcessorTime: 12.581 m
CurrentProcess.TotalProcessorTime: 18.762 m
Operating System Information
Environment.OSVersion.Platform: Windows 7 Enterprise N (SP1)
Environment.OSVersion.Version: 6.1.7601.65536
.NET Platform Information
Environment.Version.Major: 2
Environment.Version.Minor: 0
Environment.Version.Build: 50727
Environment.Version.Revision: 5446
Misc Environment Information
Environment.MachineName: RenderNode
Environment.UserName: render
Environment.SystemDirectory: C:\Windows\system32
Environment.TickCount: 174865342
Command Line
Environment.CommandLine: “C:\Apps64\thinkbox\deadline\bin\deadlineslave.exe” -name RenderNode
Environment.CommandLineArgs[0]: C:\Apps64\thinkbox\deadline\bin\deadlineslave.exe
Environment.CommandLineArgs[1]: -name
Environment.CommandLineArgs[2]: RenderNode
Current Call Stack
Environment.StackTrace: at System.Environment.get_StackTrace()
at FranticX.Diagnostics.Reporting.ErrorReporting.GetExceptionReport(Exception e, ErrorReportDetail detail)
at FranticX.Diagnostics.Reporting.ErrorReporting.WriteExceptionReport(Exception ex)
at FranticX.Diagnostics.Reporting.ErrorReporting.RecordException(Exception e)
at FranticX.Diagnostics.Reporting.ErrorReporting.OnException(Exception e)
at FranticX.Diagnostics.Reporting.ErrorReporting.OnUnhandledException(Object o, UnhandledExceptionEventArgs e)
at e__NativeCall.Void_Call_1(IntPtr , IntPtr )
at Python.Runtime.Runtime.Decref(IntPtr op)
at Python.Runtime.PythonException.Dispose()
at Python.Runtime.PythonException.Finalize()
2012-06-07 12:11:26: email file reporting failed: Sender account in the SMTP settings is blank (System.Exception)
2012-06-07 12:11:26: exception occurred while recording file: Sender account in the SMTP settings is blank (System.Exception)