AWS Thinkbox Discussion Forums

improperly handled timeout

Deprecated Deadline 8 Bug Reports
#1

A nuke render seems to have hung up, and deadline is unable to requeue the task:

2016-11-29 19:08:37:  0: STDOUT: Writing S:/ncs/inferno2/projects/gate/scenes/NXN_017_0700/images/render2d/comp/v0054_dfe_grainFix/beauty_2150x1134x1_linear/NXN_017_0700_comp_beauty_v0054.1005.exr took 114.99 seconds
2016-11-29 19:08:38:  0: STDOUT: Frame 1005 (1 of 4)
2016-11-29 19:10:14:  0: STDOUT: .9.2.5.7.1.4.7.9.2.5.7.4.9
2016-11-29 19:10:14:  0: STDOUT: Writing S:/ncs/inferno2/projects/gate/scenes/NXN_017_0700/images/render2d/comp/v0054_dfe_grainFix/beauty_2150x1134x1_linear/NXN_017_0700_comp_beauty_v0054.1006.exr took 96.13 seconds
2016-11-29 19:10:27:  0: STDOUT: Frame 1006 (2 of 4)
2016-11-29 19:11:48:  0: STDOUT: .2.6.1.5.9.2.6.2.7
2016-11-29 19:11:48:  0: STDOUT: Writing S:/ncs/inferno2/projects/gate/scenes/NXN_017_0700/images/render2d/comp/v0054_dfe_grainFix/beauty_2150x1134x1_linear/NXN_017_0700_comp_beauty_v0054.1007.exr took 91.87 seconds
2016-11-29 19:11:51:  0: STDOUT: Frame 1007 (3 of 4)
2016-11-29 19:12:57:  Listener Thread - ::ffff:172.18.3.106 has connected
2016-11-29 19:12:57:  Listener Thread - Received message: StopSlave
2016-11-29 19:12:59:  Listener Thread - Exception (::ffff:172.18.3.106): Exception of type 'System.OutOfMemoryException' was thrown.
2016-11-29 19:12:59:  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2016-11-29 19:12:59:  Exception Details
2016-11-29 19:12:59:  OutOfMemoryException -- Exception of type 'System.OutOfMemoryException' was thrown.
2016-11-29 19:12:59:  Exception.Data: ( )
2016-11-29 19:12:59:  Exception.TargetSite: System.String HandleMessage(System.String, System.Net.Sockets.Socket, Boolean ByRef)
2016-11-29 19:12:59:  Exception.Source: deadline
2016-11-29 19:12:59:  Exception.HResult: -2147024882
2016-11-29 19:12:59:    Exception.StackTrace: 
2016-11-29 19:12:59:     at Deadline.Slaves.SlaveListenerThread.HandleMessage(String message, Socket socket, Boolean& closeSocket)
2016-11-29 19:12:59:     at Deadline.ListenerThread.a(IAsyncResult A_0)
2016-11-29 19:12:59:  <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2016-11-29 19:13:00:  0: Task timed out -- canceling current task...
2016-11-29 19:13:03:  0: Unloading plugin: Nuke
2016-11-29 19:13:03:  Scheduler Thread - Render Thread 0 threw a major error: 
2016-11-29 19:13:03:  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2016-11-29 19:13:03:  Exception Details
2016-11-29 19:13:03:  JobTimeoutException -- The Slave did not complete the task before the Regular Task Timeout limit of 00d 00h 24m 00s. The Task Timeout settings can be changed for this job by right-clicking on it in the Monitor and selecting "Modify Properties...".
2016-11-29 19:13:03:  RenderPluginException.Cause: JobError (2)
2016-11-29 19:13:03:  RenderPluginException.Level: Major (1)
2016-11-29 19:13:03:  RenderPluginException.HasSlaveLog: True
2016-11-29 19:13:03:  RenderPluginException.SlaveLogFileName: C:\ProgramData\Thinkbox\Deadline8\logs\deadlineslave_secondary_renderthread_0-LAPRO0456-0000.log
2016-11-29 19:13:03:  Exception.Data: ( )
2016-11-29 19:13:03:  Exception.HResult: -2146233088
2016-11-29 19:13:03:    Exception.StackTrace: 
2016-11-29 19:13:03:      (null)
2016-11-29 19:13:03:  <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2016-11-29 19:13:05:  Error occurred while writing report log:
2016-11-29 19:13:05:  Exception Details
2016-11-29 19:13:05:  OutOfMemoryException -- Exception of type 'System.OutOfMemoryException' was thrown.
2016-11-29 19:13:05:  Exception.Data: ( )
2016-11-29 19:13:05:  Exception.TargetSite: Void AllocateCompressStructures()
2016-11-29 19:13:05:  Exception.Source: ICSharpCode.SharpZipLib
2016-11-29 19:13:05:  Exception.HResult: -2147024882
2016-11-29 19:13:05:    Exception.StackTrace: 
2016-11-29 19:13:05:     at ICSharpCode.SharpZipLib.BZip2.BZip2OutputStream.AllocateCompressStructures()
2016-11-29 19:13:05:     at ICSharpCode.SharpZipLib.BZip2.BZip2OutputStream..ctor(Stream inStream, Int32 inBlockSize)
2016-11-29 19:13:05:     at ICSharpCode.SharpZipLib.BZip2.BZip2.Compress(Stream instream, Stream outstream, Int32 blockSize)
2016-11-29 19:13:05:     at FranticX.IO.Compression.BZip2.CompressFile(String inputFilename, String outputFilename)
2016-11-29 19:13:05:     at Deadline.StorageDB.JobStorage.WriteJobReportFile(Report report, String reportFileName)
2016-11-29 19:13:08:  System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
2016-11-29 19:13:08:     at System.Threading.Thread.StartInternal(IPrincipal principal, StackCrawlMark& stackMark)
2016-11-29 19:13:08:     at System.Threading.Thread.Start(StackCrawlMark& stackMark)
2016-11-29 19:13:08:     at System.Threading.Thread.Start(Object parameter)
2016-11-29 19:13:08:     at Deadline.Slaves.CommandListener.b()
2016-11-29 19:13:08:     at Deadline.Slaves.CommandListener.c()
2016-11-29 19:13:08:     at Deadline.Slaves.CommandListener..ctor(Int32 commandPort)
2016-11-29 19:13:08:     at a.a(String[] A_0)

The slave is still hanging on that frame:
Capture.PNG

Shouldnt it get flagged as stalled eventually and the task requeued?

#2

I suspect this is the same issue as forums.thinkboxsoftware.com/vie … 10&t=14927, that the slave cannot recover if the render thread goes down. This issue was one of the reasons we introduced plugin sandboxing. Do you think you’d be able to test with sandboxing enabled once 8.0.12 comes out?

#3

yep, we can, eagerly waiting for the build!

#4

8.0.12.4 has been released: docs.thinkboxsoftware.com/produc … notes.html

#5

Thanks! In the process of integrating it :slight_smile:

Privacy | Site terms | Cookie preferences