As title says I went from 1 task to 3 concurrent tasks then back to 1 and the job went insane. Still 7.1 though.
Woah! Cool! (from a troubleshooting perspective)
Since I bet the job isn’t around anymore, are you abel to reproduce? How can I make my job explode the same way?
What version are you on? I’m sure you’ve mentioned before, but it’ll save me a search.
Latest stable build 7.1.2.1 I believe.
Unfortunately this job is under nda so no job details can go out.
Happening again on 8.0.7.3 Release. Just changing to 3 concurrent tasks filled every pending task.
Job details:
Job Name User Errors Task Progress Status Submit Date/Time Start Date/Time Finished Date/Time Clock Running Time Estimated Remaining Render Time Pool Secondary Pool Group Priority Weight Frames Concurrent Tasks Machine Limit Task Timeout Plugin Limits Submit Machine Auxiliary Files Auxiliary File Size Job Mode Job ID Batch Name Submission Region Extra Info 0 Extra Info 1 Extra Info 2 Extra Info 3 Extra Info 4 Extra Info 5 Extra Info 6 Extra Info 7 Extra Info 8 Extra Info 9
PNB_nutcracker_0010_B01.08.nk ggreenwalt 73 75 % (132/176) Rendering (17) 2016/09/27 16:02:41 2016/09/27 16:03:00 00:01:13:50 00:02:45:44 global global global 60 2545-4300 1 Nuke nuke, nukei Daedalus 1 1.755 MB Normal 57eafa93b5b7e231f407f2e7
Possibly relevant:
Slave Name Machine Name Machine User Description Status Last Status Update Offline Message Connected To Pulse Comment Assigned Pools Assigned Groups Dequeuing Mode Idle Detection Override Region Ping After Last Task Complete Job Plugin Job User Job Pool Job Group Job Name Job ID Job Priority Task Frames Task Render Time Task Progress Task Render Status Completed Tasks Failed Tasks Version License Running Time CPUs CPU Affinity GPU Affinity IP Address MAC Address Free Disk Space Memory Usage Total Memory CPU Speed CPU Usage CPU Architecture OS Limit Stubs in Use Whitelisted Limits Blacklisted Limits Excluded Limits Video Card Concurrent Task Limit Override Normalized Render Time Multiplier Normalized Task Timeout Multiplier Network Sent Network Received Swap Used Disk Read Disk Write Extra Info 0 Extra Info 1 Extra Info 2 Extra Info 3 Extra Info 4 Extra Info 5 Extra Info 6 Extra Info 7 Extra Info 8 Extra Info 9
Harvey Harvey Renderadmin Rendering (1.5 m) 2016/09/27 17:21:22 No urgent,global,local workstations,global,i7_v01,rendernodes All Jobs Off Disabled Continue Running Nuke ggreenwalt global global PNB_nutcracker_0010_B01.08.nk 57eafa93b5b7e231f407f2e7 60 4105-4114 00d 00h 01m 32s 80% .9Frame 4112 (8 of 10) 8 1 v8.0.7.3 Release (f33fcb7d3) Permanent 14.9 m 8 8 192.168.94.41 BC:5F:F4:45:78:76 282.799 GB (31.320 GB on C:\, 251.479 GB on G:\) 9.8 GB / 31.9 GB (30%) 31.9 GB 3.4 GHz 21 % x64 Windows 10 Pro nuke_r, 57eafa93b5b7e231f407f2e7 nuke_r nuke Intel(R) HD Graphics 3000 4 1 1 0.0 Bytes / second 0.0 Bytes / second 2.0 MB 0.0 Bytes / second 0.0 Bytes / second
2016-09-27 17:10:04: Scheduler - Slave is on the excluded list for the 'nuke' limit.
2016-09-27 17:10:04: Scheduler - Successfully dequeued 3 task(s). Returning.
2016-09-27 17:10:04: Scheduler - Unexpected Error Occurred
2016-09-27 17:10:04: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2016-09-27 17:10:04: Exception Details
2016-09-27 17:10:04: IndexOutOfRangeException -- Index was outside the bounds of the array.
2016-09-27 17:10:04: Exception.Data: ( )
2016-09-27 17:10:04: Exception.TargetSite: Void SetTask(Int32, Deadline.Jobs.Task)
2016-09-27 17:10:04: Exception.Source: deadline
2016-09-27 17:10:04: Exception.HResult: -2146233080
2016-09-27 17:10:04: Exception.StackTrace:
2016-09-27 17:10:04: at Deadline.Slaves.SlaveState.SetTask(Int32 index, Task task)
2016-09-27 17:10:04: at Deadline.Controllers.DataController.DequeueTasks(SlaveState slaveState, SlaveSettings slaveSettings, Boolean verboseLogging, Boolean secondaryScan)
2016-09-27 17:10:04: <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2016-09-27 17:10:04: Scheduler - Performing Job scan on Secondary Pools with scheduling order Pool, Priority, First-in First-out
2016-09-27 17:10:04: Scheduler - Slave is on the excluded list for the 'nuke' limit.
2016-09-27 17:10:04: Scheduler - Successfully dequeued 3 task(s). Returning.
2016-09-27 17:10:04: Scheduler - Unexpected Error Occurred
2016-09-27 17:10:04: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2016-09-27 17:10:04: Exception Details
2016-09-27 17:10:04: IndexOutOfRangeException -- Index was outside the bounds of the array.
2016-09-27 17:10:04: Exception.Data: ( )
2016-09-27 17:10:04: Exception.TargetSite: Void SetTask(Int32, Deadline.Jobs.Task)
2016-09-27 17:10:04: Exception.Source: deadline
2016-09-27 17:10:04: Exception.HResult: -2146233080
2016-09-27 17:10:04: Exception.StackTrace:
2016-09-27 17:10:04: at Deadline.Slaves.SlaveState.SetTask(Int32 index, Task task)
2016-09-27 17:10:04: at Deadline.Controllers.DataController.DequeueTasks(SlaveState slaveState, SlaveSettings slaveSettings, Boolean verboseLogging, Boolean secondaryScan)
2016-09-27 17:10:04: <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2016-09-27 17:10:04: Message From License Forwarder: Success: Machine unregistered.
2016-09-27 17:10:05: Message From License Forwarder: Success: Machine unregistered.
016-09-27 17:09:24: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2016-09-27 17:09:24: Exception Details
2016-09-27 17:09:24: LicenseForwarderConnectionException -- Unable to connect to the License Forwarder at 192.168.94.29:17084. Check to make sure that it is available.
2016-09-27 17:09:24: Exception.Data: ( )
2016-09-27 17:09:24: Exception.TargetSite: Boolean a(Deadline.LicenseForwarder.LicenseForwarderInfoSettings, System.String[], Deadline.LicenseForwarder.LicenseForwarderRequestType)
2016-09-27 17:09:24: Exception.Source: deadline
2016-09-27 17:09:24: Exception.HResult: -2146233088
2016-09-27 17:09:24: Exception.StackTrace:
2016-09-27 17:09:24: at Deadline.LicenseForwarder.LicenseForwarderUtils.a(LicenseForwarderInfoSettings A_0, String[] A_1, LicenseForwarderRequestType A_2)
2016-09-27 17:09:24: at Deadline.LicenseForwarder.LicenseForwarderUtils.UnregisterMachine(LicenseForwarderInfoSettings licenseForwarder)
2016-09-27 17:09:24: at Deadline.LicenseForwarder.LicenseForwarderUtils.UnregisterMachine()
2016-09-27 17:09:24: at Deadline.Configuration.Plugins.PluginManager.UnloadPlugin()
2016-09-27 17:09:24: at Deadline.Slaves.SlaveSchedulerThread.h()
2016-09-27 17:09:24: <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2016-09-27 17:09:24: Scheduler - Returning all limit stubs.
2016-09-27 17:09:24: Scheduler - returning 57eafa93b5b7e231f407f2e7
2016-09-27 17:09:24: Scheduler - returning nuke_r
2016-09-27 17:09:26: Scheduler - Performing Job scan on Primary Pools with scheduling order Pool, Priority, First-in First-out
2016-09-27 17:09:29: 0: STDOUT: DeepWrite1 .1.2.5.8
2016-09-27 17:09:29: 0: STDOUT: DeepWrite1 took 15.05 seconds
2016-09-27 17:09:29: 0: STDOUT: Total render time: 15 seconds
2016-09-27 17:09:29: 0: STDOUT: .9Frame 3654 (10 of 10)
2016-09-27 17:09:29: 0: STDOUT: READY FOR INPUT
2016-09-27 17:09:29: Scheduler - Slave is on the excluded list for the 'nuke' limit.
2016-09-27 17:09:29: Scheduler - Successfully dequeued 1 task(s). Returning.
2016-09-27 17:09:29: Scheduler - Unexpected Error Occurred
2016-09-27 17:09:29: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2016-09-27 17:09:29: Exception Details
2016-09-27 17:09:29: IndexOutOfRangeException -- Index was outside the bounds of the array.
2016-09-27 17:09:29: Exception.Data: ( )
2016-09-27 17:09:29: Exception.TargetSite: Void SetTask(Int32, Deadline.Jobs.Task)
2016-09-27 17:09:29: Exception.Source: deadline
2016-09-27 17:09:29: Exception.HResult: -2146233080
2016-09-27 17:09:29: Exception.StackTrace:
2016-09-27 17:09:29: at Deadline.Slaves.SlaveState.SetTask(Int32 index, Task task)
2016-09-27 17:09:29: at Deadline.Controllers.DataController.DequeueTasks(SlaveState slaveState, SlaveSettings slaveSettings, Boolean verboseLogging, Boolean secondaryScan)
2016-09-27 17:09:29: <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2016-09-27 17:09:29: Scheduler - Performing Job scan on Secondary Pools with scheduling order Pool, Priority, First-in First-out
2016-09-27 17:09:29: Scheduler - Slave is on the excluded list for the 'nuke' limit.
2016-09-27 17:09:29: Scheduler - Successfully dequeued 1 task(s). Returning.
2016-09-27 17:09:29: Scheduler - Unexpected Error Occurred
2016-09-27 17:09:29: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2016-09-27 17:09:29: Exception Details
2016-09-27 17:09:29: IndexOutOfRangeException -- Index was outside the bounds of the array.
2016-09-27 17:09:29: Exception.Data: ( )
2016-09-27 17:09:29: Exception.TargetSite: Void SetTask(Int32, Deadline.Jobs.Task)
2016-09-27 17:09:29: Exception.Source: deadline
2016-09-27 17:09:29: Exception.HResult: -2146233080
2016-09-27 17:09:29: Exception.StackTrace:
2016-09-27 17:09:29: at Deadline.Slaves.SlaveState.SetTask(Int32 index, Task task)
2016-09-27 17:09:29: at Deadline.Controllers.DataController.DequeueTasks(SlaveState slaveState, SlaveSettings slaveSettings, Boolean verboseLogging, Boolean secondaryScan)
2016-09-27 17:09:29: <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2016-09-27 17:09:29: 0: Done executing plugin command of type 'Render Task'
2016-09-27 17:09:29: 0: Render time for frame(s): 2.754 m
2016-09-27 17:09:29: 0: Total time for task: 2.816 m
2016-09-27 17:09:29: 0: Saving task log...
2016-09-27 17:09:30: Message From License Forwarder: Success: Machine unregistered.
2016-09-27 17:09:31: Message From License Forwarder: Success: Machine unregistered.
2016-09-27 17:09:31: 2: STDOUT: DeepWrite1 .1.2.4
2016-09-27 17:09:31: 2: STDOUT: DeepWrite1 took 7.35 seconds
2016-09-27 17:09:31: 2: STDOUT: Total render time: 7.35 seconds
2016-09-27 17:09:31: 2: STDOUT: .9Frame 3795 (1 of 10)
2016-09-27 17:09:32: Scheduler Thread - exception occurred:
2016-09-27 17:09:32: Scheduler Thread - Unexpected Error Occurred
2016-09-27 17:09:32: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2016-09-27 17:09:32: Exception Details
2016-09-27 17:09:32: LicenseForwarderConnectionException -- Unable to connect to the License Forwarder at 192.168.94.29:17084. Check to make sure that it is available.
2016-09-27 17:09:32: Exception.Data: ( )
2016-09-27 17:09:32: Exception.TargetSite: Boolean a(Deadline.LicenseForwarder.LicenseForwarderInfoSettings, System.String[], Deadline.LicenseForwarder.LicenseForwarderRequestType)
2016-09-27 17:09:32: Exception.Source: deadline
2016-09-27 17:09:32: Exception.HResult: -2146233088
2016-09-27 17:09:32: Exception.StackTrace:
2016-09-27 17:09:32: at Deadline.LicenseForwarder.LicenseForwarderUtils.a(LicenseForwarderInfoSettings A_0, String[] A_1, LicenseForwarderRequestType A_2)
2016-09-27 17:09:32: at Deadline.LicenseForwarder.LicenseForwarderUtils.UnregisterMachine(LicenseForwarderInfoSettings licenseForwarder)
2016-09-27 17:09:32: at Deadline.LicenseForwarder.LicenseForwarderUtils.UnregisterMachine()
2016-09-27 17:09:32: at Deadline.Configuration.Plugins.PluginManager.UnloadPlugin()
2016-09-27 17:09:32: at Deadline.Slaves.SlaveSchedulerThread.h()
2016-09-27 17:09:32: <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2016-09-27 17:09:32: Scheduler - Returning all limit stubs.
2016-09-27 17:09:32: Scheduler - returning nuke_r
2016-09-27 17:09:34: Scheduler - returning 57eafa93b5b7e231f407f2e7
2016-09-27 17:09:34: Scheduler - Performing Job scan on Primary Pools with scheduling order Pool, Priority, First-in First-out
2016-09-27 17:09:37: Scheduler - Slave is on the excluded list for the 'nuke' limit.
2016-09-27 17:09:37: 2: STDOUT: DeepWrite1 .1.2.5
2016-09-27 17:09:37: 2: STDOUT: DeepWrite1 took 5.71 seconds
2016-09-27 17:09:37: 2: STDOUT: Total render time: 5.71 seconds
2016-09-27 17:09:37: 2: STDOUT: .9Frame 3796 (2 of 10)
2016-09-27 17:09:37: Scheduler - Successfully dequeued 3 task(s). Returning.
2016-09-27 17:09:37: Scheduler - Unexpected Error Occurred
2016-09-27 17:09:37: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2016-09-27 17:09:37: Exception Details
2016-09-27 17:09:37: IndexOutOfRangeException -- Index was outside the bounds of the array.
2016-09-27 17:09:37: Exception.Data: ( )
2016-09-27 17:09:37: Exception.TargetSite: Void SetTask(Int32, Deadline.Jobs.Task)
2016-09-27 17:09:37: Exception.Source: deadline
2016-09-27 17:09:37: Exception.HResult: -2146233080
2016-09-27 17:09:37: Exception.StackTrace:
2016-09-27 17:09:37: at Deadline.Slaves.SlaveState.SetTask(Int32 index, Task task)
2016-09-27 17:09:37: at Deadline.Controllers.DataController.DequeueTasks(SlaveState slaveState, SlaveSettings slaveSettings, Boolean verboseLogging, Boolean secondaryScan)
2016-09-27 17:09:37: <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2016-09-27 17:09:37: Scheduler - Performing Job scan on Secondary Pools with scheduling order Pool, Priority, First-in First-out
2016-09-27 17:09:37: Scheduler - Slave is on the excluded list for the 'nuke' limit.
2016-09-27 17:09:37: Scheduler - Successfully dequeued 3 task(s). Returning.
2016-09-27 17:09:37: Scheduler - Unexpected Error Occurred
2016-09-27 17:09:37: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2016-09-27 17:09:37: Exception Details
2016-09-27 17:09:37: IndexOutOfRangeException -- Index was outside the bounds of the array.
2016-09-27 17:09:37: Exception.Data: ( )
2016-09-27 17:09:37: Exception.TargetSite: Void SetTask(Int32, Deadline.Jobs.Task)
2016-09-27 17:09:37: Exception.Source: deadline
2016-09-27 17:09:37: Exception.HResult: -2146233080
2016-09-27 17:09:37: Exception.StackTrace:
2016-09-27 17:09:37: at Deadline.Slaves.SlaveState.SetTask(Int32 index, Task task)
2016-09-27 17:09:37: at Deadline.Controllers.DataController.DequeueTasks(SlaveState slaveState, SlaveSettings slaveSettings, Boolean verboseLogging, Boolean secondaryScan)
2016-09-27 17:09:37: <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2016-09-27 17:09:37: Message From License Forwarder: Success: Machine unregistered.
2016-09-27 17:09:39: Message From License Forwarder: Success: Machine unregistered.
2016-09-27 17:09:40: Scheduler Thread - exception occurred:
2016-09-27 17:09:40: Scheduler Thread - Unexpected Error Occurred
2016-09-27 17:09:40: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2016-09-27 17:09:40: Exception Details
2016-09-27 17:09:40: LicenseForwarderConnectionException -- Unable to connect to the License Forwarder at 192.168.94.29:17084. Check to make sure that it is available.
2016-09-27 17:09:40: Exception.Data: ( )
2016-09-27 17:09:40: Exception.TargetSite: Boolean a(Deadline.LicenseForwarder.LicenseForwarderInfoSettings, System.String[], Deadline.LicenseForwarder.LicenseForwarderRequestType)
2016-09-27 17:09:40: Exception.Source: deadline
2016-09-27 17:09:40: Exception.HResult: -2146233088
2016-09-27 17:09:40: Exception.StackTrace:
2016-09-27 17:09:40: at Deadline.LicenseForwarder.LicenseForwarderUtils.a(LicenseForwarderInfoSettings A_0, String[] A_1, LicenseForwarderRequestType A_2)
2016-09-27 17:09:40: at Deadline.LicenseForwarder.LicenseForwarderUtils.UnregisterMachine(LicenseForwarderInfoSettings licenseForwarder)
2016-09-27 17:09:40: at Deadline.LicenseForwarder.LicenseForwarderUtils.UnregisterMachine()
2016-09-27 17:09:40: at Deadline.Configuration.Plugins.PluginManager.UnloadPlugin()
2016-09-27 17:09:40: at Deadline.Slaves.SlaveSchedulerThread.h()
2016-09-27 17:09:40: <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2016-09-27 17:09:40: Scheduler - Returning all limit stubs.
2016-09-27 17:09:40: Scheduler - returning nuke_r
2016-09-27 17:09:42: Scheduler - returning 57eafa93b5b7e231f407f2e7
2016-09-27 17:09:42: Scheduler - Performing Job scan on Primary Pools with scheduling order Pool, Priority, First-in First-out
2016-09-27 17:09:43: 2: STDOUT: DeepWrite1 .1.2.4
2016-09-27 17:09:43: 2: STDOUT: DeepWrite1 took 5.90 seconds
2016-09-27 17:09:43: 2: STDOUT: Total render time: 5.90 seconds
2016-09-27 17:09:43: 2: STDOUT: .9Frame 3797 (3 of 10)
2016-09-27 17:09:44: Scheduler - Slave is on the excluded list for the 'nuke' limit.
2016-09-27 17:09:44: Scheduler - Successfully dequeued 3 task(s). Returning.
2016-09-27 17:09:44: Scheduler - Unexpected Error Occurred
2016-09-27 17:09:44: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2016-09-27 17:09:44: Exception Details
2016-09-27 17:09:44: IndexOutOfRangeException -- Index was outside the bounds of the array.
2016-09-27 17:09:44: Exception.Data: ( )
2016-09-27 17:09:44: Exception.TargetSite: Void SetTask(Int32, Deadline.Jobs.Task)
2016-09-27 17:09:44: Exception.Source: deadline
2016-09-27 17:09:44: Exception.HResult: -2146233080
2016-09-27 17:09:44: Exception.StackTrace:
2016-09-27 17:09:44: at Deadline.Slaves.SlaveState.SetTask(Int32 index, Task task)
2016-09-27 17:09:44: at Deadline.Controllers.DataController.DequeueTasks(SlaveState slaveState, SlaveSettings slaveSettings, Boolean verboseLogging, Boolean secondaryScan)
2016-09-27 17:09:44: <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2016-09-27 17:09:44: Scheduler - Performing Job scan on Secondary Pools with scheduling order Pool, Priority, First-in First-out
2016-09-27 17:09:44: Scheduler - Slave is on the excluded list for the 'nuke' limit.
2016-09-27 17:09:45: Scheduler - Successfully dequeued 3 task(s). Returning.
2016-09-27 17:09:45: Scheduler - Unexpected Error Occurred
2016-09-27 17:09:45: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2016-09-27 17:09:45: Exception Details
2016-09-27 17:09:45: IndexOutOfRangeException -- Index was outside the bounds of the array.
2016-09-27 17:09:45: Exception.Data: ( )
2016-09-27 17:09:45: Exception.TargetSite: Void SetTask(Int32, Deadline.Jobs.Task)
2016-09-27 17:09:45: Exception.Source: deadline
2016-09-27 17:09:45: Exception.HResult: -2146233080
2016-09-27 17:09:45: Exception.StackTrace:
2016-09-27 17:09:45: at Deadline.Slaves.SlaveState.SetTask(Int32 index, Task task)
2016-09-27 17:09:45: at Deadline.Controllers.DataController.DequeueTasks(SlaveState slaveState, SlaveSettings slaveSettings, Boolean verboseLogging, Boolean secondaryScan)
2016-09-27 17:09:45: <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2016-09-27 17:09:45: Message From License Forwarder: Success: Machine unregistered.
2016-09-27 17:09:46: Message From License Forwarder: Success: Machine unregistered.
2016-09-27 17:09:47: Scheduler Thread - exception occurred:
2016-09-27 17:09:47: Scheduler Thread - Unexpected Error Occurred
2016-09-27 17:09:47: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2016-09-27 17:09:47: Exception Details
2016-09-27 17:09:47: LicenseForwarderConnectionException -- Unable to connect to the License Forwarder at 192.168.94.29:17084. Check to make sure that it is available.
2016-09-27 17:09:47: Exception.Data: ( )
2016-09-27 17:09:47: Exception.TargetSite: Boolean a(Deadline.LicenseForwarder.LicenseForwarderInfoSettings, System.String[], Deadline.LicenseForwarder.LicenseForwarderRequestType)
2016-09-27 17:09:47: Exception.Source: deadline
2016-09-27 17:09:47: Exception.HResult: -2146233088
2016-09-27 17:09:47: Exception.StackTrace:
2016-09-27 17:09:47: at Deadline.LicenseForwarder.LicenseForwarderUtils.a(LicenseForwarderInfoSettings A_0, String[] A_1, LicenseForwarderRequestType A_2)
2016-09-27 17:09:47: at Deadline.LicenseForwarder.LicenseForwarderUtils.UnregisterMachine(LicenseForwarderInfoSettings licenseForwarder)
2016-09-27 17:09:47: at Deadline.LicenseForwarder.LicenseForwarderUtils.UnregisterMachine()
2016-09-27 17:09:47: at Deadline.Configuration.Plugins.PluginManager.UnloadPlugin()
2016-09-27 17:09:47: at Deadline.Slaves.SlaveSchedulerThread.h()
2016-09-27 17:09:47: <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2016-09-27 17:09:47: Scheduler - Returning all limit stubs.
2016-09-27 17:09:47: Scheduler - returning nuke_r
2016-09-27 17:09:50: 2: STDOUT: DeepWrite1 .1.2
2016-09-27 17:09:50: 2: STDOUT: DeepWrite1 took 5.99 seconds
2016-09-27 17:09:50: 2: STDOUT: Total render time: 5.99 seconds
2016-09-27 17:09:50: 2: STDOUT: .9Frame 3798 (4 of 10)
2016-09-27 17:09:53: Scheduler - returning 57eafa93b5b7e231f407f2e7
2016-09-27 17:09:53: Scheduler - Performing Job scan on Primary Pools with scheduling order Pool, Priority, First-in First-out
2016-09-27 17:09:55: Scheduler - Slave is on the excluded list for the 'nuke' limit.
2016-09-27 17:09:55: Scheduler - Successfully dequeued 3 task(s). Returning.
2016-09-27 17:09:55: Scheduler - Unexpected Error Occurred
2016-09-27 17:09:55: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2016-09-27 17:09:55: Exception Details
2016-09-27 17:09:55: IndexOutOfRangeException -- Index was outside the bounds of the array.
2016-09-27 17:09:55: Exception.Data: ( )
2016-09-27 17:09:55: Exception.TargetSite: Void SetTask(Int32, Deadline.Jobs.Task)
2016-09-27 17:09:55: Exception.Source: deadline
2016-09-27 17:09:55: Exception.HResult: -2146233080
2016-09-27 17:09:55: Exception.StackTrace:
2016-09-27 17:09:55: at Deadline.Slaves.SlaveState.SetTask(Int32 index, Task task)
2016-09-27 17:09:55: at Deadline.Controllers.DataController.DequeueTasks(SlaveState slaveState, SlaveSettings slaveSettings, Boolean verboseLogging, Boolean secondaryScan)
2016-09-27 17:09:55: <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2016-09-27 17:09:55: Scheduler - Performing Job scan on Secondary Pools with scheduling order Pool, Priority, First-in First-out
2016-09-27 17:09:55: Scheduler - Slave is on the excluded list for the 'nuke' limit.
2016-09-27 17:09:55: Scheduler - Successfully dequeued 3 task(s). Returning.
2016-09-27 17:09:55: Scheduler - Unexpected Error Occurred
2016-09-27 17:09:55: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2016-09-27 17:09:55: Exception Details
2016-09-27 17:09:55: IndexOutOfRangeException -- Index was outside the bounds of the array.
2016-09-27 17:09:55: Exception.Data: ( )
2016-09-27 17:09:55: Exception.TargetSite: Void SetTask(Int32, Deadline.Jobs.Task)
2016-09-27 17:09:55: Exception.Source: deadline
2016-09-27 17:09:55: Exception.HResult: -2146233080
2016-09-27 17:09:55: Exception.StackTrace:
2016-09-27 17:09:55: at Deadline.Slaves.SlaveState.SetTask(Int32 index, Task task)
2016-09-27 17:09:55: at Deadline.Controllers.DataController.DequeueTasks(SlaveState slaveState, SlaveSettings slaveSettings, Boolean verboseLogging, Boolean secondaryScan)
2016-09-27 17:09:55: <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2016-09-27 17:09:56: Message From License Forwarder: Success: Machine unregistered.
2016-09-27 17:09:56: 2: STDOUT: DeepWrite1 .1.2.7
2016-09-27 17:09:56: 2: STDOUT: DeepWrite1 took 6.62 seconds
2016-09-27 17:09:56: 2: STDOUT: Total render time: 6.62 seconds
2016-09-27 17:09:56: 2: STDOUT: .9Frame 3799 (5 of 10)
2016-09-27 17:09:57: Message From License Forwarder: Success: Machine unregistered.
Gave this a try today. No luck yet.
Made a job where my machine just pings localhost 3600 times (runs for about an hour). Tried originally with a single Slave to see what effects it might have, everything worked swell.
Changed two things, now at three Slaves, and marked the job as interruptible so the Slaves will keep scanning for job changes while they’re rendering. That seems the best avenue to break this stuff since it’s not on by default on jobs and the Slaves should now detect the change while they’re rendering something.
Waiting on this to cook, but can you take a dump of your DB, zip it up, and send it over? You can just use mongodump --port=27080 in the db folder and it’ll create a convenient folder of all your database goods. Just e-mail it directly to me and I’ll make a tracking ticket internally so we can have all the data around. I’m really only interested in the job’s internal state, and I don’t know if archiving it will patch and problems it finds with the data.
Do I need to do a dump while it’s going crazy or can I do it now?
While crazy is highly preferred. Plus some bookmark help like the job id of the one that’s gone mad.
Also, thusly far I’ve got nothing on my test job. It’s been happily “rendering” since yesterday, and I’ve swapped the concurrent tasks a few times now.
Also, anything interesting in the job history for that last one that had problems? I’m expecting something in the Slave’s Scheduler thread has run aground as is freaking out as the Slave only has 16 render threads to work with.
I’ve got a complicated limit scheme. That’s the only thing strange I can think of. That and UBL.
Nuke_r
White: Rendernodes1,2,3,4
Exclude: Workstations 1,2,3,4
Nuke_i
White: Workstations 1,2,3,4
Exclude: Rendernodes1,2,3,4
Ok, I had to switch it back and forth a whole bunch of times… then I had one task that was just switching assignees like every 5 seconds without an error. Then it went insane.
I private messaged you a link to the database dump.
Okay. I’ve got it now and imported it into our tooling here in support for diagnosing this stuff. I"ll open up a tracking issue now for the dev team and will take a look when I can.
Oh! Also, can you send along the Slave log for the machine(s) that were causing trouble? As I said, I’m pretty sure the issue is in the scheduler thread, so hopefully there is something there.
Hey Gavin,
How many cores do your node machines have? How many slaves are running on each physical machine?
Physical cores: between 4 and 20. Mostly 4-6. VM-## 1-5 is (4) cores. i7-## 05-10 is (4) cores. VM-## 11-15 are (6) cores. VM-X## are 20 cores.
Always one slave per machine.
The relevant bits seem to be in here and I’ll email you the full log:
2016-09-27 17:11:24: 1: STDOUT: [17:11:06 Pacific Daylight Time] Read nuke script: C:/Users/renderadmin/AppData/Local/Thinkbox/Deadline8/slave/Render-i7-08/jobsData/57eafa93b5b7e231f407f2e7/thread1_tempwSQuC0/PNB_nutcracker_0010_B01.08.nk
2016-09-27 17:11:24: 0: STDOUT: DeepWrite1 .1.2.3.4.6.8.9
2016-09-27 17:11:24: 0: STDOUT: DeepWrite1 took 34.03 seconds
2016-09-27 17:11:24: 0: STDOUT: Total render time: 34 seconds
2016-09-27 17:11:24: 0: STDOUT: Frame 3709 (5 of 10)
2016-09-27 17:11:24: 0: STDOUT: DeepWrite1 .1.2.3
2016-09-27 17:11:24: 1: Stopped job: PNB_nutcracker_0010_B01.08.nk
2016-09-27 17:11:24: 1: Unloading plugin: Nuke
2016-09-27 17:11:24: 1: Encountered an error while executing plugin command of type 'Render Task'
2016-09-27 17:11:24: 1: An exception occurred: Object reference not set to an instance of an object. (Deadline.Plugins.RenderPluginException)
2016-09-27 17:11:24: 1: Could not unload plugin because: Object reference not set to an instance of an object. (System.NullReferenceException)
2016-09-27 17:11:24: Exception Details
2016-09-27 17:11:24: NullReferenceException -- Object reference not set to an instance of an object.
2016-09-27 17:11:24: Exception.Data: ( )
2016-09-27 17:11:24: Exception.TargetSite: Void b()
2016-09-27 17:11:24: Exception.Source: deadline
2016-09-27 17:11:24: Exception.HResult: -2147467261
2016-09-27 17:11:24: Exception.StackTrace:
2016-09-27 17:11:24: at Deadline.Slaves.SlaveRenderThread.b()
2016-09-27 17:11:24: at Deadline.Slaves.SlaveRenderThread.a(Exception A_0, TaskLogWriter A_1)
2016-09-27 17:11:24: 0: STDOUT: DeepWrite1 took 8.98 seconds
2016-09-27 17:11:24: 0: STDOUT: Total render time: 8.98 seconds
2016-09-27 17:11:24: 0: STDOUT: .9Frame 3885 (1 of 10)
2016-09-27 17:11:24: Scheduler Thread - Render Thread 1 threw a major error:
2016-09-27 17:11:24: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2016-09-27 17:11:24: Exception Details
2016-09-27 17:11:24: RenderPluginException -- Object reference not set to an instance of an object.
2016-09-27 17:11:24: RenderPluginException.Cause: JobError (2)
2016-09-27 17:11:24: RenderPluginException.Level: Major (1)
2016-09-27 17:11:24: RenderPluginException.HasSlaveLog: True
2016-09-27 17:11:24: RenderPluginException.SlaveLogFileName: C:\ProgramData\Thinkbox\Deadline8\logs\deadlineslave_renderthread_1-Render-i7-08-0000.log
2016-09-27 17:11:24: Exception.Data: ( )
2016-09-27 17:11:24: Exception.TargetSite: Void a(Deadline.Net.DeadlineMessage)
2016-09-27 17:11:24: Exception.Source: deadline
2016-09-27 17:11:24: Exception.HResult: -2146233088
2016-09-27 17:11:24: Exception.StackTrace:
2016-09-27 17:11:24: at Deadline.Plugins.SandboxedPlugin.a(DeadlineMessage A_0)
2016-09-27 17:11:24: at Deadline.Plugins.SandboxedPlugin.RenderTask(String taskId, Int32 startFrame, Int32 endFrame)
2016-09-27 17:11:24: at Deadline.Slaves.SlaveRenderThread.a(TaskLogWriter A_0)
2016-09-27 17:11:24: <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2016-09-27 17:11:24: Scheduler Thread - Seconds before next task scan: 2
2016-09-27 17:11:24: Message From License Forwarder: Success: Machine registered.
2016-09-27 17:11:24: Scheduler - Performing Job scan on Primary Pools with scheduling order Pool, Priority, First-in First-out
2016-09-27 17:11:24: Scheduler - Slave is on the excluded list for the 'nuke' limit.
2016-09-27 17:11:24: Scheduler - Successfully dequeued 1 task(s). Returning.
2016-09-27 17:11:24: Message From License Forwarder: Success: Machine unregistered.
2016-09-27 17:11:24: 0: STDOUT: DeepWrite1 .1.2
2016-09-27 17:11:24: 0: STDOUT: DeepWrite1 took 3.97 seconds
2016-09-27 17:11:24: 0: STDOUT: Total render time: 3.97 seconds
2016-09-27 17:11:24: 0: STDOUT: .9Frame 3886 (2 of 10)
2016-09-27 17:11:24: 1: Got task!
2016-09-27 17:11:24: Message From License Forwarder: Success: Machine unregistered.
2016-09-27 17:11:24: 1: Loaded plugin Nuke (\\sfs-dc\deadlinerepository8\plugins\Nuke)
2016-09-27 17:11:24: Scheduler Thread - exception occurred:
2016-09-27 17:11:24: Scheduler Thread - Unexpected Error Occurred
2016-09-27 17:11:24: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2016-09-27 17:11:24: Exception Details
2016-09-27 17:11:24: LicenseForwarderConnectionException -- Unable to connect to the License Forwarder at 192.168.94.29:17084. Check to make sure that it is available.
2016-09-27 17:11:24: Exception.Data: ( )
2016-09-27 17:11:24: Exception.TargetSite: Boolean a(Deadline.LicenseForwarder.LicenseForwarderInfoSettings, System.String[], Deadline.LicenseForwarder.LicenseForwarderRequestType)
2016-09-27 17:11:24: Exception.Source: deadline
2016-09-27 17:11:24: Exception.HResult: -2146233088
2016-09-27 17:11:24: Exception.StackTrace:
2016-09-27 17:11:24: at Deadline.LicenseForwarder.LicenseForwarderUtils.a(LicenseForwarderInfoSettings A_0, String[] A_1, LicenseForwarderRequestType A_2)
2016-09-27 17:11:24: at Deadline.LicenseForwarder.LicenseForwarderUtils.UnregisterMachine(LicenseForwarderInfoSettings licenseForwarder)
2016-09-27 17:11:24: at Deadline.LicenseForwarder.LicenseForwarderUtils.UnregisterMachine()
2016-09-27 17:11:24: at Deadline.Configuration.Plugins.PluginManager.UnloadPlugin()
2016-09-27 17:11:24: at Deadline.Slaves.SlaveSchedulerThread.h()
2016-09-27 17:11:24: <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2016-09-27 17:11:24: Scheduler - Returning all limit stubs.
2016-09-27 17:11:24: Scheduler - returning nuke_r
2016-09-27 17:11:24: 1: Start Job timeout is disabled.
2016-09-27 17:11:24: 1: Task timeout is disabled.
2016-09-27 17:11:24: 1: Loaded job: PNB_nutcracker_0010_B01.08.nk (57eafa93b5b7e231f407f2e7)
2016-09-27 17:11:24: 1: Skipping drive mapping because they have already been mapped for this job
2016-09-27 17:11:24: 1: Plugin rendering frame(s): 3905-3914
2016-09-27 17:11:24: Scheduler Thread - Task "134_3885-3894" could not be found because task has been modified:
2016-09-27 17:11:24: current status = Rendering, new status = Queued
2016-09-27 17:11:24: current slave = Render-i7-08, new slave =
2016-09-27 17:11:24: current frames = 3885-3894, new frames = 3885-3894
2016-09-27 17:11:24: Scheduler Thread - Cancelling task...
2016-09-27 17:11:24: 0: Executing plugin command of type 'Cancel Task'
2016-09-27 17:11:24: 0: Done executing plugin command of type 'Cancel Task'
2016-09-27 17:11:24: 0: Done executing plugin command of type 'Render Task'
2016-09-27 17:11:24: 0: In the process of canceling current task: ignoring exception thrown by PluginLoader
2016-09-27 17:11:24: 0: Unloading plugin: Nuke
2016-09-27 17:11:25: 1: Executing plugin command of type 'Initialize Plugin'
2016-09-27 17:11:25: 1: INFO: Executing plugin script 'C:\Users\renderadmin\AppData\Local\Thinkbox\Deadline8\slave\render-i7-08\plugins\57eafa93b5b7e231f407f2e7\Nuke.py'
2016-09-27 17:11:25: 1: INFO: About: Nuke Plugin for Deadline
2016-09-27 17:11:25: 1: INFO: The job's environment will be merged with the current environment before rendering
2016-09-27 17:11:25: 1: Done executing plugin command of type 'Initialize Plugin'
2016-09-27 17:11:25: 1: Executing plugin command of type 'Start Job'
2016-09-27 17:11:25: 1: INFO: Prepping OFX cache
2016-09-27 17:11:25: 1: INFO: Checking Nuke temp path: C:\Users\renderadmin\AppData\Local\Temp\nuke
2016-09-27 17:11:25: 1: INFO: Path already exists
2016-09-27 17:11:25: 1: INFO: OFX cache prepped
2016-09-27 17:11:25: 1: INFO: Starting monitored managed process Nuke
2016-09-27 17:11:25: 1: INFO: Stdout Redirection Enabled: True
2016-09-27 17:11:25: 1: INFO: Stdout Handling Enabled: True
2016-09-27 17:11:25: 1: INFO: Popup Handling Enabled: True
2016-09-27 17:11:25: 1: INFO: QT Popup Handling Enabled: False
2016-09-27 17:11:25: 1: INFO: WindowsForms10.Window.8.app.* Popup Handling Enabled: False
2016-09-27 17:11:25: 1: INFO: Using Process Tree: True
2016-09-27 17:11:25: 1: INFO: Hiding DOS Window: True
2016-09-27 17:11:25: 1: INFO: Creating New Console: False
2016-09-27 17:11:25: 1: INFO: Running as user: renderadmin
2016-09-27 17:11:25: 1: INFO: Executable: "\\sfs-dc\tools\Program Files\Nuke10.0v1\Nuke10.0.exe"
2016-09-27 17:11:25: 1: INFO: Rendering with NukeX
2016-09-27 17:11:25: 1: INFO: Rendering using Full Resolution
2016-09-27 17:11:25: 1: INFO: Enabling GPU rendering
2016-09-27 17:11:25: 1: INFO: Argument: -V 2 --nukex -f --gpu -t "C:\Users\renderadmin\AppData\Local\Thinkbox\Deadline8\slave\render-i7-08\jobsData\57eafa93b5b7e231f407f2e7\thread1_tempIzyzY0\PNB_nutcracker_0010_B01.08.nk"
2016-09-27 17:11:25: 1: INFO: Full Command: "\\sfs-dc\tools\Program Files\Nuke10.0v1\Nuke10.0.exe" -V 2 --nukex -f --gpu -t "C:\Users\renderadmin\AppData\Local\Thinkbox\Deadline8\slave\render-i7-08\jobsData\57eafa93b5b7e231f407f2e7\thread1_tempIzyzY0\PNB_nutcracker_0010_B01.08.nk"
2016-09-27 17:11:25: 1: INFO: Startup Directory: "\\sfs-dc\tools\Program Files\Nuke10.0v1"
2016-09-27 17:11:25: 1: INFO: Process Priority: BelowNormal
2016-09-27 17:11:25: 1: INFO: Process Affinity: default
2016-09-27 17:11:25: 1: INFO: Process is now running
2016-09-27 17:11:25: 1: Done executing plugin command of type 'Start Job'
2016-09-27 17:11:25: 1: Executing plugin command of type 'Render Task'
2016-09-27 17:11:25: 1: INFO: Rendering write node DeepWrite1
2016-09-27 17:11:25: 1: STDOUT: NOTE : NukeX functionality is only available with an interactive license. If you wish to
2016-09-27 17:11:25: 1: STDOUT: use this, please specify '--nukex -i' for an interactive license in terminal mode.
2016-09-27 17:11:25: 1: STDOUT: Nuke 10.0v1, 64 bit, built Apr 23 2016.
2016-09-27 17:11:25: 1: STDOUT: Copyright (c) 2016 The Foundry Visionmongers Ltd. All Rights Reserved.
2016-09-27 17:11:25: Scheduler - returning 57eafa93b5b7e231f407f2e7
2016-09-27 17:11:25: Scheduler - Performing Job scan on Primary Pools with scheduling order Pool, Priority, First-in First-out
2016-09-27 17:11:25: 1: STDOUT: DeepWrite1 took 6.73 seconds
2016-09-27 17:11:25: 1: STDOUT: Total render time: 6.73 seconds
2016-09-27 17:11:25: 1: STDOUT: .9Frame 4102 (8 of 10)
2016-09-27 17:11:25: 1: STDOUT: DeepWrite1 .1.2.3
2016-09-27 17:11:25: 1: STDOUT: DeepWrite1 took 6.97 seconds
2016-09-27 17:11:25: 1: STDOUT: Total render time: 6.97 seconds
2016-09-27 17:11:25: 1: STDOUT: .9Frame 4103 (9 of 10)
2016-09-27 17:11:25: 1: STDOUT: DeepWrite1 .1.2.3.8
2016-09-27 17:11:25: 1: STDOUT: DeepWrite1 took 10.62 seconds
2016-09-27 17:11:25: 1: STDOUT: Total render time: 11 seconds
2016-09-27 17:11:25: 1: STDOUT: .9Frame 4104 (10 of 10)
2016-09-27 17:11:25: 1: STDOUT: READY FOR INPUT
2016-09-27 17:11:25: 1: Done executing plugin command of type 'Render Task'
2016-09-27 17:11:25: 1: Executing plugin command of type 'End Job'
2016-09-27 17:11:25: 1: INFO: Ending Nuke Job
2016-09-27 17:11:25: 1: Done executing plugin command of type 'End Job'
2016-09-27 17:11:26: 1: STDOUT: A license for 'nuke' was not found.
2016-09-27 17:11:26: 1: STDOUT: FOUNDRY LICENSE ERROR REPORT
2016-09-27 17:11:26: 1: STDOUT: ----------------------------
2016-09-27 17:11:26: 1: STDOUT: Timestamp: Tue Sep 27 17:11:26 2016
2016-09-27 17:11:26: 1: STDOUT: License(s) Requested:
2016-09-27 17:11:26: 1: STDOUT: nuke 2016.0423 render only with options all
2016-09-27 17:11:26: 1: STDOUT: Extended Info: None Provided
2016-09-27 17:11:26: 1: STDOUT: Host : Render-i7-08
2016-09-27 17:11:26: 1: STDOUT: System ID(s) : e840f23dcddc
2016-09-27 17:11:26: 1: STDOUT: RLM Environment Info: C:\Users\renderadmin\FoundryLicensing\e840f23dcddc;4101@192.168.94.100;C:\Program Files\The Foundry\RLM;C:\ProgramData\The Foundry\RLM;C:\Program Files (x86)\The Foundry\RLM;C:\Users\renderadmin\FoundryLicensing
2016-09-27 17:11:26: 1: STDOUT: Reason for failure:
2016-09-27 17:11:26: 1: STDOUT: RLM : Failed to understand response from server.
2016-09-27 17:11:26: 1: STDOUT: FLEXlm : Cannot find license file.
2016-09-27 17:11:26: 1: STDOUT: The license files (or license server system network addresses) attempted are
2016-09-27 17:11:26: 1: STDOUT: listed below. Use LM_LICENSE_FILE to use a different license file,
2016-09-27 17:11:26: 1: STDOUT: or contact your software provider for a license file.
2016-09-27 17:11:26: 1: STDOUT: Feature: nuke_r
2016-09-27 17:11:26: 1: STDOUT: Filename: C:\Program Files (x86)\The Foundry\FLEXlm
2016-09-27 17:11:26: 1: STDOUT: License path: C:\Program Files\The Foundry\FLEXlm;C:\ProgramData\The -
2016-09-27 17:11:26: 1: STDOUT: Foundry\FLEXlm;C:\Program Files (x86)\The Foundry\FLEXlm
2016-09-27 17:11:26: 1: STDOUT: FLEXnet Licensing error:-1,359. System Error: 2 "No such file or directory"
2016-09-27 17:11:26: 1: Done executing plugin command of type 'Render Task'
2016-09-27 17:11:26: 1: An exception occurred: Error: FLEXnet Licensing error:-1,359. System Error: 2 "No such file or directory"
2016-09-27 17:11:26: at Deadline.Plugins.PluginWrapper.RenderTasks(String taskId, Int32 startFrame, Int32 endFrame, String& outMessage, AbortLevel& abortLevel) (Deadline.Plugins.RenderPluginException)
2016-09-27 17:11:26: 1: Unloading plugin: Nuke
2016-09-27 17:11:27: Scheduler - Slave is on the excluded list for the 'nuke' limit.
2016-09-27 17:11:27: Scheduler - Successfully dequeued 1 task(s). Returning.
2016-09-27 17:11:27: Scheduler - Unexpected Error Occurred
2016-09-27 17:11:27: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2016-09-27 17:11:27: Exception Details
2016-09-27 17:11:27: IndexOutOfRangeException -- Index was outside the bounds of the array.
2016-09-27 17:11:27: Exception.Data: ( )
2016-09-27 17:11:27: Exception.TargetSite: Void SetTask(Int32, Deadline.Jobs.Task)
2016-09-27 17:11:27: Exception.Source: deadline
2016-09-27 17:11:27: Exception.HResult: -2146233080
2016-09-27 17:11:27: Exception.StackTrace:
2016-09-27 17:11:27: at Deadline.Slaves.SlaveState.SetTask(Int32 index, Task task)
2016-09-27 17:11:27: at Deadline.Controllers.DataController.DequeueTasks(SlaveState slaveState, SlaveSettings slaveSettings, Boolean verboseLogging, Boolean secondaryScan)
2016-09-27 17:11:27: <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2016-09-27 17:11:27: Scheduler - Performing Job scan on Secondary Pools with scheduling order Pool, Priority, First-in First-out
2016-09-27 17:11:27: Scheduler - Slave is on the excluded list for the 'nuke' limit.
2016-09-27 17:11:27: Scheduler - Successfully dequeued 1 task(s). Returning.
2016-09-27 17:11:27: Scheduler - Unexpected Error Occurred
2016-09-27 17:11:27: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2016-09-27 17:11:27: Exception Details
2016-09-27 17:11:27: IndexOutOfRangeException -- Index was outside the bounds of the array.
2016-09-27 17:11:27: Exception.Data: ( )
2016-09-27 17:11:27: Exception.TargetSite: Void SetTask(Int32, Deadline.Jobs.Task)
2016-09-27 17:11:27: Exception.Source: deadline
2016-09-27 17:11:27: Exception.HResult: -2146233080
2016-09-27 17:11:27: Exception.StackTrace:
2016-09-27 17:11:27: at Deadline.Slaves.SlaveState.SetTask(Int32 index, Task task)
2016-09-27 17:11:27: at Deadline.Controllers.DataController.DequeueTasks(SlaveState slaveState, SlaveSettings slaveSettings, Boolean verboseLogging, Boolean secondaryScan)
2016-09-27 17:11:27: <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2016-09-27 17:11:28: Message From License Forwarder: Success: Machine unregistered.
2016-09-27 17:11:29: Message From License Forwarder: Success: Machine unregistered.
2016-09-27 17:11:30: Scheduler Thread - exception occurred:
2016-09-27 17:11:30: Scheduler Thread - Unexpected Error Occurred
2016-09-27 17:11:30: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2016-09-27 17:11:30: Exception Details
2016-09-27 17:11:30: LicenseForwarderConnectionException -- Unable to connect to the License Forwarder at 192.168.94.29:17084. Check to make sure that it is available.
2016-09-27 17:11:30: Exception.Data: ( )
2016-09-27 17:11:30: Exception.TargetSite: Boolean a(Deadline.LicenseForwarder.LicenseForwarderInfoSettings, System.String[], Deadline.LicenseForwarder.LicenseForwarderRequestType)
2016-09-27 17:11:30: Exception.Source: deadline
2016-09-27 17:11:30: Exception.HResult: -2146233088
2016-09-27 17:11:30: Exception.StackTrace:
2016-09-27 17:11:30: at Deadline.LicenseForwarder.LicenseForwarderUtils.a(LicenseForwarderInfoSettings A_0, String[] A_1, LicenseForwarderRequestType A_2)
2016-09-27 17:11:30: at Deadline.LicenseForwarder.LicenseForwarderUtils.UnregisterMachine(LicenseForwarderInfoSettings licenseForwarder)
2016-09-27 17:11:30: at Deadline.LicenseForwarder.LicenseForwarderUtils.UnregisterMachine()
2016-09-27 17:11:30: at Deadline.Configuration.Plugins.PluginManager.UnloadPlugin()
2016-09-27 17:11:30: at Deadline.Slaves.SlaveSchedulerThread.h()
2016-09-27 17:11:30: <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2016-09-27 17:11:30: Scheduler - Returning all limit stubs.
2016-09-27 17:11:30: Scheduler - returning nuke_r
2016-09-27 17:11:32: Scheduler - returning 57eafa93b5b7e231f407f2e7
2016-09-27 17:11:32: Scheduler - Performing Job scan on Primary Pools with scheduling order Pool, Priority, First-in First-out
2016-09-27 17:11:35: Scheduler - Slave is on the excluded list for the 'nuke' limit.
2016-09-27 17:11:35: Scheduler - Successfully dequeued 2 task(s). Returning.
2016-09-27 17:11:35: Scheduler - Unexpected Error Occurred
2016-09-27 17:11:35: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2016-09-27 17:11:35: Exception Details
2016-09-27 17:11:35: IndexOutOfRangeException -- Index was outside the bounds of the array.
2016-09-27 17:11:35: Exception.Data: ( )
2016-09-27 17:11:35: Exception.TargetSite: Void SetTask(Int32, Deadline.Jobs.Task)
2016-09-27 17:11:35: Exception.Source: deadline
2016-09-27 17:11:35: Exception.HResult: -2146233080
2016-09-27 17:11:35: Exception.StackTrace:
2016-09-27 17:11:35: at Deadline.Slaves.SlaveState.SetTask(Int32 index, Task task)
2016-09-27 17:11:35: at Deadline.Controllers.DataController.DequeueTasks(SlaveState slaveState, SlaveSettings slaveSettings, Boolean verboseLogging, Boolean secondaryScan)
2016-09-27 17:11:35: <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2016-09-27 17:11:35: Scheduler - Performing Job scan on Secondary Pools with scheduling order Pool, Priority, First-in First-out
2016-09-27 17:11:35: Scheduler - Slave is on the excluded list for the 'nuke' limit.
2016-09-27 17:11:35: Scheduler - Successfully dequeued 2 task(s). Returning.
2016-09-27 17:11:35: Scheduler - Unexpected Error Occurred
2016-09-27 17:11:35: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2016-09-27 17:11:35: Exception Details
2016-09-27 17:11:35: IndexOutOfRangeException -- Index was outside the bounds of the array.
2016-09-27 17:11:35: Exception.Data: ( )
2016-09-27 17:11:35: Exception.TargetSite: Void SetTask(Int32, Deadline.Jobs.Task)
2016-09-27 17:11:35: Exception.Source: deadline
2016-09-27 17:11:35: Exception.HResult: -2146233080
2016-09-27 17:11:35: Exception.StackTrace:
2016-09-27 17:11:35: at Deadline.Slaves.SlaveState.SetTask(Int32 index, Task task)
2016-09-27 17:11:35: at Deadline.Controllers.DataController.DequeueTasks(SlaveState slaveState, SlaveSettings slaveSettings, Boolean verboseLogging, Boolean secondaryScan)
2016-09-27 17:11:35: <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2016-09-27 17:11:35: Message From License Forwarder: Success: Machine unregistered.
2016-09-27 17:11:37: Message From License Forwarder: Success: Machine unregistered.
2016-09-27 17:11:38: Scheduler Thread - exception occurred:
2016-09-27 17:11:38: Scheduler Thread - Unexpected Error Occurred
2016-09-27 17:11:38: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2016-09-27 17:11:38: Exception Details
2016-09-27 17:11:38: LicenseForwarderConnectionException -- Unable to connect to the License Forwarder at 192.168.94.29:17084. Check to make sure that it is available.
2016-09-27 17:11:38: Exception.Data: ( )
2016-09-27 17:11:38: Exception.TargetSite: Boolean a(Deadline.LicenseForwarder.LicenseForwarderInfoSettings, System.String[], Deadline.LicenseForwarder.LicenseForwarderRequestType)
2016-09-27 17:11:38: Exception.Source: deadline
2016-09-27 17:11:38: Exception.HResult: -2146233088
2016-09-27 17:11:38: Exception.StackTrace:
2016-09-27 17:11:38: at Deadline.LicenseForwarder.LicenseForwarderUtils.a(LicenseForwarderInfoSettings A_0, String[] A_1, LicenseForwarderRequestType A_2)
2016-09-27 17:11:38: at Deadline.LicenseForwarder.LicenseForwarderUtils.UnregisterMachine(LicenseForwarderInfoSettings licenseForwarder)
2016-09-27 17:11:38: at Deadline.LicenseForwarder.LicenseForwarderUtils.UnregisterMachine()
2016-09-27 17:11:38: at Deadline.Configuration.Plugins.PluginManager.UnloadPlugin()
2016-09-27 17:11:38: at Deadline.Slaves.SlaveSchedulerThread.h()
2016-09-27 17:11:38: <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2016-09-27 17:11:38: Scheduler - Returning all limit stubs.
2016-09-27 17:11:38: Scheduler - returning nuke_r
2016-09-27 17:11:40: Scheduler - returning 57eafa93b5b7e231f407f2e7
2016-09-27 17:11:40: Scheduler - Performing Job scan on Primary Pools with scheduling order Pool, Priority, First-in First-out
2016-09-27 17:11:42: Scheduler - Slave is on the excluded list for the 'nuke' limit.
2016-09-27 17:11:42: Scheduler - Successfully dequeued 2 task(s). Returning.
2016-09-27 17:11:42: Scheduler - Unexpected Error Occurred
2016-09-27 17:11:42: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2016-09-27 17:11:42: Exception Details
2016-09-27 17:11:42: IndexOutOfRangeException -- Index was outside the bounds of the array.
2016-09-27 17:11:42: Exception.Data: ( )
2016-09-27 17:11:42: Exception.TargetSite: Void SetTask(Int32, Deadline.Jobs.Task)
2016-09-27 17:11:42: Exception.Source: deadline
2016-09-27 17:11:42: Exception.HResult: -2146233080
2016-09-27 17:11:42: Exception.StackTrace:
2016-09-27 17:11:42: at Deadline.Slaves.SlaveState.SetTask(Int32 index, Task task)
2016-09-27 17:11:42: at Deadline.Controllers.DataController.DequeueTasks(SlaveState slaveState, SlaveSettings slaveSettings, Boolean verboseLogging, Boolean secondaryScan)
2016-09-27 17:11:42: <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2016-09-27 17:11:42: Scheduler - Performing Job scan on Secondary Pools with scheduling order Pool, Priority, First-in First-out
2016-09-27 17:11:42: Scheduler - Slave is on the excluded list for the 'nuke' limit.
2016-09-27 17:11:42: Scheduler - Successfully dequeued 2 task(s). Returning.
2016-09-27 17:11:42: Scheduler - Unexpected Error Occurred
2016-09-27 17:11:42: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2016-09-27 17:11:42: Exception Details
2016-09-27 17:11:42: IndexOutOfRangeException -- Index was outside the bounds of the array.
2016-09-27 17:11:42: Exception.Data: ( )
2016-09-27 17:11:42: Exception.TargetSite: Void SetTask(Int32, Deadline.Jobs.Task)
2016-09-27 17:11:42: Exception.Source: deadline
2016-09-27 17:11:42: Exception.HResult: -2146233080
2016-09-27 17:11:42: Exception.StackTrace:
2016-09-27 17:11:42: at Deadline.Slaves.SlaveState.SetTask(Int32 index, Task task)
2016-09-27 17:11:42: at Deadline.Controllers.DataController.DequeueTasks(SlaveState slaveState, SlaveSettings slaveSettings, Boolean verboseLogging, Boolean secondaryScan)
2016-09-27 17:11:42: <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2016-09-27 17:11:42: Message From License Forwarder: Success: Machine unregistered.
2016-09-27 17:11:43: 0: STDOUT: DeepWrite1 .1.2.3.4
2016-09-27 17:11:43: 0: STDOUT: DeepWrite1 took 29.41 seconds
2016-09-27 17:11:43: 0: STDOUT: Total render time: 29 seconds
2016-09-27 17:11:43: 0: STDOUT: .9Frame 3710 (6 of 10)
2016-09-27 17:11:44: Message From License Forwarder: Success: Machine unregistered.
2016-09-27 17:11:45: Scheduler Thread - exception occurred:
2016-09-27 17:11:45: Scheduler Thread - Unexpected Error Occurred
After investigating, we found that it’s possible for there to be a dequeue loop if the Slave failed to talk to the license forwarder (it wouldn’t be able to unload the plugin). We’ve fixed that in 8.0, so it should be in the next service pack.