consistently stalled slave

[im_thatoneguy]

2015-06-26 22:30:24: 0: INFO: Killed ADSK Communication Center process via Callback: #preRender
2015-06-26 22:53:06: Scheduler Thread - Task “49_2905-2905” could not be found because task has been modified:
2015-06-26 22:53:06: current status = Rendering, new status = Rendering
2015-06-26 22:53:06: current slave = Render-i7-13, new slave = Daedalus
2015-06-26 22:53:06: current frames = 2905-2905, new frames = 2905-2905
2015-06-26 22:53:06: Scheduler Thread - Cancelling task…
2015-06-26 22:54:00: 0: In the process of canceling current task: ignoring exception thrown by PluginLoader
2015-06-26 22:54:04: 0: Unloading plugin: 3dsmax
2015-06-26 22:54:32: Scheduler Thread - In the process of canceling current tasks: ignoring exception thrown by render thread 0
2015-06-26 22:54:32: Scheduler Thread - Seconds before next job scan: 2
2015-06-26 22:55:05: Scheduler - Performing Job scan on Primary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:55:07: Scheduler - Slave has been marked bad for job 558df36ab5b7e178802f5633, skipping this job.
2015-06-26 22:55:07: Scheduler - Preliminary check: Slave is blacklisted for 558df605b5b7e169a82ec649 limit.
2015-06-26 22:55:07: Scheduler - Performing Job scan on Secondary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:55:07: Scheduler - Job chooser found no jobs.
2015-06-26 22:55:07: 0: Shutdown
2015-06-26 22:55:07: 0: Exited ThreadMain(), cleaning up…
2015-06-26 22:55:09: Scheduler - Returning all limit stubs.
2015-06-26 22:55:09: Scheduler - returning 558df36ab5b7e178802f5633
2015-06-26 22:55:31: Loading event plugin ConfigSlave (\#####\custom\events\ConfigSlave)
2015-06-26 22:55:31: Scheduler Thread - Seconds before next job scan: 5
2015-06-26 22:55:39: Scheduler - Performing Job scan on Primary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:55:39: Scheduler - Slave has been marked bad for job 558df36ab5b7e178802f5633, skipping this job.
2015-06-26 22:55:39: Scheduler - Preliminary check: Slave is blacklisted for 558df605b5b7e169a82ec649 limit.
2015-06-26 22:55:39: Scheduler - Performing Job scan on Secondary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:55:39: Scheduler - Job chooser found no jobs.
2015-06-26 22:55:40: Scheduler Thread - Seconds before next job scan: 5
2015-06-26 22:55:49: Scheduler - Performing Job scan on Primary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:55:49: Scheduler - Slave has been marked bad for job 558df36ab5b7e178802f5633, skipping this job.
2015-06-26 22:55:49: Scheduler - Preliminary check: Slave is blacklisted for 558df605b5b7e169a82ec649 limit.
2015-06-26 22:55:49: Scheduler - Performing Job scan on Secondary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:55:49: Scheduler - Job chooser found no jobs.
2015-06-26 22:55:50: Scheduler Thread - Seconds before next job scan: 7
2015-06-26 22:55:57: Scheduler - Performing Job scan on Primary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:55:57: Scheduler - Slave has been marked bad for job 558df36ab5b7e178802f5633, skipping this job.
2015-06-26 22:55:57: Scheduler - Preliminary check: Slave is blacklisted for 558df605b5b7e169a82ec649 limit.
2015-06-26 22:55:57: Scheduler - Performing Job scan on Secondary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:55:57: Scheduler - Job chooser found no jobs.
2015-06-26 22:55:58: Scheduler Thread - Seconds before next job scan: 7
2015-06-26 22:56:05: Scheduler - Performing Job scan on Primary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:56:05: Scheduler - Slave has been marked bad for job 558df36ab5b7e178802f5633, skipping this job.
2015-06-26 22:56:05: Scheduler - Preliminary check: Slave is blacklisted for 558df605b5b7e169a82ec649 limit.
2015-06-26 22:56:05: Scheduler - Performing Job scan on Secondary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:56:05: Scheduler - Job chooser found no jobs.
2015-06-26 22:56:06: Scheduler Thread - Seconds before next job scan: 7
2015-06-26 22:56:13: Scheduler - Performing Job scan on Primary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:56:13: Scheduler - Slave has been marked bad for job 558df36ab5b7e178802f5633, skipping this job.
2015-06-26 22:56:13: Scheduler - Preliminary check: Slave is blacklisted for 558df605b5b7e169a82ec649 limit.
2015-06-26 22:56:13: Scheduler - Performing Job scan on Secondary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:56:13: Scheduler - Job chooser found no jobs.
2015-06-26 22:56:14: Scheduler Thread - Seconds before next job scan: 7
2015-06-26 22:56:21: Scheduler - Performing Job scan on Primary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:56:21: Scheduler - Slave has been marked bad for job 558df36ab5b7e178802f5633, skipping this job.
2015-06-26 22:56:21: Scheduler - Preliminary check: Slave is blacklisted for 558df605b5b7e169a82ec649 limit.
2015-06-26 22:56:21: Scheduler - Performing Job scan on Secondary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:56:21: Scheduler - Job chooser found no jobs.
2015-06-26 22:56:22: Scheduler Thread - Seconds before next job scan: 5
2015-06-26 22:56:27: Scheduler - Performing Job scan on Primary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:56:27: Scheduler - Slave has been marked bad for job 558df36ab5b7e178802f5633, skipping this job.
2015-06-26 22:56:27: Scheduler - Preliminary check: Slave is blacklisted for 558df605b5b7e169a82ec649 limit.
2015-06-26 22:56:27: Scheduler - Performing Job scan on Secondary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:56:27: Scheduler - Job chooser found no jobs.
2015-06-26 22:56:28: Scheduler Thread - Seconds before next job scan: 6
2015-06-26 22:56:35: Scheduler - Performing Job scan on Primary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:56:35: Scheduler - Slave has been marked bad for job 558df36ab5b7e178802f5633, skipping this job.
2015-06-26 22:56:35: Scheduler - Preliminary check: Slave is blacklisted for 558df605b5b7e169a82ec649 limit.
2015-06-26 22:56:35: Scheduler - Performing Job scan on Secondary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:56:35: Scheduler - Job chooser found no jobs.
2015-06-26 22:56:36: Scheduler Thread - Seconds before next job scan: 6
2015-06-26 22:56:42: Scheduler - Performing Job scan on Primary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:56:42: Scheduler - Slave has been marked bad for job 558df36ab5b7e178802f5633, skipping this job.
2015-06-26 22:56:42: Scheduler - Preliminary check: Slave is blacklisted for 558df605b5b7e169a82ec649 limit.
2015-06-26 22:56:42: Scheduler - Performing Job scan on Secondary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:56:42: Scheduler - Job chooser found no jobs.
2015-06-26 22:56:43: Scheduler Thread - Seconds before next job scan: 5
2015-06-26 22:56:48: Scheduler - Performing Job scan on Primary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:56:48: Scheduler - Slave has been marked bad for job 558df36ab5b7e178802f5633, skipping this job.
2015-06-26 22:56:48: Scheduler - Preliminary check: Slave is blacklisted for 558df605b5b7e169a82ec649 limit.
2015-06-26 22:56:48: Scheduler - Performing Job scan on Secondary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:56:48: Scheduler - Job chooser found no jobs.
2015-06-26 22:56:49: Scheduler Thread - Seconds before next job scan: 6
2015-06-26 22:56:55: Scheduler - Performing Job scan on Primary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:56:55: Scheduler - Slave has been marked bad for job 558df36ab5b7e178802f5633, skipping this job.
2015-06-26 22:56:55: Scheduler - Preliminary check: Slave is blacklisted for 558df605b5b7e169a82ec649 limit.
2015-06-26 22:56:55: Scheduler - Performing Job scan on Secondary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:56:55: Scheduler - Job chooser found no jobs.
2015-06-26 22:56:56: Scheduler Thread - Seconds before next job scan: 6
2015-06-26 22:57:02: Scheduler - Performing Job scan on Primary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:57:02: Scheduler - Slave has been marked bad for job 558df36ab5b7e178802f5633, skipping this job.
2015-06-26 22:57:02: Scheduler - Preliminary check: Slave is blacklisted for 558df605b5b7e169a82ec649 limit.
2015-06-26 22:57:02: Scheduler - Performing Job scan on Secondary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:57:02: Scheduler - Job chooser found no jobs.
2015-06-26 22:57:03: Scheduler Thread - Seconds before next job scan: 7
2015-06-26 22:57:11: Scheduler - Performing Job scan on Primary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:57:11: Scheduler - Slave has been marked bad for job 558df36ab5b7e178802f5633, skipping this job.
2015-06-26 22:57:11: Scheduler - Preliminary check: Slave is blacklisted for 558df605b5b7e169a82ec649 limit.
2015-06-26 22:57:11: Scheduler - Performing Job scan on Secondary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:57:11: Scheduler - Job chooser found no jobs.
2015-06-26 22:57:12: Scheduler Thread - Seconds before next job scan: 5
2015-06-26 22:57:17: Scheduler - Performing Job scan on Primary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:57:17: Scheduler - Slave has been marked bad for job 558df36ab5b7e178802f5633, skipping this job.
2015-06-26 22:57:17: Scheduler - Preliminary check: Slave is blacklisted for 558df605b5b7e169a82ec649 limit.
2015-06-26 22:57:17: Scheduler - Performing Job scan on Secondary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:57:17: Scheduler - Job chooser found no jobs.
2015-06-26 22:57:18: Scheduler Thread - Seconds before next job scan: 6
2015-06-26 22:57:24: Scheduler - Performing Job scan on Primary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:57:24: Scheduler - Slave has been marked bad for job 558df36ab5b7e178802f5633, skipping this job.
2015-06-26 22:57:24: Scheduler - Preliminary check: Slave is blacklisted for 558df605b5b7e169a82ec649 limit.
2015-06-26 22:57:24: Scheduler - Performing Job scan on Secondary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:57:24: Scheduler - Job chooser found no jobs.
2015-06-26 22:57:25: Scheduler Thread - Seconds before next job scan: 7
2015-06-26 22:57:32: Scheduler - Performing Job scan on Primary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:57:32: Scheduler - Slave has been marked bad for job 558df36ab5b7e178802f5633, skipping this job.
2015-06-26 22:57:32: Scheduler - Preliminary check: Slave is blacklisted for 558df605b5b7e169a82ec649 limit.
2015-06-26 22:57:32: Scheduler - Performing Job scan on Secondary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:57:32: Scheduler - Job chooser found no jobs.
2015-06-26 22:57:33: Scheduler Thread - Seconds before next job scan: 6
2015-06-26 22:57:39: Scheduler - Performing Job scan on Primary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:57:39: Scheduler - Slave has been marked bad for job 558df36ab5b7e178802f5633, skipping this job.
2015-06-26 22:57:39: Scheduler - Preliminary check: Slave is blacklisted for 558df605b5b7e169a82ec649 limit.
2015-06-26 22:57:39: Scheduler - Performing Job scan on Secondary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:57:39: Scheduler - Job chooser found no jobs.
2015-06-26 22:57:40: Scheduler Thread - Seconds before next job scan: 7
2015-06-26 22:57:48: Scheduler - Performing Job scan on Primary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:57:48: Scheduler - Slave has been marked bad for job 558df36ab5b7e178802f5633, skipping this job.
2015-06-26 22:57:48: Scheduler - Preliminary check: Slave is blacklisted for 558df605b5b7e169a82ec649 limit.
2015-06-26 22:57:48: Scheduler - Performing Job scan on Secondary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:57:48: Scheduler - Job chooser found no jobs.
2015-06-26 22:57:49: Scheduler Thread - Seconds before next job scan: 5
2015-06-26 22:57:54: Scheduler - Performing Job scan on Primary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:57:54: Scheduler - Slave has been marked bad for job 558df36ab5b7e178802f5633, skipping this job.
2015-06-26 22:57:54: Scheduler - Preliminary check: Slave is blacklisted for 558df605b5b7e169a82ec649 limit.
2015-06-26 22:57:54: Scheduler - Performing Job scan on Secondary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:57:54: Scheduler - Job chooser found no jobs.
2015-06-26 22:57:55: Scheduler Thread - Seconds before next job scan: 5
2015-06-26 22:58:00: Scheduler - Performing Job scan on Primary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:58:00: Scheduler - Slave has been marked bad for job 558df36ab5b7e178802f5633, skipping this job.
2015-06-26 22:58:00: Scheduler - Preliminary check: Slave is blacklisted for 558df605b5b7e169a82ec649 limit.
2015-06-26 22:58:00: Scheduler - Performing Job scan on Secondary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:58:00: Scheduler - Job chooser found no jobs.
2015-06-26 22:58:01: Scheduler Thread - Seconds before next job scan: 7
2015-06-26 22:58:08: Scheduler - Performing Job scan on Primary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:58:08: Scheduler - Slave has been marked bad for job 558df36ab5b7e178802f5633, skipping this job.
2015-06-26 22:58:08: Scheduler - Preliminary check: Slave is blacklisted for 558df605b5b7e169a82ec649 limit.
2015-06-26 22:58:08: Scheduler - Performing Job scan on Secondary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:58:08: Scheduler - Job chooser found no jobs.
2015-06-26 22:58:09: Scheduler Thread - Seconds before next job scan: 5
2015-06-26 22:58:14: Scheduler - Performing Job scan on Primary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:58:14: Scheduler - Slave has been marked bad for job 558df36ab5b7e178802f5633, skipping this job.
2015-06-26 22:58:14: Scheduler - Preliminary check: Slave is blacklisted for 558df605b5b7e169a82ec649 limit.
2015-06-26 22:58:14: Scheduler - Performing Job scan on Secondary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:58:14: Scheduler - Job chooser found no jobs.
2015-06-26 22:58:15: Scheduler Thread - Seconds before next job scan: 7
2015-06-26 22:58:22: Scheduler - Performing Job scan on Primary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:58:22: Scheduler - Slave has been marked bad for job 558df36ab5b7e178802f5633, skipping this job.
2015-06-26 22:58:22: Scheduler - Preliminary check: Slave is blacklisted for 558df605b5b7e169a82ec649 limit.
2015-06-26 22:58:22: Scheduler - Performing Job scan on Secondary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:58:22: Scheduler - Job chooser found no jobs.
2015-06-26 22:58:23: Scheduler Thread - Seconds before next job scan: 5
2015-06-26 22:58:28: Scheduler - Performing Job scan on Primary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:58:28: Scheduler - Slave has been marked bad for job 558df36ab5b7e178802f5633, skipping this job.
2015-06-26 22:58:28: Scheduler - Preliminary check: Slave is blacklisted for 558df605b5b7e169a82ec649 limit.
2015-06-26 22:58:28: Scheduler - Performing Job scan on Secondary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:58:28: Scheduler - Job chooser found no jobs.
2015-06-26 22:58:29: Scheduler Thread - Seconds before next job scan: 5
2015-06-26 22:58:34: Scheduler - Performing Job scan on Primary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:58:34: Scheduler - Slave has been marked bad for job 558df36ab5b7e178802f5633, skipping this job.
2015-06-26 22:58:34: Scheduler - Preliminary check: Slave is blacklisted for 558df605b5b7e169a82ec649 limit.
2015-06-26 22:58:34: Scheduler - Performing Job scan on Secondary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:58:34: Scheduler - Job chooser found no jobs.
2015-06-26 22:58:35: Scheduler Thread - Seconds before next job scan: 7
2015-06-26 22:58:43: Scheduler - Performing Job scan on Primary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:58:43: Scheduler - Slave has been marked bad for job 558df36ab5b7e178802f5633, skipping this job.
2015-06-26 22:58:43: Scheduler - Preliminary check: Slave is blacklisted for 558df605b5b7e169a82ec649 limit.
2015-06-26 22:58:43: Scheduler - Performing Job scan on Secondary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:58:43: Scheduler - Job chooser found no jobs.
2015-06-26 22:58:44: Scheduler Thread - Seconds before next job scan: 6
2015-06-26 22:58:50: Scheduler - Performing Job scan on Primary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:58:50: Scheduler - Slave has been marked bad for job 558df36ab5b7e178802f5633, skipping this job.
2015-06-26 22:58:50: Scheduler - Preliminary check: Slave is blacklisted for 558df605b5b7e169a82ec649 limit.
2015-06-26 22:58:50: Scheduler - Performing Job scan on Secondary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:58:50: Scheduler - Job chooser found no jobs.
2015-06-26 22:58:51: Scheduler Thread - Seconds before next job scan: 7
2015-06-26 22:58:58: Scheduler - Performing Job scan on Primary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:58:58: Scheduler - Slave has been marked bad for job 558df36ab5b7e178802f5633, skipping this job.
2015-06-26 22:58:58: Scheduler - Preliminary check: Slave is blacklisted for 558df605b5b7e169a82ec649 limit.
2015-06-26 22:58:58: Scheduler - Performing Job scan on Secondary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:58:58: Scheduler - Job chooser found no jobs.
2015-06-26 22:58:59: Scheduler Thread - Seconds before next job scan: 5
2015-06-26 22:59:04: Scheduler - Performing Job scan on Primary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:59:04: Scheduler - Slave has been marked bad for job 558df36ab5b7e178802f5633, skipping this job.
2015-06-26 22:59:04: Scheduler - Preliminary check: Slave is blacklisted for 558df605b5b7e169a82ec649 limit.
2015-06-26 22:59:04: Scheduler - Performing Job scan on Secondary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:59:04: Scheduler - Job chooser found no jobs.
2015-06-26 22:59:05: Scheduler Thread - Seconds before next job scan: 7
2015-06-26 22:59:12: Scheduler - Performing Job scan on Primary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:59:12: Scheduler - Slave has been marked bad for job 558df36ab5b7e178802f5633, skipping this job.
2015-06-26 22:59:12: Scheduler - Preliminary check: Slave is blacklisted for 558df605b5b7e169a82ec649 limit.
2015-06-26 22:59:12: Scheduler - Performing Job scan on Secondary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:59:12: Scheduler - Job chooser found no jobs.
2015-06-26 22:59:13: Scheduler Thread - Seconds before next job scan: 6
2015-06-26 22:59:19: Scheduler - Performing Job scan on Primary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:59:19: Scheduler - Slave has been marked bad for job 558df36ab5b7e178802f5633, skipping this job.
2015-06-26 22:59:19: Scheduler - Preliminary check: Slave is blacklisted for 558df605b5b7e169a82ec649 limit.
2015-06-26 22:59:19: Scheduler - Performing Job scan on Secondary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:59:19: Scheduler - Job chooser found no jobs.
2015-06-26 22:59:20: Scheduler Thread - Seconds before next job scan: 5
2015-06-26 22:59:25: Scheduler - Performing Job scan on Primary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:59:25: Scheduler - Slave has been marked bad for job 558df36ab5b7e178802f5633, skipping this job.
2015-06-26 22:59:25: Scheduler - Preliminary check: Slave is blacklisted for 558df605b5b7e169a82ec649 limit.
2015-06-26 22:59:25: Scheduler - Performing Job scan on Secondary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:59:25: Scheduler - Job chooser found no jobs.
2015-06-26 22:59:26: Scheduler Thread - Seconds before next job scan: 5
2015-06-26 22:59:31: Scheduler - Performing Job scan on Primary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:59:31: Scheduler - Slave has been marked bad for job 558df36ab5b7e178802f5633, skipping this job.
2015-06-26 22:59:31: Scheduler - Preliminary check: Slave is blacklisted for 558df605b5b7e169a82ec649 limit.
2015-06-26 22:59:31: Scheduler - Performing Job scan on Secondary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:59:31: Scheduler - Job chooser found no jobs.
2015-06-26 22:59:32: Scheduler Thread - Seconds before next job scan: 5
2015-06-26 22:59:37: Scheduler - Performing Job scan on Primary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:59:37: Scheduler - Slave has been marked bad for job 558df36ab5b7e178802f5633, skipping this job.
2015-06-26 22:59:37: Scheduler - Preliminary check: Slave is blacklisted for 558df605b5b7e169a82ec649 limit.
2015-06-26 22:59:37: Scheduler - Performing Job scan on Secondary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:59:37: Scheduler - Job chooser found no jobs.
2015-06-26 22:59:38: Scheduler Thread - Seconds before next job scan: 6
2015-06-26 22:59:45: Scheduler - Performing Job scan on Primary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:59:45: Scheduler - Slave has been marked bad for job 558df36ab5b7e178802f5633, skipping this job.
2015-06-26 22:59:45: Scheduler - Preliminary check: Slave is blacklisted for 558df605b5b7e169a82ec649 limit.
2015-06-26 22:59:45: Scheduler - Performing Job scan on Secondary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:59:45: Scheduler - Job chooser found no jobs.
2015-06-26 22:59:46: Scheduler Thread - Seconds before next job scan: 5
2015-06-26 22:59:51: Scheduler - Performing Job scan on Primary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:59:51: Scheduler - Slave has been marked bad for job 558df36ab5b7e178802f5633, skipping this job.
2015-06-26 22:59:51: Scheduler - Preliminary check: Slave is blacklisted for 558df605b5b7e169a82ec649 limit.
2015-06-26 22:59:51: Scheduler - Performing Job scan on Secondary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:59:51: Scheduler - Job chooser found no jobs.
2015-06-26 22:59:52: Scheduler Thread - Seconds before next job scan: 5
2015-06-26 22:59:57: Scheduler - Performing Job scan on Primary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:59:57: Scheduler - Slave has been marked bad for job 558df36ab5b7e178802f5633, skipping this job.
2015-06-26 22:59:57: Scheduler - Preliminary check: Slave is blacklisted for 558df605b5b7e169a82ec649 limit.
2015-06-26 22:59:57: Scheduler - Performing Job scan on Secondary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 22:59:57: Scheduler - Job chooser found no jobs.
2015-06-26 22:59:58: Scheduler Thread - Seconds before next job scan: 6
2015-06-26 23:00:04: Scheduler - Performing Job scan on Primary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 23:00:04: Scheduler - Slave has been marked bad for job 558df36ab5b7e178802f5633, skipping this job.
2015-06-26 23:00:04: Scheduler - Preliminary check: Slave is blacklisted for 558df605b5b7e169a82ec649 limit.
2015-06-26 23:00:04: Scheduler - Performing Job scan on Secondary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 23:00:04: Scheduler - Job chooser found no jobs.
2015-06-26 23:00:05: Scheduler Thread - Seconds before next job scan: 7
2015-06-26 23:00:12: Scheduler - Performing Job scan on Primary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 23:00:12: Scheduler - Slave has been marked bad for job 558df36ab5b7e178802f5633, skipping this job.
2015-06-26 23:00:12: Scheduler - Preliminary check: Slave is blacklisted for 558df605b5b7e169a82ec649 limit.
2015-06-26 23:00:12: Scheduler - Performing Job scan on Secondary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 23:00:12: Scheduler - Job chooser found no jobs.
2015-06-26 23:00:13: Scheduler Thread - Seconds before next job scan: 5
2015-06-26 23:00:18: Scheduler - Performing Job scan on Primary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 23:00:18: Scheduler - Slave has been marked bad for job 558df36ab5b7e178802f5633, skipping this job.
2015-06-26 23:00:18: Scheduler - Preliminary check: Slave is blacklisted for 558df605b5b7e169a82ec649 limit.
2015-06-26 23:00:18: Scheduler - Performing Job scan on Secondary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 23:00:18: Scheduler - Job chooser found no jobs.
2015-06-26 23:00:19: Scheduler Thread - Seconds before next job scan: 7
2015-06-26 23:00:26: Scheduler - Performing Job scan on Primary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 23:00:26: Scheduler - Slave has been marked bad for job 558df36ab5b7e178802f5633, skipping this job.
2015-06-26 23:00:26: Scheduler - Preliminary check: Slave is blacklisted for 558df605b5b7e169a82ec649 limit.
2015-06-26 23:00:26: Scheduler - Performing Job scan on Secondary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 23:00:26: Scheduler - Job chooser found no jobs.
2015-06-26 23:00:27: Scheduler Thread - Seconds before next job scan: 7
2015-06-26 23:00:34: Scheduler - Performing Job scan on Primary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 23:00:34: Scheduler - Slave has been marked bad for job 558df36ab5b7e178802f5633, skipping this job.
2015-06-26 23:00:34: Scheduler - Preliminary check: Slave is blacklisted for 558df605b5b7e169a82ec649 limit.
2015-06-26 23:00:34: Scheduler - Performing Job scan on Secondary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 23:00:34: Scheduler - Job chooser found no jobs.
2015-06-26 23:00:35: Scheduler Thread - Seconds before next job scan: 5
2015-06-26 23:00:41: Scheduler - Performing Job scan on Primary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 23:00:41: Scheduler - Slave has been marked bad for job 558df36ab5b7e178802f5633, skipping this job.
2015-06-26 23:00:41: Scheduler - Preliminary check: Slave is blacklisted for 558df605b5b7e169a82ec649 limit.
2015-06-26 23:00:41: Scheduler - Performing Job scan on Secondary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 23:00:41: Scheduler - Job chooser found no jobs.
2015-06-26 23:00:42: Scheduler Thread - Seconds before next job scan: 5
2015-06-26 23:00:47: Scheduler - Performing Job scan on Primary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 23:00:47: Scheduler - Slave has been marked bad for job 558df36ab5b7e178802f5633, skipping this job.
2015-06-26 23:00:47: Scheduler - Preliminary check: Slave is blacklisted for 558df605b5b7e169a82ec649 limit.
2015-06-26 23:00:47: Scheduler - Performing Job scan on Secondary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 23:00:47: Scheduler - Job chooser found no jobs.
2015-06-26 23:00:48: Scheduler Thread - Seconds before next job scan: 6
2015-06-26 23:00:54: Scheduler - Performing Job scan on Primary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 23:00:54: Scheduler - Slave has been marked bad for job 558df36ab5b7e178802f5633, skipping this job.
2015-06-26 23:00:54: Scheduler - Preliminary check: Slave is blacklisted for 558df605b5b7e169a82ec649 limit.
2015-06-26 23:00:54: Scheduler - Performing Job scan on Secondary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 23:00:54: Scheduler - Job chooser found no jobs.
2015-06-26 23:00:55: Scheduler Thread - Seconds before next job scan: 7
2015-06-26 23:01:02: Scheduler - Performing Job scan on Primary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 23:01:02: Scheduler - Slave has been marked bad for job 558df36ab5b7e178802f5633, skipping this job.
2015-06-26 23:01:02: Scheduler - Preliminary check: Slave is blacklisted for 558df605b5b7e169a82ec649 limit.
2015-06-26 23:01:02: Scheduler - Performing Job scan on Secondary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 23:01:02: Scheduler - Job chooser found no jobs.
2015-06-26 23:01:03: Scheduler Thread - Seconds before next job scan: 5
2015-06-26 23:01:08: Scheduler - Performing Job scan on Primary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 23:01:08: Scheduler - Slave has been marked bad for job 558df36ab5b7e178802f5633, skipping this job.
2015-06-26 23:01:08: Scheduler - Preliminary check: Slave is blacklisted for 558df605b5b7e169a82ec649 limit.
2015-06-26 23:01:08: Scheduler - Performing Job scan on Secondary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 23:01:08: Scheduler - Job chooser found no jobs.
2015-06-26 23:01:09: Scheduler Thread - Seconds before next job scan: 5
2015-06-26 23:01:14: Scheduler - Performing Job scan on Primary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 23:01:14: Scheduler - Slave has been marked bad for job 558df36ab5b7e178802f5633, skipping this job.
2015-06-26 23:01:14: Scheduler - Preliminary check: Slave is blacklisted for 558df605b5b7e169a82ec649 limit.
2015-06-26 23:01:14: Scheduler - Performing Job scan on Secondary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 23:01:14: Scheduler - Job chooser found no jobs.
2015-06-26 23:01:15: Scheduler Thread - Seconds before next job scan: 6
2015-06-26 23:01:21: Scheduler - Performing Job scan on Primary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 23:01:21: Scheduler - Slave has been marked bad for job 558df36ab5b7e178802f5633, skipping this job.
2015-06-26 23:01:21: Scheduler - Preliminary check: Slave is blacklisted for 558df605b5b7e169a82ec649 limit.
2015-06-26 23:01:21: Scheduler - Performing Job scan on Secondary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 23:01:21: Scheduler - Job chooser found no jobs.
2015-06-26 23:01:22: Scheduler Thread - Seconds before next job scan: 5
2015-06-26 23:01:27: Scheduler - Performing Job scan on Primary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 23:01:27: Scheduler - Slave has been marked bad for job 558df36ab5b7e178802f5633, skipping this job.
2015-06-26 23:01:27: Scheduler - Preliminary check: Slave is blacklisted for 558df605b5b7e169a82ec649 limit.
2015-06-26 23:01:27: Scheduler - Performing Job scan on Secondary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 23:01:27: Scheduler - Job chooser found no jobs.
2015-06-26 23:01:28: Scheduler Thread - Seconds before next job scan: 5
2015-06-26 23:01:33: Scheduler - Performing Job scan on Primary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 23:01:33: Scheduler - Slave has been marked bad for job 558df36ab5b7e178802f5633, skipping this job.
2015-06-26 23:01:33: Scheduler - Preliminary check: Slave is blacklisted for 558df605b5b7e169a82ec649 limit.
2015-06-26 23:01:33: Scheduler - Performing Job scan on Secondary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 23:01:33: Scheduler - Job chooser found no jobs.
2015-06-26 23:01:34: Scheduler Thread - Seconds before next job scan: 6
2015-06-26 23:01:41: Scheduler - Performing Job scan on Primary Pools with scheduling order Pool, Priority, First-in First-out
2015-06-26 23:01:41: Scheduler - Slave has been marked bad for job 558df36ab5b7e178802f5633, skipping this job.
2015-06-26 23:01:41: Scheduler - Preliminary check: Slave is blacklisted for 558df605b5b7e169a82ec649 limit.
2015-06-26 23:01:41: Scheduler - Successfully dequeued 1 task(s). Returning.
2015-06-26 23:01:41: Scheduler - Returning limit stubs not in use.
2015-06-26 23:01:41: Loading event plugin ConfigSlave (\#####\custom\events\ConfigSlave)
2015-06-26 23:01:41: Scheduler Thread - Synchronizing job auxiliary files from \#####\jobs\558df36ab5b7e178802f5633
2015-06-26 23:01:41: Scheduler Thread - All job files are already synchronized
2015-06-26 23:01:41: Scheduler Thread - Synchronizing plugin files from \#####\plugins\3dsmax
2015-06-26 23:01:41: Scheduler Thread - Synchronization time for plugin files: 686.391 ms
2015-06-26 23:01:41: Loading event plugin ConfigSlave (\#####\custom\events\ConfigSlave)
2015-07-01 09:03:49: Connecting to Slave log: Render-i7-13
2015-07-01 09:03:55: Listener Thread - fe80::a867:5afa:fe07:7e8e%11 has connected
2015-07-01 09:03:55: Listener Thread - Received message: StreamLog

As you can see it’s just been spinning its wheels for 5 days. The slave looks to be running properly. The system isn’t frozen.

It’s still according to itself rendering an old job ‘####_0010_MASTER_B09.01’

But the monitor aptly calls it “Stalled” instead of rendering. However the slave still accepts commands and I can have it restart the slave but it still lists it as offline or stalled depending on if I set it to “offline”. The ony thing that seems to get it to render again after this state is a reboot.

[eamsler]

There’s a thread responsible for updating the Slave’s status. I don’t remember what we call it offhand, but “slave info thread” seems right at the moment.

When you restart the Slave, does it at least update its status once? If not, I’ve got to assume there’s something in that code path that’s causing the thread to exit. I’ve actually heard of this problem before, but it’s been a long time (never did track it down). Is there anything in the Slave log when you stop and start the Slave process?

[eamsler]

Hey Gavin, would you be able to kick that whole log over to me in a rar or 7z? It’s edwin.amsler@thinkboxsoftware.com

I should be able to figure out where the heck the slave info thread died (if it did).

[maxplugins]

Edwin, did you ever find out what was causing this?

Because I’ve got exactly the same thing happening on one of our slaves.
I just realised that it hasn’t actually rendered anything for days.
It doesn’t stall, it just accepts the task, sits there thinking about it for 5 minutes, and then another slave will render the task.
It also doesn’t throw an error.

However, the log file looks the same:

2015-12-04 09:49:16: 0: Plugin rendering frame(s): 970-974 2015-12-04 09:49:16: Scheduler Thread - Task "194_970-974" could not be found because task has been modified: 2015-12-04 09:49:16: current status = Rendering, new status = Rendering 2015-12-04 09:49:16: current slave = schimpanse20, new slave = schimpanse51 2015-12-04 09:49:16: current frames = 970-974, new frames = 970-974 2015-12-04 09:49:16: Scheduler Thread - Cancelling task...

This is happening with After Effects jobs right now, but I guess it’s also been doing the same with Max, Nuke and C4D (no time to check right now).

Dave

[MikeOwen]

Could you confirm what exact version of Deadline your running? I thought this was fixed ages ago.

[maxplugins]

v7.2.0.18

From 80 slaves, this is the only one having the problem.

Dave

Update: now the other slaves are starting to do the same.
This is making all of the render jobs take a lot longer, and there aren’t even any errors being generated so unless you sit and watch the monitor for long periods of time, you wouldn’t know it was happening.

[maxplugins]

Hi Mike,

any news on this?
The render farm has spent about half of the weekend just accepting tasks, thinking about it for 5 to 10 minutes, and then passing the task on to another slave…

Cheers,

Dave

[MikeOwen]

Hi Dave,

I was thinking you might be running an older version than 7.2.0.18 as that would explain it as the orginal thread was back in July against 7.1 IIRC.

Could you drop a note to support@thinkboxsoftware.com, referencing this thread and I’ll get the support team to schedule a remote debug session with you this afternoon.

Just to confirm, in the slave & pulse panel, the “Version” column field is displaying exactly the same version of Deadline for all slaves & pulses? Can you run a House Cleaning and Repository Repair from your Monitor -> super-user -> tools and if it’s not already enabled, can you enable verbose logging for Pulse and Slave here in your repo config:
docs.thinkboxsoftware.com/produc … ation-data
If you are running Pulse, no harm in giving the application a restart.
Finally, has anything changed recently in your pipeline, which might explain this behaviour? Have IT changed anything/ deployed any different settings across your machines?

[maxplugins]

Thanks for the help Mike.

All machines are up to date, and pretty much identical.
The only thing that has happened recently is that we ramped up from 50 to 80 slaves.

I’ve turned on the verbose logging, but I’m going to have to wait before I submit a support ticket as we’re currently completely stressed out and I just don’t have any time to spare.

In any case it’s back to being one machine that does this all the time, and the other machines have a hiccup every now and then and miss a frame or two, but then they are back to normal.

Cheers,

Dave

[MikeOwen]

ok, understood Dave. I’m sorry for your troubles.
Perhaps, “disable” this one slave that is causing you issues, leave it a few days to confirm it’s definitely just this one machine? Perhaps, if you used Auto-Upgrade, this went screwy on this one machine. Either way, it won’t do any harm, just to manually un-install the Deadline client software on this one slave and re-install it a fresh, to see if that helps? The verbose slave logs on this one machine should hopefully stand out as being different to all the others for some reason.

[maxplugins]

I just tried the uninstall and re-install, and it made no difference.

I’m going to get IT to copy the original image back onto the slave, and then we’ll see if that makes a difference.

Dave

[eamsler]

So, after digging into some previous tickets, my guess is that there may be two Slaves running on the same machine. Because the Slaves have the same name they’ll show up in the database as the same Slave. Because of this they get confused because they see themselves working on something when they’re actually idle. They then requeue that task.

Can you take a look at the running processes on that machine and see if there are duplicates? This would be really rare on Windows, but worth a look.

[maxplugins]

There was definitely only one slave running at any time, I spent a while yesterday just watching the task manager to see if anything strange was happening.

Dave

[eamsler]

Okay, so next test. This should be a problem somewhere in Deadline’s house cleaning. Are you running Pulse? If not, can you start up Pulse on some machine so that it will handle house cleaning on behalf of the Slaves and we’ll watch it? We’ll

docs.thinkboxsoftware.com/produc … ning-pulse

Just to be safe, enable verbose logging so we catch everything:

docs.thinkboxsoftware.com/produc … ation-data

We’ll follow along and see what ends up happening. I just remembered a client who had this problem ended up not having any issues once house cleaning was handled by Pulse, so if that’s the case for you, I’d like to revert back and dig a bit deeper so we can solve this one.

[maxplugins]

Hi Edwin,

we are running pulse, and a couple of days ago I turned on the verbose logging.
At the moment the problem slave is disabled, but as soon as the worst is out of the way I’ll enable it again so we can see what Pulse has to say about the problem.

Cheers,

Dave

[maxplugins]

Okay, a bit more information:

The slaves are accepting a task, and then basically spend much too long in the “Starting job” phase without actually starting the job.
While they are just sitting there doing nothing, Pulse does a check for orphaned tasks and reassigns the task to another slave:

Orphaned Task Scan - Requeuing orphaned task '58' for schimpanse61: the task is in the rendering state, but the slave is no longer rendering this task

After this happens, the original slave puts this in the log file:

2015-12-04 09:49:16: Scheduler Thread - Task "194_970-974" could not be found because task has been modified: 2015-12-04 09:49:16: current status = Rendering, new status = Rendering 2015-12-04 09:49:16: current slave = schimpanse20, new slave = schimpanse51 2015-12-04 09:49:16: current frames = 970-974, new frames = 970-974 2015-12-04 09:49:16: Scheduler Thread - Cancelling task...

So the real problem has more to do with the fact that the slaves accept a task and then seeming to do nothing.

This keeps on happening irregularly, and after a couple of times the slaves seem to get themselves sorted out and start rendering properly again (all except the one machine that doesn’t render at all any more).

Yesterday evening this happened simultaneously on about 30 machines, and I nearly had a heart attack…
It’s just happened again with about 10 machines, but most of them are back to rendering again.

Dave

[eamsler]

Sounds good! You can bundle them all together (with 7zip in WinRAR) once you have some time and send them into the support system. I’ll take a look and see what I can find out.