dequeue - should return tokens earlier?

LaszloSebo · July 24, 2014, 5:25pm

We have several hundred nuke licenses, but because of the extremely slow dequeuing mechanism, only have around 20-30 machines actually rendering with nuke. What happens is that a machine picks up a nuke job, renders a frame, then tries to dequeue the next task/job. While its doing that, it does not yet return the tokens it currently owns. So its holding on to the nuke tokens for example (amongst others).
However, the dequeing attempt can take up to 5-10 minutes. So we have several hundred machines optimistically looking for nuke jobs holding on to nuke tokens, for extended periods of time…

Here is a log of a 12 minute dequeue wait… : 09:50:44 - 10:02:58

2014-07-24 09:50:44:  0: Render time for frame(s): 17.531 s
2014-07-24 09:50:44:  0: Total time for task: 21.641 s
2014-07-24 09:50:45:  Scheduler Thread - Render Thread 0 completed its task
2014-07-24 09:50:45:  Scheduler Thread - Seconds before next job scan: 2
2014-07-24 09:50:48:  Scheduler - Performing Job scan on Primary Pools with scheduling order Pool, Weighted, Balanced
2014-07-24 09:50:48:  Scheduler - The 53d10b114577089b08024a86 limit is maxed out.
2014-07-24 09:50:49:  Scheduler - The 53d0fa171afcc14360cdf2c0 limit is maxed out.
2014-07-24 09:50:49:  Scheduler - The 53d1094ab81d07169837d83a limit is maxed out.
2014-07-24 09:50:49:  Scheduler - Slave has been marked bad for job 53d10e75ee0ff51d0c705325, skipping this job.
2014-07-24 09:50:50:  Scheduler - The 53d1368f13ead4333b6d0870 limit is maxed out.
2014-07-24 09:50:50:  Scheduler - The 53d13808457708fa68b858df limit is maxed out.
2014-07-24 09:50:51:  Scheduler - The 53d138aee1fbb10964031ee6 limit is maxed out.
2014-07-24 09:50:51:  Scheduler - The 53d13581a8fa775b2014f83b limit is maxed out.
2014-07-24 09:50:51:  Scheduler - The 53d13924e1fbb1084c96df55 limit is maxed out.
2014-07-24 09:50:52:  Scheduler - The 53d13935e1fbb10ab4f892b9 limit is maxed out.
2014-07-24 09:50:53:  Scheduler - The 53d12cffeea0db29e4d00be2 limit is maxed out.
2014-07-24 09:50:53:  Scheduler - The 53d130c0654bf669147fd253 limit is maxed out.
2014-07-24 09:50:53:  Scheduler - The 53d12cb47c43281d602ba00f limit is maxed out.
2014-07-24 09:50:54:  Scheduler - The 53d12e9d021bc77c50a1b870 limit is maxed out.
2014-07-24 09:50:54:  Scheduler - The 53d1340f54c2555f5073de9f limit is maxed out.
2014-07-24 09:50:55:  Scheduler - The 53d1395be1fbb1018844df9b limit is maxed out.
2014-07-24 09:50:58:  Scheduler - The 53d13851e1fbb10754dd5390 limit is maxed out.
2014-07-24 09:50:58:  Scheduler - The 53d11b8b6df1883b6893451b limit is maxed out.
2014-07-24 09:50:58:  Scheduler - Slave has been marked bad for job 53d0c6eabbb1ed90683c6da7, skipping this job.
2014-07-24 09:50:58:  Scheduler - The 53d109df1c5e270c941dab57 limit is maxed out.
2014-07-24 09:50:59:  Scheduler - The 53d11d5177660d4d447d7dcc limit is maxed out.
2014-07-24 09:51:00:  Scheduler - The 53d02c6ccf7159fae82f49ff limit is maxed out.
2014-07-24 09:51:00:  Scheduler - The 53cfee9034e08f11241aaf9f limit is maxed out.
2014-07-24 09:51:00:  Scheduler - The 53cff3176882d038f004db35 limit is maxed out.
2014-07-24 09:51:01:  Scheduler - The 53cffa2abb649a0d80bae4a6 limit is maxed out.
2014-07-24 09:51:01:  Scheduler - The 53cffa66bb649a1a4462c325 limit is maxed out.
2014-07-24 09:51:02:  Scheduler - The 53cffa6fbb649a1d2c276ace limit is maxed out.
2014-07-24 09:51:02:  Scheduler - The 53d063a18b53a01e8c5616ac limit is maxed out.
2014-07-24 09:51:02:  Scheduler - The 53d07b998b53a023447aa83e limit is maxed out.
2014-07-24 09:51:03:  Scheduler - The 53d07ba08b53a03300e14edb limit is maxed out.
2014-07-24 09:51:03:  Scheduler - The 53d07ba88b53a03024d79678 limit is maxed out.
2014-07-24 09:51:03:  Scheduler - The 53d07bb18b53a02e7478864a limit is maxed out.
2014-07-24 09:51:04:  Scheduler - The 53d07bda8b53a02554cd7a96 limit is maxed out.
2014-07-24 09:51:04:  Scheduler - The 53cff31f6882d0dda4400754 limit is maxed out.
2014-07-24 09:51:04:  Scheduler - The 53cffab3bb649a162cf72f07 limit is maxed out.
2014-07-24 09:51:05:  Scheduler - The 53cffaf4bb649a20d81c94d6 limit is maxed out.
2014-07-24 09:51:05:  Scheduler - The 53d05a2901f73e789455fa4e limit is maxed out.
2014-07-24 09:51:05:  Scheduler - The 53d0638b8b53a00af8d0f715 limit is maxed out.
2014-07-24 09:51:06:  Scheduler - The 53d0638e8b53a02d0c0d7c22 limit is maxed out.
2014-07-24 09:51:07:  Scheduler - The 53d063908b53a01cbc24a69f limit is maxed out.
2014-07-24 09:51:07:  Scheduler - The 53d063928b53a01d346fa939 limit is maxed out.
2014-07-24 09:51:07:  Scheduler - The 53d064288b53a021b8f012c6 limit is maxed out.
2014-07-24 09:51:08:  Scheduler - The 53d064448b53a023300dc492 limit is maxed out.
2014-07-24 09:51:08:  Scheduler - The 53d07bcb8b53a0288c0698ee limit is maxed out.
2014-07-24 09:51:08:  Scheduler - The 53d07be18b53a023841f83b8 limit is maxed out.
2014-07-24 09:51:08:  Scheduler - The 53d063a78b53a025a009b59e limit is maxed out.
2014-07-24 09:51:09:  Scheduler - The 53d064268b53a02f98dd1804 limit is maxed out.
2014-07-24 09:51:09:  Scheduler - The 53d0642a8b53a01f2c1e1a46 limit is maxed out.
2014-07-24 09:51:10:  Scheduler - The 53d0642c8b53a030a0691ce0 limit is maxed out.
2014-07-24 09:51:10:  Scheduler - The 53d063a38b53a017382668c4 limit is maxed out.
2014-07-24 09:51:11:  Scheduler - The 53d063a58b53a031a0e40d94 limit is maxed out.
2014-07-24 09:51:11:  Scheduler - The 53d0643d8b53a01a04240c25 limit is maxed out.
2014-07-24 09:51:13:  Scheduler - The 53d064418b53a025c4032945 limit is maxed out.
2014-07-24 09:51:14:  Scheduler - The 53d07bd38b53a0222cabb87b limit is maxed out.
2014-07-24 09:51:14:  Scheduler - The 53d0643f8b53a015441de887 limit is maxed out.
2014-07-24 09:51:15:  Scheduler - The 53cfee7a34e08f1778c34dbd limit is maxed out.
2014-07-24 09:51:16:  Scheduler - The 53d138e5e4b5eb0200d33e80 limit is maxed out.
2014-07-24 09:51:16:  Scheduler - The 53d138f5e4b5eb0bf489b08f limit is maxed out.
2014-07-24 09:51:16:  Scheduler - The 53cffad6bb649a20449d0197 limit is maxed out.
2014-07-24 09:51:16:  Scheduler - The 53cff3066882d0a0e8ae1099 limit is maxed out.
2014-07-24 09:51:17:  Scheduler - The 53cff3096882d0184c484936 limit is maxed out.
2014-07-24 09:51:17:  Scheduler - The 53cff30b6882d034505f71dc limit is maxed out.
2014-07-24 09:51:18:  Scheduler - The 53cff30e6882d08144de9dbc limit is maxed out.
2014-07-24 09:51:19:  Scheduler - The 53cff31d6882d00ae8181fc3 limit is maxed out.
2014-07-24 09:51:20:  Scheduler - The 53d05a1801f73e1a3428b9fc limit is maxed out.
2014-07-24 09:51:20:  Scheduler - The 53d05a1a01f73e2d841418d1 limit is maxed out.
2014-07-24 09:51:20:  Scheduler - The 53d05fd6d3e2b41060f664bd limit is maxed out.
2014-07-24 09:51:20:  Scheduler - The 53cff31a6882d0854c28930b limit is maxed out.
2014-07-24 09:51:21:  Scheduler - The 53cee9d2d3e2b4491ce35b03 limit is maxed out.
2014-07-24 09:51:21:  Scheduler - The 53d06a19cf715990e873dc01 limit is maxed out.
2014-07-24 09:51:22:  Scheduler - The 53cee9cfd3e2b4c03836751f limit is maxed out.
2014-07-24 09:51:22:  Scheduler - The 53cee9cad3e2b4604c232eb9 limit is maxed out.
2014-07-24 09:51:22:  Scheduler - The 53cee9ccd3e2b4a5d4001001 limit is maxed out.
2014-07-24 09:51:23:  Scheduler - The 53d029a744ec7a1e94b412cf limit is maxed out.
2014-07-24 09:51:23:  Scheduler - The 53d0299a44ec7a170ce8ff94 limit is maxed out.
2014-07-24 09:51:24:  Scheduler - The 53d02c97cf71590da04422a1 limit is maxed out.
2014-07-24 09:51:28:  Scheduler - Slave is not whitelisted for mx_128gb limit.
2014-07-24 09:51:29:  Scheduler - Slave is not whitelisted for mx_128gb limit.
2014-07-24 09:51:30:  Scheduler - Slave is not whitelisted for mx_fastmachines limit.
2014-07-24 09:51:32:  Scheduler - Slave is not whitelisted for mx_fastmachines limit.
2014-07-24 09:51:38:  Scheduler - Slave is not whitelisted for mx_fastmachines limit.
2014-07-24 09:51:41:  Scheduler - Slave is not whitelisted for mx_fastmachines limit.
2014-07-24 09:51:43:  Scheduler - The 53d13629dbebb721188a72ee limit is maxed out.
2014-07-24 09:51:44:  Scheduler - The 53d1356dd7c81214d0605ba5 limit is maxed out.
2014-07-24 09:51:44:  Scheduler - Slave is not whitelisted for mx_fastmachines limit.
2014-07-24 09:51:45:  Scheduler - Performing Job scan on Secondary Pools with scheduling order Pool, Weighted, Balanced
2014-07-24 09:51:45:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:51:47:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:51:48:  Scheduler - The 53d13924e1fbb1084c96df55 limit is maxed out.
2014-07-24 09:51:49:  Scheduler - The 53d13995e1fbb108b0d165bb limit is maxed out.
2014-07-24 09:51:50:  Scheduler - Slave is not whitelisted for nfs limit.
2014-07-24 09:51:52:  Scheduler - The 53d12cffeea0db29e4d00be2 limit is maxed out.
2014-07-24 09:51:55:  Scheduler - The 53d130c0654bf669147fd253 limit is maxed out.
2014-07-24 09:51:56:  Scheduler - The 53d13808457708fa68b858df limit is maxed out.
2014-07-24 09:51:56:  Scheduler - The 53d138aee1fbb10964031ee6 limit is maxed out.
2014-07-24 09:51:56:  Scheduler - The 53d12cb47c43281d602ba00f limit is maxed out.
2014-07-24 09:51:57:  Scheduler - The 53d12e9d021bc77c50a1b870 limit is maxed out.
2014-07-24 09:51:57:  Scheduler - The 53d1340f54c2555f5073de9f limit is maxed out.
2014-07-24 09:51:57:  Scheduler - The 53d13581a8fa775b2014f83b limit is maxed out.
2014-07-24 09:51:58:  Scheduler - Slave is not whitelisted for nfs limit.
2014-07-24 09:51:58:  Scheduler - The 53d13851e1fbb10754dd5390 limit is maxed out.
2014-07-24 09:52:00:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:52:02:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:52:04:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:52:05:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:52:06:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:52:07:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:52:09:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:52:10:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:52:11:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:52:13:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:52:14:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:52:15:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:52:17:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:52:19:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:52:20:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:52:20:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:52:22:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:52:23:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:52:25:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:52:26:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:52:28:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:52:29:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:52:31:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:52:33:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:52:34:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:52:37:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:52:39:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:52:41:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:52:43:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:52:44:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:52:45:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:52:46:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:52:47:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:52:49:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:52:51:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:52:53:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:52:54:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:52:55:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:52:56:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:52:57:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:52:58:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:52:59:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:53:00:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:53:01:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:53:03:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:53:04:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:53:05:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:53:06:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:53:07:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:53:08:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:53:10:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:53:12:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:53:13:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:53:15:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:53:16:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:53:17:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:53:19:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:53:20:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:53:22:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:53:23:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:53:24:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:53:25:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:53:27:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:53:28:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:53:29:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:53:30:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:53:32:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:53:33:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:53:36:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:53:38:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:53:39:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:53:43:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:53:44:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:53:45:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:53:47:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:53:48:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:53:50:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:53:51:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:53:52:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:53:54:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:53:55:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:53:56:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:53:58:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:53:59:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:54:01:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:54:02:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:54:03:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:54:04:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:54:05:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:54:06:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:54:07:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:54:09:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:54:10:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:54:12:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:54:14:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:54:14:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:54:16:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:54:17:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:54:18:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:54:18:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:54:20:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:54:21:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:54:22:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:54:23:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:54:24:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:54:26:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:54:27:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:54:29:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:54:30:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:54:32:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:54:34:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:54:35:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:54:36:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:54:38:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:54:39:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:54:40:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:54:41:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:54:42:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:54:43:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:54:43:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:54:44:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:54:45:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:54:46:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:54:48:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:54:50:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:54:50:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:54:52:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:54:54:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:54:55:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:54:59:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:55:00:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:55:01:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:55:02:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:55:04:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:55:04:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:55:06:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:55:08:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:55:09:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:55:10:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:55:11:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:55:12:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:55:13:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:55:16:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:55:17:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:55:19:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:55:21:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:55:23:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:55:24:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:55:25:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:55:26:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:55:27:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:55:28:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:55:30:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:55:31:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:55:34:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:55:35:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:55:36:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:55:37:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:55:38:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:55:40:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:55:41:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:55:42:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:55:44:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:55:44:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:55:45:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:55:47:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:55:48:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:55:49:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:55:51:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:55:52:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:55:53:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:55:54:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:55:55:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:55:57:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:55:58:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:56:00:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:56:01:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:56:03:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:56:04:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:56:05:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:56:07:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:56:10:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:56:13:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:56:14:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:56:15:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:56:16:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:56:17:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:56:18:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:56:19:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:56:21:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:56:22:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:56:24:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:56:25:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:56:27:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:56:28:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:56:29:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:56:30:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:56:33:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:56:35:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:56:36:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:56:38:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:56:39:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:56:41:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:56:43:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:56:44:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:56:46:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:56:47:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:56:47:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:56:49:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:56:50:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:56:52:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:56:52:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:56:57:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:56:58:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:57:00:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:57:01:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:57:01:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:57:03:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:57:05:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:57:06:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:57:08:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:57:09:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:57:11:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:57:12:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:57:13:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:57:14:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:57:15:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:57:17:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:57:18:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:57:20:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:57:21:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:57:23:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:57:24:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:57:25:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:57:27:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:57:29:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:57:31:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:57:32:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:57:35:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:57:35:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:57:36:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:57:37:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:57:39:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:57:42:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:57:47:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:57:48:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:57:50:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:57:51:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:57:52:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:57:54:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:57:56:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:57:58:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:57:59:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:58:00:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:58:02:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:58:03:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:58:05:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:58:08:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:58:09:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:58:11:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:58:12:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:58:13:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:58:14:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:58:16:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:58:17:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:58:18:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:58:19:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:58:20:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:58:23:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:58:24:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:58:26:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:58:28:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:58:30:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:58:32:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:58:34:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:58:34:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:58:37:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:58:38:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:58:39:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:58:40:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:58:42:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:58:44:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:58:45:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:58:47:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:58:48:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:58:49:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:58:50:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:58:51:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:58:52:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:58:53:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:58:57:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:58:58:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:58:59:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:59:00:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:59:02:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:59:04:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:59:05:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:59:06:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:59:07:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:59:08:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:59:10:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:59:12:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:59:13:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:59:15:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:59:17:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:59:19:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:59:20:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:59:22:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:59:24:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:59:26:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:59:28:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:59:30:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:59:31:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:59:32:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:59:34:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:59:35:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:59:37:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:59:38:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:59:39:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:59:41:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:59:44:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:59:45:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:59:46:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:59:47:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:59:49:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:59:51:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:59:52:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:59:54:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:59:56:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:59:57:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:59:58:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 09:59:59:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:00:00:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:00:02:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:00:06:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:00:09:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:00:11:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:00:12:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:00:13:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:00:14:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:00:16:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:00:17:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:00:19:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:00:20:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:00:21:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:00:23:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:00:24:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:00:25:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:00:27:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:00:29:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:00:30:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:00:31:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:00:32:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:00:33:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:00:34:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:00:36:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:00:37:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:00:38:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:00:40:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:00:41:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:00:42:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:00:45:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:00:46:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:00:47:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:00:48:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:00:49:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:00:50:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:00:52:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:00:53:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:00:55:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:00:56:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:00:57:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:00:58:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:00:59:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:01:00:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:01:01:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:01:03:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:01:04:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:01:05:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:01:06:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:01:07:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:01:09:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:01:10:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:01:11:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:01:12:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:01:13:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:01:16:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:01:17:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:01:18:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:01:19:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:01:20:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:01:22:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:01:23:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:01:25:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:01:27:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:01:29:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:01:29:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:01:30:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:01:31:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:01:34:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:01:34:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:01:35:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:01:37:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:01:39:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:01:40:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:01:41:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:01:41:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:01:42:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:01:43:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:01:44:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:01:45:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:01:48:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:01:48:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:01:49:  Connecting to slave log: LAPRO0095
2014-07-24 10:01:49:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:01:50:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:01:51:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:01:52:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:01:53:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:01:54:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:01:55:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:01:57:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:02:00:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:02:02:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:02:03:  Listener Thread - fe80::cb4:203d:53ed:fe3c%15 has connected
2014-07-24 10:02:03:  Listener Thread - Received message: StreamLog
2014-07-24 10:02:03:  Listener Thread - Responded with: Success
2014-07-24 10:02:05:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:02:07:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:02:08:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:02:09:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:02:10:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:02:11:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:02:12:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:02:14:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:02:14:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:02:16:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:02:17:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:02:18:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:02:19:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:02:20:  Scheduler - Slave is not whitelisted for nfs limit.
2014-07-24 10:02:21:  Scheduler - The 53d13935e1fbb10ab4f892b9 limit is maxed out.
2014-07-24 10:02:22:  Scheduler - The 53d11d5177660d4d447d7dcc limit is maxed out.
2014-07-24 10:02:22:  Scheduler - The 53d11b8b6df1883b6893451b limit is maxed out.
2014-07-24 10:02:24:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:02:26:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:02:28:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:02:29:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:02:30:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:02:31:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:02:32:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:02:34:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:02:35:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:02:35:  Scheduler - The 53d1368f13ead4333b6d0870 limit is maxed out.
2014-07-24 10:02:36:  Scheduler - Slave is not whitelisted for nfs limit.
2014-07-24 10:02:37:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:02:38:  Scheduler - Slave has been marked bad for job 53d0c6eabbb1ed90683c6da7, skipping this job.
2014-07-24 10:02:38:  Scheduler - The 53d109df1c5e270c941dab57 limit is maxed out.
2014-07-24 10:02:38:  Scheduler - Slave is not whitelisted for nfs limit.
2014-07-24 10:02:41:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:02:42:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:02:43:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:02:44:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:02:45:  Scheduler - Slave is not whitelisted for nfs limit.
2014-07-24 10:02:46:  Scheduler - Slave is not whitelisted for nfs limit.
2014-07-24 10:02:46:  Scheduler - Slave is not whitelisted for nfs limit.
2014-07-24 10:02:48:  0: Shutdown
2014-07-24 10:02:48:  0: Shutdown
2014-07-24 10:02:49:  0: Exited ThreadMain(), cleaning up...
2014-07-24 10:02:49:  0: Shutdown
2014-07-24 10:02:49:  0: Stopped job: [GOLD] ShootBot: Jpg CS_000_0145_2d_comps_main_v0002
2014-07-24 10:02:49:  0: Unloading plugin: Nuke
2014-07-24 10:02:50:  Scheduler - Returning limit stubs not in use.
2014-07-24 10:02:50:  Scheduler -   returning 53d029c144ec7a1dfc2b4698
2014-07-24 10:02:50:  Scheduler -   returning 53d0781d8b53a018f896ab97
2014-07-24 10:02:50:  Scheduler -   returning 53d063998b53a025d81d283c
2014-07-24 10:02:50:  Scheduler -   returning 53d05a3801f73e646c91bbdc
2014-07-24 10:02:50:  Scheduler -   returning 53cff32a6882d071dcdd130f
2014-07-24 10:02:51:  Scheduler -   returning vray3
2014-07-24 10:02:51:  Scheduler -   returning 53d063ae8b53a02278873b15
2014-07-24 10:02:51:  Scheduler -   returning 53d07bed8b53a01fbcd5aa24
2014-07-24 10:02:52:  Scheduler -   returning 53d07bbf8b53a00d249f5b0a
2014-07-24 10:02:52:  Scheduler -   returning 53d064338b53a008781efb58
2014-07-24 10:02:52:  Scheduler -   returning 53d0644c8b53a02ee46e4b40
2014-07-24 10:02:52:  Scheduler -   returning 53d00ebabb649a191c5ce500
2014-07-24 10:02:53:  Scheduler -   returning 53cffb17bb649a22ec9b171b
2014-07-24 10:02:54:  Scheduler -   returning 53cffa8ebb649a0da887c195
2014-07-24 10:02:54:  Scheduler -   returning vray2
2014-07-24 10:02:55:  Scheduler -   returning 53d0405e162dfe11acf52182
2014-07-24 10:02:55:  Scheduler -   returning 53cffa0bbb649a1e9002ae9a
2014-07-24 10:02:56:  Scheduler -   returning mx_render
2014-07-24 10:02:56:  Scheduler -   returning mx_flowlinerender
2014-07-24 10:02:56:  Scheduler -   returning 53cffa48bb649a1d70857c56
2014-07-24 10:02:56:  Scheduler -   returning 53d138a0e1fbb106c43e2d71
2014-07-24 10:02:57:  Scheduler -   returning win7
2014-07-24 10:02:58:  Scheduler -   returning nuke
2014-07-24 10:02:58:  Scheduler -   returning 53d137f74577084dec9b2bee

rrussell · July 24, 2014, 7:05pm

The long dequeue times could be a result of the current load on your database, and perhaps reducing the frequency at which the slaves update their states will help here.

The reason the slave holds on to the stubs (aka: tokens) is so that if the slave’s next task belongs to the same job, or another that requires a nuke license, it can work on it. Otherwise, it would be possible for the nuke limit to be maxed out between returning it and trying to get it back, which could result in unnecessary jumping around between jobs.

Cheers,
Ryan

LaszloSebo · July 24, 2014, 7:13pm

Yes, it makes sense to hold onto them, if the dequeue test takes <1-5 seconds. But if it takes 10+ minutes, thats a whole different situation.

The problem is that these frequent queries/writes actually greatly contribute to the db overload. Could the dequeueing mechanism be refactored so that instead of multiple mini reads, it does 1 read of multiple jobs & limit stub states?

rrussell · July 24, 2014, 7:18pm

Yeah, it might be possible to do it in batches, but we would have to investigate. I’ll log it as an issue so that we can look into it.

Cheers,
Ryan