AWS Thinkbox Discussion Forums

dequeing - token checks

Deprecated Deadline 6 Bug Reports
#1

Hi there,

It seems that deadline slaves are asking for jobs one at a time, and then check their limits each, again one at a time. This means that some slaves can be stuck looking for jobs for minutes, when the db is overloaded, and ironically, its overloaded BECAUSE these machines are doing the checks in such a weird way.
For example, we have ~300 ingestion jobs queued up, but their renders are throttled via a limit group (so that the network doesn’t get overloaded with large plate transfers).

Look at this log of a poor slave trying to dequeue a job:

2014-07-24 10:02:59:  Scheduler Thread - Seconds before next job scan: 196
2014-07-24 10:06:15:  Scheduler - Performing Job scan on Primary Pools with scheduling order Pool, Weighted, Balanced
2014-07-24 10:06:15:  Scheduler - The 53d10b114577089b08024a86 limit is maxed out.
2014-07-24 10:06:15:  Scheduler - The 53d1368f13ead4333b6d0870 limit is maxed out.
2014-07-24 10:06:16:  Scheduler - The 53d13581a8fa775b2014f83b limit is maxed out.
2014-07-24 10:06:16:  Scheduler - The 53d13935e1fbb10ab4f892b9 limit is maxed out.
2014-07-24 10:06:16:  Scheduler - The 53d139fae1fbb103bcd02e41 limit is maxed out.
2014-07-24 10:06:16:  Scheduler - The 53d13b75e74b5439c405ef47 limit is maxed out.
2014-07-24 10:06:16:  Scheduler - The 53d130c0654bf669147fd253 limit is maxed out.
2014-07-24 10:06:17:  Scheduler - The 53d13851e1fbb10754dd5390 limit is maxed out.
2014-07-24 10:06:17:  Scheduler - The 53d1340f54c2555f5073de9f limit is maxed out.
2014-07-24 10:06:17:  Scheduler - Slave has been marked bad for job 53d0c6eabbb1ed90683c6da7, skipping this job.
2014-07-24 10:06:17:  Scheduler - The 53d109df1c5e270c941dab57 limit is maxed out.
2014-07-24 10:06:17:  Scheduler - The 53d11b8b6df1883b6893451b limit is maxed out.
2014-07-24 10:06:18:  Scheduler - The 53d11d5177660d4d447d7dcc limit is maxed out.
2014-07-24 10:06:18:  Scheduler - The 53d02c6ccf7159fae82f49ff limit is maxed out.
2014-07-24 10:06:18:  Scheduler - The 53cfee9034e08f11241aaf9f limit is maxed out.
2014-07-24 10:06:18:  Scheduler - The 53cff3176882d038f004db35 limit is maxed out.
2014-07-24 10:06:18:  Scheduler - The 53cffa2abb649a0d80bae4a6 limit is maxed out.
2014-07-24 10:06:18:  Scheduler - The 53cffa66bb649a1a4462c325 limit is maxed out.
2014-07-24 10:06:18:  Scheduler - The 53cffa6fbb649a1d2c276ace limit is maxed out.
2014-07-24 10:06:18:  Scheduler - The 53d063a18b53a01e8c5616ac limit is maxed out.
2014-07-24 10:06:19:  Scheduler - The 53d07b998b53a023447aa83e limit is maxed out.
2014-07-24 10:06:19:  Scheduler - The 53d07ba08b53a03300e14edb limit is maxed out.
2014-07-24 10:06:19:  Scheduler - The 53d07ba88b53a03024d79678 limit is maxed out.
2014-07-24 10:06:19:  Scheduler - The 53d07bb18b53a02e7478864a limit is maxed out.
2014-07-24 10:06:19:  Scheduler - The 53d07bda8b53a02554cd7a96 limit is maxed out.
2014-07-24 10:06:19:  Scheduler - The 53d13ba810f8522b4ca8d05b limit is maxed out.
2014-07-24 10:06:19:  Scheduler - The 53cffab3bb649a162cf72f07 limit is maxed out.
2014-07-24 10:06:20:  Scheduler - The 53cffaf4bb649a20d81c94d6 limit is maxed out.
2014-07-24 10:06:20:  Scheduler - The 53d0638b8b53a00af8d0f715 limit is maxed out.
2014-07-24 10:06:20:  Scheduler - The 53d0638e8b53a02d0c0d7c22 limit is maxed out.
2014-07-24 10:06:21:  Scheduler - The 53d063908b53a01cbc24a69f limit is maxed out.
2014-07-24 10:06:21:  Scheduler - The 53d063928b53a01d346fa939 limit is maxed out.
2014-07-24 10:06:21:  Scheduler - The 53d064288b53a021b8f012c6 limit is maxed out.
2014-07-24 10:06:21:  Scheduler - The 53d064448b53a023300dc492 limit is maxed out.
2014-07-24 10:06:21:  Scheduler - The 53d07bcb8b53a0288c0698ee limit is maxed out.
2014-07-24 10:06:22:  Scheduler - The 53d07be18b53a023841f83b8 limit is maxed out.
2014-07-24 10:06:22:  Scheduler - The 53d063a78b53a025a009b59e limit is maxed out.
2014-07-24 10:06:22:  Scheduler - The 53d064268b53a02f98dd1804 limit is maxed out.
2014-07-24 10:06:22:  Scheduler - The 53d0642a8b53a01f2c1e1a46 limit is maxed out.
2014-07-24 10:06:22:  Scheduler - The 53d0642c8b53a030a0691ce0 limit is maxed out.
2014-07-24 10:06:22:  Scheduler - The 53cff31d6882d00ae8181fc3 limit is maxed out.
2014-07-24 10:06:23:  Scheduler - The 53d063a38b53a017382668c4 limit is maxed out.
2014-07-24 10:06:23:  Scheduler - The 53d063a58b53a031a0e40d94 limit is maxed out.
2014-07-24 10:06:23:  Scheduler - The 53d0643d8b53a01a04240c25 limit is maxed out.
2014-07-24 10:06:23:  Scheduler - The 53d064418b53a025c4032945 limit is maxed out.
2014-07-24 10:06:23:  Scheduler - The 53d07bd38b53a0222cabb87b limit is maxed out.
2014-07-24 10:06:23:  Scheduler - The 53d0643f8b53a015441de887 limit is maxed out.
2014-07-24 10:06:24:  Scheduler - The 53cee9cad3e2b4604c232eb9 limit is maxed out.
2014-07-24 10:06:25:  Scheduler - The 53cee9ccd3e2b4a5d4001001 limit is maxed out.
2014-07-24 10:06:25:  Scheduler - The 53cee9cfd3e2b4c03836751f limit is maxed out.
2014-07-24 10:06:25:  Scheduler - The 53cee9d2d3e2b4491ce35b03 limit is maxed out.
2014-07-24 10:06:25:  Scheduler - The 53cfee7a34e08f1778c34dbd limit is maxed out.
2014-07-24 10:06:26:  Scheduler - The 53d138e5e4b5eb0200d33e80 limit is maxed out.
2014-07-24 10:06:26:  Scheduler - The 53d138f5e4b5eb0bf489b08f limit is maxed out.
2014-07-24 10:06:26:  Scheduler - The 53d13c158b53a01cf8c92517 limit is maxed out.
2014-07-24 10:06:26:  Scheduler - The 53cff31a6882d0854c28930b limit is maxed out.
2014-07-24 10:06:26:  Scheduler - The 53cff31f6882d0dda4400754 limit is maxed out.
2014-07-24 10:06:27:  Scheduler - The 53d05a2901f73e789455fa4e limit is maxed out.
2014-07-24 10:06:27:  Scheduler - The 53d13bbc10f852327cd65e28 limit is maxed out.
2014-07-24 10:06:27:  Scheduler - The 53cffad6bb649a20449d0197 limit is maxed out.
2014-07-24 10:06:27:  Scheduler - The 53cff3066882d0a0e8ae1099 limit is maxed out.
2014-07-24 10:06:27:  Scheduler - The 53cff3096882d0184c484936 limit is maxed out.
2014-07-24 10:06:27:  Scheduler - The 53cff30b6882d034505f71dc limit is maxed out.
2014-07-24 10:06:28:  Scheduler - The 53cff30e6882d08144de9dbc limit is maxed out.
2014-07-24 10:06:28:  Scheduler - The 53d05a1801f73e1a3428b9fc limit is maxed out.
2014-07-24 10:06:28:  Scheduler - The 53d05a1a01f73e2d841418d1 limit is maxed out.
2014-07-24 10:06:28:  Scheduler - The 53d05fd6d3e2b41060f664bd limit is maxed out.
2014-07-24 10:06:28:  Scheduler - The 53d06a19cf715990e873dc01 limit is maxed out.
2014-07-24 10:06:28:  Scheduler - The 53d029a744ec7a1e94b412cf limit is maxed out.
2014-07-24 10:06:29:  Scheduler - The 53d0299a44ec7a170ce8ff94 limit is maxed out.
2014-07-24 10:06:29:  Scheduler - The 53d02c97cf71590da04422a1 limit is maxed out.
2014-07-24 10:06:30:  Scheduler - Slave is not whitelisted for mx_128gb limit.
2014-07-24 10:06:30:  Scheduler - Slave is not whitelisted for mx_128gb limit.
2014-07-24 10:06:30:  Scheduler - Slave is not whitelisted for mx_fastmachines limit.
2014-07-24 10:06:31:  Scheduler - Slave is not whitelisted for mx_fastmachines limit.
2014-07-24 10:06:33:  Scheduler - The 53d13629dbebb721188a72ee limit is maxed out.
2014-07-24 10:06:33:  Scheduler - The 53d13a35cf71592960bfa313 limit is maxed out.
2014-07-24 10:06:33:  Scheduler - Slave is not whitelisted for mx_fastmachines limit.
2014-07-24 10:06:34:  Scheduler - The 53d13c9d65c1af1e842da163 limit is maxed out.
2014-07-24 10:06:34:  Scheduler - The 53d13a0dcf71590140e17d09 limit is maxed out.
2014-07-24 10:06:35:  Scheduler - Slave is not whitelisted for mx_fastmachines limit.
2014-07-24 10:06:35:  Scheduler - Performing Job scan on Secondary Pools with scheduling order Pool, Weighted, Balanced
2014-07-24 10:06:36:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:06:36:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:06:36:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:06:37:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:06:37:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:06:38:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:06:38:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:06:39:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:06:39:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:06:40:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:06:40:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:06:41:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:06:42:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:06:42:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:06:43:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:06:43:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:06:44:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:06:44:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:06:45:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:06:45:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:06:46:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:06:46:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:06:47:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:06:47:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:06:48:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:06:49:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:06:50:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:06:50:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:06:51:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:06:51:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:06:51:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:06:52:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:06:52:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:06:53:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:06:53:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:06:54:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:06:55:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:06:55:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:06:56:  Scheduler - The 53d13581a8fa775b2014f83b limit is maxed out.
2014-07-24 10:06:56:  Scheduler - Slave is not whitelisted for nfs limit.
2014-07-24 10:06:57:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:06:58:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:06:58:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:06:59:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:00:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:01:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:01:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:02:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:03:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:04:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:04:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:05:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:07:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:08:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:08:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:09:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:10:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:11:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:12:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:12:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:13:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:14:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:15:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:15:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:16:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:17:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:18:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:18:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:18:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:19:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:20:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:20:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:21:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:21:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:22:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:23:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:23:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:25:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:26:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:27:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:27:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:28:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:29:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:29:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:30:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:30:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:31:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:31:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:32:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:33:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:34:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:34:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:35:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:36:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:36:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:37:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:38:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:39:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:40:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:41:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:42:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:43:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:43:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:44:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:44:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:45:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:45:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:46:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:47:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:47:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:48:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:49:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:50:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:50:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:51:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:52:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:53:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:53:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:54:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:55:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:55:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:56:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:58:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:58:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:07:59:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:00:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:00:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:01:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:02:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:02:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:03:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:04:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:04:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:05:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:06:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:06:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:07:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:07:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:08:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:08:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:09:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:10:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:10:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:11:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:11:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:12:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:13:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:14:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:15:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:15:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:16:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:17:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:17:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:19:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:20:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:21:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:21:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:22:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:22:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:23:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:24:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:25:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:25:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:26:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:26:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:27:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:28:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:29:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:29:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:30:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:31:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:32:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:32:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:33:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:34:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:35:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:35:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:37:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:37:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:38:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:39:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:40:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:40:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:41:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:42:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:42:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:44:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:45:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:46:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:47:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:48:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:49:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:50:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:50:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:51:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:52:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:53:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:54:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:55:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:55:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:56:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:57:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:58:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:58:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:08:59:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:09:00:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:09:01:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:09:02:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:09:04:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:09:07:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:09:07:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:09:08:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:09:09:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:09:11:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:09:12:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:09:12:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:09:13:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:09:14:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:09:15:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:09:15:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:09:17:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:09:17:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:09:18:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:09:18:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:09:19:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:09:20:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:09:21:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:09:21:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:09:23:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:09:24:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:09:24:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:09:26:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:09:27:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:09:28:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:09:29:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:09:30:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:09:31:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:09:32:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:09:33:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:09:34:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:09:35:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:09:37:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:09:38:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:09:39:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:09:40:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:09:40:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:09:41:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:09:42:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:09:42:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:09:44:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:09:45:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:09:45:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:09:46:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:09:47:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:09:48:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:09:49:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:09:50:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:09:51:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:09:54:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:09:55:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:09:55:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:09:56:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:09:57:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:10:04:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:10:05:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:10:06:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:10:07:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:10:07:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:10:08:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:10:09:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:10:09:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:10:11:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:10:12:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:10:13:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:10:15:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:10:16:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:10:17:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:10:18:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:10:19:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:10:20:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:10:21:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:10:22:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:10:23:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:10:24:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:10:25:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:10:26:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:10:26:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:10:27:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:10:28:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:10:28:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:10:30:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:10:31:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:10:32:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:10:32:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:10:33:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:10:34:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:10:35:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:10:35:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:10:36:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:10:37:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:10:39:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:10:40:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:10:41:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:10:42:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:10:43:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:10:44:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:10:45:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:10:46:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:10:49:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:10:50:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:10:52:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:10:53:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:10:54:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:10:55:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:10:57:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:10:58:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:11:00:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:11:01:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:11:02:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:11:03:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:11:04:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:11:05:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:11:05:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:11:07:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:11:08:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:11:09:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:11:10:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:11:11:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:11:12:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:11:13:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:11:13:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:11:14:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:11:15:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:11:17:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:11:18:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:11:19:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:11:20:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:11:22:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:11:23:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:11:26:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:11:27:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:11:28:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:11:30:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:11:31:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:11:31:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:11:33:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:11:34:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:11:36:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:11:37:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:11:39:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:11:39:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:11:41:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:11:42:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:11:44:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:11:45:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:11:47:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:11:50:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:11:51:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:11:54:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:11:55:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:11:56:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:11:57:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:11:58:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:12:00:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:12:01:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:12:02:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:12:05:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:12:07:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:12:08:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:12:11:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:12:13:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:12:15:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:12:16:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:12:19:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:12:20:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:12:21:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:12:22:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:12:24:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:12:25:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:12:26:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:12:26:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:12:27:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:12:29:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:12:30:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:12:31:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:12:32:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:12:36:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:12:37:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:12:38:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:12:40:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:12:40:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:12:42:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:12:43:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:12:44:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:12:45:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:12:45:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:12:48:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:12:49:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:12:50:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:12:52:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:12:53:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:12:55:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:12:56:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:12:57:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:13:00:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:13:02:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:13:04:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:13:05:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:13:05:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:13:07:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:13:09:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:13:10:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:13:11:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:13:12:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:13:13:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:13:16:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:13:17:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:13:19:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:13:21:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:13:22:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:13:24:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:13:26:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:13:27:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:13:28:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:13:30:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:13:31:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:13:33:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:13:35:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:13:37:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:13:38:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:13:39:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:13:41:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:13:43:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:13:46:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:13:47:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:13:48:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:13:50:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:13:51:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:13:53:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:13:54:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:13:55:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:13:56:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:13:58:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:13:58:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:14:00:  Scheduler - Slave is not whitelisted for nfs limit.
2014-07-24 10:14:01:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:14:03:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:14:03:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:14:04:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:14:06:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:14:07:  Scheduler - Slave is not whitelisted for nfs limit.
2014-07-24 10:14:08:  Scheduler - Slave is not whitelisted for nfs limit.
2014-07-24 10:14:09:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:14:09:  Scheduler - The 53d13935e1fbb10ab4f892b9 limit is maxed out.
2014-07-24 10:14:10:  Scheduler - The nuke limit is maxed out.
2014-07-24 10:14:12:  Scheduler - Slave is not whitelisted for nfs limit.
2014-07-24 10:14:13:  Scheduler - The 53d13851e1fbb10754dd5390 limit is maxed out.
2014-07-24 10:14:13:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:14:15:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:14:16:  Scheduler - The 53d139fae1fbb103bcd02e41 limit is maxed out.
2014-07-24 10:14:17:  Scheduler - Slave is not whitelisted for nfs limit.
2014-07-24 10:14:18:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:14:19:  Scheduler - The 53d1340f54c2555f5073de9f limit is maxed out.
2014-07-24 10:14:19:  Scheduler - The 53d1368f13ead4333b6d0870 limit is maxed out.
2014-07-24 10:14:20:  Scheduler - The ingest2d limit is maxed out.
2014-07-24 10:14:20:  Scheduler - Slave has been marked bad for job 53d0c6eabbb1ed90683c6da7, skipping this job.
2014-07-24 10:14:21:  Scheduler - The 53d109df1c5e270c941dab57 limit is maxed out.
2014-07-24 10:14:21:  Scheduler - The 53d11b8b6df1883b6893451b limit is maxed out.
2014-07-24 10:14:22:  Scheduler - Slave is not whitelisted for nfs limit.
2014-07-24 10:14:22:  Scheduler - The 53d11d5177660d4d447d7dcc limit is maxed out.
2014-07-24 10:14:23:  Scheduler - Slave is not whitelisted for nfs limit.
2014-07-24 10:14:23:  Scheduler - Slave is not whitelisted for nfs limit.
2014-07-24 10:14:24:  Scheduler - Slave is not whitelisted for nfs limit.
2014-07-24 10:14:25:  Scheduler - Slave is not whitelisted for nfs limit.
2014-07-24 10:14:26:  Scheduler - Slave is not whitelisted for nfs limit.
2014-07-24 10:14:26:  Scheduler - Slave is not whitelisted for nfs limit.
2014-07-24 10:14:27:  Scheduler - Slave is not whitelisted for nfs limit.
2014-07-24 10:14:29:  Scheduler - The nuke limit is maxed out.
2014-07-24 10:14:30:  Scheduler - Slave is not whitelisted for nfs limit.
2014-07-24 10:14:31:  Scheduler - Slave is not whitelisted for nfs limit.
2014-07-24 10:14:32:  Scheduler - Slave is not whitelisted for nfs limit.
2014-07-24 10:14:32:  Scheduler - Returning limit stubs not in use.
2014-07-24 10:14:32:  Scheduler -   returning 53d05a3801f73e646c91bbdc
2014-07-24 10:14:33:  Scheduler -   returning 53cff32a6882d071dcdd130f
2014-07-24 10:14:33:  Scheduler -   returning vray3
2014-07-24 10:14:36:  Scheduler -   returning 53d029c144ec7a1dfc2b4698
2014-07-24 10:14:38:  Scheduler -   returning 53d063998b53a025d81d283c
2014-07-24 10:14:38:  Scheduler -   returning 53d063ae8b53a02278873b15
2014-07-24 10:14:39:  Scheduler -   returning 53d07bed8b53a01fbcd5aa24
2014-07-24 10:14:39:  Scheduler -   returning 53d07bbf8b53a00d249f5b0a
2014-07-24 10:14:39:  Scheduler -   returning 53d064338b53a008781efb58
2014-07-24 10:14:39:  Scheduler -   returning 53d0644c8b53a02ee46e4b40
2014-07-24 10:14:40:  Scheduler -   returning vray2
2014-07-24 10:14:40:  Scheduler -   returning 53d0781d8b53a018f896ab97
2014-07-24 10:14:40:  Scheduler -   returning 53d00ebabb649a191c5ce500
2014-07-24 10:14:41:  Scheduler -   returning 53cffb17bb649a22ec9b171b
2014-07-24 10:14:41:  Scheduler -   returning 53cffa8ebb649a0da887c195
2014-07-24 10:14:41:  Scheduler -   returning 53cffa0bbb649a1e9002ae9a
2014-07-24 10:14:42:  Scheduler -   returning mx_render
2014-07-24 10:14:43:  Scheduler -   returning mx_flowlinerender
2014-07-24 10:14:43:  Scheduler -   returning 53cffa48bb649a1d70857c56

This one dequeue check took 8 and a half minutes!!! From 10:06:15 - 10:14:43.
This is insane. Each of those separate requests lock the db every <1 second. On several hundred machines…

#2

Another curious thing is, why does it need to return tokens at the end? It didn’t dequeue anything!

#3

Yes, these long dequeue times are likely a result of heavy db load. Hopefully reducing how often the slaves update their state will help here, while as I mentioned in another thread, Deadline 7 will improve on performance as well.

When a slave goes to dequeue a task for a job, it tries to acquire the limit stubs that the job needs first. It will keep acquiring these stubs until it eventually finds a job it can work on, and then it will return any stubs it doesn’t need. This is so that once a slave has a stub during the dequeue process, it doesn’t have to try and acquire it again if the current job has no more available tasks, which would have a greater impact on performance than the current system.

Cheers,
Ryan

#4

I will try your suggested settings. Although, those updates are once every 10 seconds, while these dequeue related reads/writes happen multiple times each second. So i think the way dequeuing works is a larger contributor to the db overload than the actual slave info updates.

Could it simply get a list of jobs + all limit group stub states, then filter out the ones that don’t match, and just try the remaining ones? For example in this list, it should only do a “how many ingest2d” stubs are available test ONCE, then filter out all the jobs that require that stub right away.

Instead, it goes job by job, checking if the stub is available for each…

#5

As per your other thread, I’ve logged an issue to see if we can improve the limit acquisition algorithm. Loading ALL jobs and ALL limits might have efficiency problems as well, but maybe there is middle ground here.

Cheers,
Ryan

#6

Just an update. In Deadline 7, we will now be caching any failed limit stub attempts during the time that a slave is searching for a job. If a job requires a limit that is already in this failed limit cache, it will simply be skipped, instead of trying to acquire the limit stub again. Then at the end of each search, regardless of whether or not a task was actually dequeued, the cache is cleared so that the slave can start fresh during the next search.

This should help in cases where many jobs require the same limit, but that limit is maxed out. It will also help avoid redundant checks during the slave’s secondary pool scan (when the primary scan doesn’t yield any jobs).

Cheers,
Ryan

#7

Glorious thanks for the update Ryan!

Privacy | Site terms | Cookie preferences