housecleaning slow

[LaszloSebo]

With about 15k+ jobs, housecleaning is starting to slow down, with avg 12+ min runtimes.

The slowest bit seems to be parsing the folders of the repo:

2016-05-26 16:55:25:  Triggering House Cleaning Events
2016-05-26 16:56:40:  BEGIN - deadline01.scanlinevfxla.com\root
2016-05-26 16:56:40:  Deadline Command 8.0 [v8.0.1.0 Release (a63b4ba06)]
...
2016-05-26 16:56:53:  Purging Old Job Auxiliary Files
2016-05-26 16:56:53:      Auxiliary File Scan - Scanning for auxiliary directories
2016-05-26 17:03:56:      Auxiliary File Scan - Purged 2 auxiliary folders in 7.051 m

2016-05-26 17:03:56:  Purging Old Job Reports
2016-05-26 17:03:56:      Job Report Scan - Loading job report collections
2016-05-26 17:03:56:      Job Report Scan - Found 20034 report collections in 206.289 ms
2016-05-26 17:03:56:      Job Report Scan - Loading job IDs
2016-05-26 17:03:57:      Job Report Scan - Loaded 20033 job IDs in 249.655 ms
2016-05-26 17:05:31:      Job Report Scan - Purging job reports for '5746e0dd03337a64c477bdb3' because the job no longer exists
2016-05-26 17:05:31:      Job Report Scan - Purging job reports for '5746e0e003337a4d183130f0' because the job no longer exists
2016-05-26 17:07:17:      Job Report Scan - Purged 2 report collections in 3.341 m

But also the limit scan seems to take a while:

2016-05-26 17:07:37:  Purging Old Limits
2016-05-26 17:07:37:      Old Limit Scan - Loading machine limits
2016-05-26 17:07:37:      Old Limit Scan - Found 15419 machine limits in 318.952 ms
2016-05-26 17:07:37:      Old Limit Scan - Loading job IDs
2016-05-26 17:07:37:      Old Limit Scan - Loaded 20100 job IDs in 287.345 ms
2016-05-26 17:09:36:      Old Limit Scan - Purged 0 machine limits in 1.984 m
2016-05-26 17:09:36:      Old Limit Scan - Done.

[jgaudet]

Just took a look at this code, and it turns out we’re doing a lot of ‘.Contains’ on arrays that in your case would be 20k items long… gross.

Should hopefully be able to speed this up quite a bit just by switching that to use hash sets

[LaszloSebo]

Awesome man, cause we are grinning here trying to find the new limit of d8. So the more we can push this the better

[robert.crowther]

Had housecleaning take over 4 hours to complete for us. Just wanted to let you guys know!

2016-06-10 12:16:41: Job Report Scan - Purging job reports for ‘575a2af85c6aff15fc35a547’ because the job no longer exists
2016-06-10 12:16:42: Job Report Scan - Purging job reports for ‘575a2bb2fffe7026989ff72c’ because the job no longer exists
2016-06-10 12:16:42: Job Report Scan - Purged 6000 report collections in 4.076 hrs
2016-06-10 12:16:42: Job Report Scan - Purging old job report files
2016-06-10 12:16:42: Job Report Scan - Purging a maximum of 1000 report files

[jgaudet]

Which version was this running? Presumably this was after you guys moved to 8.0.2? I’ll have another look at the Job Report cleaning specifically, might be some speedups that I missed.

[LaszloSebo]

8.1.0.2, it seems like the deletion part is very slow. Not sure if thats just file access speed or something else

[jgaudet]

based on my napkin math 6k deletes over 4 hours is about 2-2.5s per delete – which is about consistent with what I’m seeing here testing with smaller numbers. I think it might be a combination of file access speed and likely some inefficiencies on our part – we should at least be able to get that number down some…

[LaszloSebo]

Sounds about right… do you think multithreading that process might help?

[jgaudet]

Could definitely be an avenue to explore, this is on my list of things to look at this week