Hey there,
Did a quick sum of the log folder contents, and have found that it:
Has 5.6 gigs of theoretical size, and 462Gigs (!) of actual on disk size
Contains 1.2 million files in 8706 folders.
Seems excessive… Would also explain why deleting jobs takes that long. Could that data be compacted, at least for complete jobs?
Each log file is already bzipped. We do and indexed-based folder structure to provide even distribution of logs across folders. A job’s logs can be spread across hundreds of folders (along with the logs from all other jobs), so I’m not sure how we would go about compacting it…
It seems to be a bottleneck when you have thousands of jobs. Is there a reason why the logs have to be randomly distributed? Why not just have it organized in folders by jobID?
That would simplify ‘compacting’.
For that you could have:
Maybe for 7.0 we could change it so that we use the job ID as the subfolder inside the existing indexed folder system, and then all the job’s reports would go in that subfolder. It’s pretty much how we were thinking of setting up the job auxiliary file subfolders. Then they shouldn’t have to be compacted, since deletion would just involve deleting a directory…