Draft Event Plugin Race Condition

Deadline
1

Hey all.

We have our render farm set up with a draft plugin to create Draft jobs that render videos and upload them to Shotgun when the initial job has rendered out the frames.

Recently we set up a new more powerful machine that has been set up with 6 slave instances running concurrently. We noticed that more often there were jobs that finished without actually creating the draft jobs, and when I looked at the post job reports, I saw while:

[code]=======================================================
Log

2017-10-05 09:27:04: BEGIN - BLADE2-WIN10\cartoon
2017-10-05 09:27:04: Adjusting framerate to 23.976 for Pete the Cat
2017-10-05 09:27:07: Set new version ID: 101327
2017-10-05 09:27:07: Submitting Shotgun H264 Movie Creation Job to Deadline…
2017-10-05 09:27:08: Deadline Command BG 7.2 [v7.2.0.18 R (79a3ddcab)]
2017-10-05 09:27:08: Error: The process cannot access the file ‘C:\Users\cartoon\AppData\Local\Thinkbox\Deadline7\temp\draft_event_Shotgun_H264_Movie_Creation_0.job’ because it is being used by another process. (System.IO.IOException)
2017-10-05 09:27:08: at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
2017-10-05 09:27:08: at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
2017-10-05 09:27:08: at System.IO.FileStream…ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
2017-10-05 09:27:08: at System.IO.StreamReader…ctor(String path, Encoding encoding, Boolean detectEncodingFromByteOrderMarks, Int32 bufferSize, Boolean checkHost)
2017-10-05 09:27:08: at System.IO.StreamReader…ctor(String path, Encoding encoding)
2017-10-05 09:27:08: at FranticX.Text.DictionaryReader.FromTextFile(String fileName, Boolean acceptEmptyValue, Boolean acceptControlCharacters, Boolean trimKeyValues)
2017-10-05 09:27:08: at Deadline.Submission.SubmissionUtils.SubmitNewJob(DataController dataController, String[] args)
2017-10-05 09:27:08: at Deadline.Submission.Submit.Perform(String[] args)
2017-10-05 09:27:08: Adjusting framerate to 23.976 for Pete the Cat
2017-10-05 09:27:11: Set new version ID: 101327
2017-10-05 09:27:11: Submitting Shotgun Filmstrip Creation Job to Deadline…
2017-10-05 09:27:12: Deadline Command BG 7.2 [v7.2.0.18 R (79a3ddcab)]
2017-10-05 09:27:12: Error: The process cannot access the file ‘C:\Users\cartoon\AppData\Local\Thinkbox\Deadline7\temp\draft_event_Shotgun_Filmstrip_Creation_0.job’ because it is being used by another process. (System.IO.IOException)
2017-10-05 09:27:12: at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
2017-10-05 09:27:12: at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
2017-10-05 09:27:12: at System.IO.FileStream…ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
2017-10-05 09:27:12: at System.IO.StreamReader…ctor(String path, Encoding encoding, Boolean detectEncodingFromByteOrderMarks, Int32 bufferSize, Boolean checkHost)
2017-10-05 09:27:12: at System.IO.StreamReader…ctor(String path, Encoding encoding)
2017-10-05 09:27:12: at FranticX.Text.DictionaryReader.FromTextFile(String fileName, Boolean acceptEmptyValue, Boolean acceptControlCharacters, Boolean trimKeyValues)
2017-10-05 09:27:12: at Deadline.Submission.SubmissionUtils.SubmitNewJob(DataController dataController, String[] args)
2017-10-05 09:27:12: at Deadline.Submission.Submit.Perform(String[] args)[/code]

It looks like they’re all just using the same file name, so if multiple instances on the same machine try to run at the same time then they clash because they obvious can’t overwrite the file while another machine is using it.

We’re still on Deadline 7.2 so I don’t know if this has been fixed in later releases, but I thought I should flag it before I look through for a way to solve it myself.

2

It doesn’t look like it has actually. I’ll log that as an issue. We’ll just need to change the call to GetDeadlineTempPath() use something a bit more random.

3

Thanks! Out of curiosity, how do you plan to do that? Just because I imagine you’d want to avoid it being totally random and having a series of files created that never gets cleared out. I initially thought that basing it off the slave name would be a good idea, but it didn’t seem to be as easily accessible as I hoped so we’re opting to just incrementing numbers when a file write fails. This way it’ll still overwrite jobfilename.job and jobfilename_1.job, without creating many files left on the folder.

4

Ideally we’d use DeadlinePlugin.CreateTempDirectory():
docs.thinkboxsoftware.com/produ … 5199f9ed8c

That registers the folder to get cleaned up when the Slave picks up a new job. It’s automatically perfect.