Deadline reports extreme number of corrupted jobs

Deadline
1

This is a strange one.

I’ve been testing some scripting and some new plugins on my farm and while I was constantly performing the submit new jobt - suspend job - resubmit job - suspend job - delete job sequence, I received about 200 e-mails from Deadline notifying me of corrupted jobs in my repository. I submitted, overall, about 8 jobs, but I received almost 200 notifications!

This is crazy and it is being worse and worse as I see Deadline doing this more and more often.

Any thoughts?

My repository sits on a file server which, otherwise, has no problems what so ever with anything. It serves files without issues, all goes well. Only Deadline seems to complain.

2

Interestingly enough, most of the error reports are being sent from my license server where Pulse is also being run (but with issues). I disabled Pulse for now as I first need to resolve the super-slow-responses from Deadline when connecting to the Pulse running on the license server.

3

It seems only two machines report these corrupted jobs.

The, already mentioned, license server that I want to run Pulse on and one of the five render slaves.

I honestly have no idea what is going on in there.

Could this be a .NET version mismatch? The error reports mention: String was not recognized as a valid DateTime. which might suggest a .NET version mismatch as the latest render node as well as the lic. server are the newest machines in the whole setup. Hmm… I’ll try to run a Win. Update on them all as I have it disabled and see if that helps.

4

Can you confirm that all your render nodes and workstations have the same version of Deadline installed? If the problem is only happening on a couple machines, maybe it’s a version mismatch that is causing this problem.

Cheers,

  • Ryan
5

The Deadline monitor shows all the nodes running the v4.1.0.43205 R

And also the dedicated License and Pulse server when Pulse is fired up reports about 100 corrupted jobs (even though I’m currently having only 4 jobs in the queue and others are deleted).

6

By any chance do any of your machines have a German language OS installed? We’ve just discovered a Date/Time conversion issue on German OS’s that can result in corrupted jobs. I’m just curious if the problem you’re having is related…

Thanks,

  • Ryan
7

The same could be said for other European languages, I assume. Czech, for example :slight_smile:

8

hi,

we have the same problem. is there a fix for this problem at the time?

kind regards

9

Of course! Now it makes sense!

Although I have strictly EN versions of Windows on all my machines, the Regional and Language settings are all set to Czech and Czech Republic, incl. the date and time and currency values etc…

Can this be fixed in Deadline? I mean, there has to be a way in .NET for this to be corrected, right?

I still get tons of “corrupted jobs” reports from the primary license server. The render slaves should be left intact as we don’t need to work on these. However the Workstations are all set for our locales.

10

This can be fixed, and we already have a plan on how to do it. I guess a current workaround would be to modify the Short Date format in your regional settings. Deadline expects the format to look like this:

11/12/2010

Anything other than this, like “11.12.2010”, would cause the jobs to get corrupted.

Cheers,

  • Ryan
11

Thanks for the tip. I just modified the short date on the Pulse machine as that was the one reporting all the corrupted jobs.

However, I also have a serious problem with submitting a Nuke job to the repo via the Monitor as the job gets flagged Corrupted immediately and doesn’t go through.

I’ll test to re-submit again since I changed the locale settings (I’ll restart the server), for now here’s the concrete job that got flagged immediately as well as what Pulse is reporting. Hope it helps.

By the way, do I have to change the short date format on all my machines?
corrupted_job.zip (36.1 KB)

12

Just letting you know that it seems, so far, to have helped! Thank you!

13

Thanks for the update! Glad to hear that works for now.

14

We also have this problem (using german os)… any eta on a real fix?

15

It will be fixed in the next Deadline release. We’re hoping to start the beta soon, but unfortunately with all the internal changes from transitioning to Thinkbox Software, I can’t give an exact ETA at this time.

Cheers,

  • Ryan
16

Changing the short date on our german OS Systems worked perfectly.
No more waiting till slave grabs a job + priorities work now.

great that this problem has been solved (with a workaround) and will be gone in the next release (hopefully)