Auxiliary files folder

Deadline
1

Hello Guyz !

Just wanted to share a recent experience where we loose more than 1 To of auxiliary files (~2000 jobs), if this can avoid this for anyone of you :mrgreen:

We recently begin a new production where our 3dsmax files (our primary activity) increased in size significantly ( 200-500 Mo each).
So we had to move all of submitted jobs from our standard repository place (1To SSD) to a stronger location ( VNX storage).

Thanks Deadline, this case has been planned.
So on a Testing repository i’ve tried to submit some jobs after activated Auxiliary alternate path.
Every think works fine so i decided to activate it on our main repo.

First mistake, i manually made the job copy from repo\path to alternate\path while it looks like it is deadline which is managing it. So has soon as i’ve activated it, the folder was clean… first stress !

I didn’t find any information in Doc about how it is managed :
Which machine is doing the copy ? when ? is there’s a priority ( creation time or priority or next to be dequeue ?..)
By the way, i wanted to avoid a network rush be copying jobs before enabling it, this was a fail.
I also wanted to have an activity recovery fast, and has my jobs where delete, i had some problems of jobs didn’t find because not yet in the alternate path.
#1 - would be nice to have more deep informations about how those copy are managed.

By the way, few hours after, the activity was stabilized and working fine.
No more space problems, except that now all slave must have access to the new path, but without mounting a network path (for obvious security reasons)

2 - could you add an authentification logging informations or use mapped paths to force all slave to access the auxiliary path ?

And last important mistake i’ve forgot, in my secondary testing repo, to delete the path to auxiliary files (which was the same path)
So, 2 weeks after the transition, we create activity on this 2nd repo (for testing another thing) and this had for consequences to immediate clean the auxiliary folder (has my 2nd repo only got 3 or 4 jobs).
We loose near to 1To of max files in seconds, corresponding to about 2k jobs.
Luckily, we were in a slack period, so we didn’t loose too many worktime / rendertime. We just re-submit the jobs that weren’t finished.
But how the hell a 3 jobs repo get overrides over my main repo when they were both pointing to the same path ?
#3 - how can we had security about massive file dropping after a desynchro between DB & repository ?

Now we have entered in a full production submitions and i’d like to be SURE that i won’t lost all job done by graphistes.
How can I, as overall administrator, receive notification (like repository space control ) or massive job deletion ?

Thanks !
-p-

2

Okay! Some details on how this works are definitely in order!

The Auxilary path cannot be shared between Repositories because of how cleanup occurs. Mainly if the machine running house cleaning does not recognize the job folders based on the information it has in the Repository it will delete them. There is no way for us to tell the difference between an old job folder for a deleted job. We ere on the side of cleanup since users would rather we make extra effort to clean up folders in case of file system error when the job was removed from the database.

Path mapping is an excellent idea, but we already have the ability to specify different folders for each operating system. Would the mapping just save typing, or is there another reason you’d like to see it? I’m not overly sure what you mean by authentication logging. Could you explain that in greater detail?

I’m not sure if there is a consistent way to be safe by not deleting files. As far as the house cleaning event goes, it believed it should clean things up. What would be a safe way for it to not delete large files that would not trigger in a healthy farm?

3

Maybe you could add, in the auxiliary folder an internal security, like read-only. And only the repository master ( which is linked to that folder) could delete jobs. If a 2nd repository come to life and is trying to cleanup this folder, it will send errors. I don’t know how you could do it.

My path mapping problem is :

  • The old path, was in the repository folder and free to access : standard windows shared folder
  • The new path is on a VNX and has to be password protected. So i have to give this login/password information to all my slaves, and it has to be persistante after reboot machine. That was just to avoid a bat sending all over the slaves :slight_smile: Now the path is correctly mounted on all slave, i don’t have any problems.
  • Does the mapped paths is used for replacing path from an OS to another or is it also used to simply be sure to access a path (as my needs) ?

Thanks !

4

I have no idea how to defined a master either. :smiley:

Actually, I’m not sure if it’s possible at all beyond having the user define one is a master, and at that point, it’s likely just easier to make sure you don’t use the same folder… We are putting some notes in the documentation about it though.

For the authentication… What about using mapped drives for this new share? Deadline is capable of doing that before a job starts, which should make sure things are mapped before the assets get synced over. Could that work?

5

Yep could work, but this means, all my 80 graphistes (from storyboard to final edit, aka not realated with any renders ^^) have a direct access to my job folder, and I don’t need that :slight_smile:
I’m doing the same thing ( /net use) but it make the connection invisible.

For the master repository, you could try the Adobe Style : a .TXT file at the root of the folder with the repository path :mrgreen: