[Bug] MaxStartup file sync

Deadline
1

We are running into some odd behavior with the new 3dsmax startup file sync process. Previously these files were all in the 3dsmax plugin folder, but now live under repository\maxStartup<maxversion>, and are synced to the local machine by the 3dsmax.py process.

In some cases, we are seeing the slave fail during the sync with the following error:

Initialize: Error: IOError : [Errno 13] Permission denied: u'C:\\Users\\scanlinevfx\\AppData\\Local\\Thinkbox\\Deadline10\\slave\\lapro0722\\maxStartup\\2016\\deadlineStartupMax2016.xml' (Python.Runtime.PythonException) File "C:\Users\scanlinevfx\AppData\Local\Thinkbox\Deadline10\slave\lapro0722\plugins\5a9366014358ee33f4b40f3a\3dsmax.py", line 107, in InitializeProcess self.MaxStartupAutoUpdate( maxStartupLocalPath, maxStartupRepoPath ) File "C:\Users\scanlinevfx\AppData\Local\Thinkbox\Deadline10\slave\lapro0722\plugins\5a9366014358ee33f4b40f3a\3dsmax.py", line 86, in MaxStartupAutoUpdate shutil.copy2( os.path.join(repoPath, fileName), os.path.join(localPath, fileName) ) File "C:\Program Files\Thinkbox\Deadline10\bin\lib\shutil.py", line 130, in copy2 copyfile(src, dst) File "C:\Program Files\Thinkbox\Deadline10\bin\lib\shutil.py", line 83, in copyfile with open(dst, 'wb') as fdst:

The server files are not readonly, but on occasion we have observed the local files to become readonly for some odd reason.

In addition, another oddity is that in all cases, we have found an additional file to appear in this local sync folder named “toServer.txt” with the word “ready” in the file. This additional file causes a resynchronization of the startup files in every case (so the local files are not acting like a cache), because the code that checks whether there should be any resync or not detects the ‘toServer.txt’ file as a difference:

dcmp = dircmp( localPath, repoPath ) diffFiles = len(dcmp.diff_files) + len( dcmp.left_only ) + len( dcmp.right_only )

dcmp.left_only will contain toServer.txt
Somewhat related to this, but a resync does not clean the destination folder, simply copies the files from the network again, this to toServer file will never disappear.
Not sure what makes the toServer.txt file, as it sometimes contains errors like this:

error An unexpected exception has occurred in the network renderer and it is terminating.

It seems that the file is being made there by accident (maybe active folder is changed during the sync copy?)

cheers
laszlo

2

I’ll go open a dev issue for this one. Do you think moving the sync folder into the plugininfo folder would help here? The Slave erases it when it moves onto another job, so the overhead will be larger than with the new system.

Do you need help working around this one or does your pipeline team have a fix in place already? Trying to gauge severity as you’re the first to report this one.

3

We can probably work around it by patching 3dsmax.py to remove the read-only flags from the files, although its a mystery what makes them readonly in the first place…

What i can’t really figure out is where the toServer.txt file is coming from and whether its ok to delete it. The existance of that file (and it seems like it keeps popping up) means the current solution provides the same overhead as the previous system - as it forces a resync every time the 3dsmax plugin is used.

4

I may have figured this out:

Seems we might want to not use copy2 because it’ll take the permission settings from the Repo files.

5

The repo path is NOT read-only though… thats whats confusing here.
Also, the metadata includes things like modified date, which is probably something that should be copied.

6

Good argument. Maybe there is an issue with how copystat() is bringing files over (which is what’s bringing the metadata through). They changed things quite a bit from 2.7 to 3:

github.com/python/cpython/blob/ … il.py#L173
github.com/python/cpython/blob/ … il.py#L107

The 3DS Max plugin is the only place we use copy2 at the moment in the entire repository so there isn’t much we can use for A/B testing. I wonder if we should either change the read-only property or think of a different way around this. Silently not-updating the files there doesn’t seem like a good plan to me.

7

Hey Lazlo,

Here is a new 3dsmax plugin that should help address this issue. I have updated the sync to remove all files from the destination directory before copying which should make it so it should no longer get stuck in loops. I have also updated it to make the copied files write accessible by default.

For the toServer.txt, that is a file that is automatically being created by Max and should not be being created in that directory. I have added a some logging to the plugin that will say where this should be being created. If you continue to see this being created then any logs would be useful for tracking down where the error is occurring.

Grant
3dsmax.zip (36.5 KB)

8

Thanks Grant!

9

Hey Laszlo, are you folks using concurrent tasks? We have a guess here that concurrent tasks might cause a race-condition if we’re deleting that cache. I’m not sure if that’s the problem here, but I wanted to make sure.

10

No, we aren’t using concurrent tasks. On some machines we have 2 slaves running, but they each do their own thing (and always different plugins).