AWS Thinkbox Discussion Forums

Caching For Deadline

I know we spoke a long time about this, but we’ve had a few things recently which have caused us some headaches which would be easily solved using caching to the local render server.

Streaming assets like Vrmesh, tx and tileEXR require a connection left open to the file server during render… A recent job had 1000 vrmesh files and this meant there were say 100 machines rendering, causing 100,000 concurrent connections to the file server. Which slows things down and causes big rendering issues like Deadline not being able to save files back and some machines not being able to open a file.

We implemented a simple caching system to solve this problem but it would be good if Deadline had this built in so we could have efficient checkSum verification and cache clearing jobs.

We did start work on that in Maya and there’s preliminary support. Check it out here:

2017-08-08 08_50_41-support-01 - Remote Desktop Connection.png

I’d be curious to see if it’ll work for the vrmeshs and friends. Also, I’m assuming you’re using Maya since Max has had the ability to copy everything for awhile now and Cinema 4D is slightly less likely and I had to guess. :smiley:

Fun thing with checksums: Often times it takes longer to compute them than is worthwhile, so I think we use the last modified value on these assets and hope the files don’t become corrupted in transit (which is pretty rare).

The 3dsmax option would collect all the necessary files and put them in the repository correct?

Yup. It’ll also copy all of them to the Slave’s local temporary directory.

Anything that doesn’t copy over we’d consider a bug. Sometimes we have to rename assets that may be in different folders with the same file name, but I believe we handle that fairly gracefully too.

Yeah that’s not the most efficient way to transfer the 10gb of assets…

Our current system relies on strict versioning of assets so we never need to overwrite any cached files, because the only updates should be in new files.

We have a post-open script that substitutes the network path letter for the C drive and looks on the local disk for the file, if it doesn’t exist it copies it down and repaths, or if it does exist on the local machine it uses that path instead.

Works for some simple projects, but obviously will fall apart on some larger projects.

We completely agree with you there. Especially over a pipe as small as the Internet. We’re working on something better that’s coming in Deadline 10, but Maya is getting the love first because remapping things in 3DSMax land is difficult (can’t happen within Max itself). I also don’t believe this will work for local farms as of yet, but I’m hoping what we learn from Deadline 10 will apply to caching to local nodes.

Just an FYI, don’t know if it affects Maya as well but if you remap a vrmesh file to a new location it will wipe the include/exclude list in the mesh visibility properties in 3dsmax. I had to write a store and reset function either side of changing the vrmesh path.

that smells like a bug that should be fixed in V-Ray. Have you reported it?

Yep but I think even Vlado might have reached saturation point with the number of issues I’ve been submitting lately!

Privacy | Site terms | Cookie preferences