AWS Thinkbox Discussion Forums

Selective asset sync when cloud rendering

Hi guys,

We have a use case where we submit a simulation job on the cloud, it generates a large cache, then a dependent job starts to render, and we get the renders back. Now, we only want the renders, not the cache back, which is quite huge. Is there an option to selectively only sync some assets?

Thanks

I know we have a blacklist to prevent copying back certain large files to your office. I’m not sure however if that will prevents saving a copy to the S3 bucket or not. I’ve asked the dev team here and we’ll see what they say.

Just to make sure - We’ll need the first job to save it, the second to render it, but not to sync back the caches.

Thanks

Can you clarify what type of cache you are producing? What is the format extension?
Under the Tools>Asset Server Settings, there is a long list of white-listed formats that may be synced back. They include .abc, .pc, .prt, .vrmesh, .vrscene, .rs and .ass files, among others. If you want to keep those files in the cloud without syncing them back to the local storage, you should remove them from the list. If the file format is something else not listed there and it still gets copied back, please let us know.

As far as I can see, .aur, .fxd and .vdb are not on the whitelist, so I guess you are not talking about fluid caches?

Ah that’s great - if we could say which formats can be synced in which direction, we should be all set. I’ll check it out in the morning

There’s another wrinkle I hadn’t considered. The render nodes have a cache locally for these files, so it’s possible that if you SIM doesn’t fit into the working set of the local cache you may run into problems.

Windows-based nodes currently have a limitation that they cannot directly sync from S3, so they share a volume over Samba from the Gateway instance. You can actually see that additional volume attached to that instance and grow it as required, but that means the network can be the bottleneck (S3 access is very quick).

So far I’m told that files not in the whitelist will be blocked from uploading to S3 so it’s likely that the SIM has to come back either way for now. You can see that whitelist here: (find this in the ‘tools’ menu)

[attachment=0]2018-05-30 09_53_00-support-win-01.thinkbox.corp.amazon.com - Remote Desktop Connection.png[/attachment]

1 Like
Privacy | Site terms | Cookie preferences