I’ve been trying to figure out what’s going on with our production renders on 3DSMAX / VRAY GPU / DEADLINE for the past 2 months.
I have searched, all the options I knew, all the tests I knew how to do and I managed to “circle” the problem but impossible to understand what causes it. There is strong chance that the problem is more on the side of DEADLINE than VRAY / 3DSMAX, but I’ve asked on all side.
VRAY GPU 6.00.20
We are working on a project with jewelry (as often in the studio) and the jewels arrive in 3DSMAX in .ABC from animations and rigs with position constraints made in MAYA.
The .ABC are composed of multiple sub-objects corresponding to each element of the jewel (in this case links of chains among others)
The loading of .ABC in 3DSMAX is done without worries, then shading and lighting in 3DSMAX, then we send all that on DEADLINE for the rendering. Classic.
Our concern is that when 3DSMAX will load for the first time the project on deadline order, one of the sub-objects of the ABC can load badly and be positioned at the first frame of the ABC.
Computer 10 arrives in the DEADLINE and receives the order to render frame 43 because the other 42 frames are already rendered or being rendered by other computers.
Computer 10 loads the scene (without any apparent linked error on DEADLINE) and renders frame 43 (without any apparent linked error on DEADLINE)
In this precise case (first time that the computer loads the scene) it is “possible” that one of the sub-objects of the ABC loads the position of this sub-object as if it renders the frame 00 (we are not 100% sure of this analysis, is it the timing of the ABC, is it the position contriantes that jump …).
And we are talking here about a single sub-object (always a single one, never 2), any one of them (no matter size, polycount …), on any computer (we have 10).
When we restart the task manually to correct the problem on DEADLINE, the frame is restarted by an available computer and the dice are rerun. It is quite possible that this frame has the same problem but on another subobject of the ABC. Still only one object, but any object. It is also possible that the frame renders well and that the problem does not arise.
When the computer moves on to the second frame of its list, there are no more errors of this kind, until it finishes its job, but without ever leaving the queue. If for some reason it has to reload the scene via DEADLINE, the dice are rolled again and the error can return on any of the sub-objects.
Things got even more murky for me when we discovered that this could also affect an animated VRAY light.
The scene can have 10 VRAY lights, just the animated one can have a one frame rollback of a single light when 3DSMAX loads the scene via DEADLINE. According to exactly the same rules as the problem with ABCs.
We have tried a LOT of solutions, none of them worked for the moment.
There is no other third party plugin needed to render these frames.
We were in 3DSMAX 2021, VRAY 5XXXX and DEADLINE 10.1.20.3, we updated everything, it did not change anything.
We made multiple tests with our weak knowledge in DEADLINE. All our tests proved to be unsuccessful for the moment.
We thought of a network problem, but DEADLINE not indicating ANY related error, it seems to me a weak hypothesis.
We made tests in CPU and it seems that the problem does not exist (same scene, same setup, just switch to CPU).
We made tests in GPU with a local render (no DEADLINE involved) and the render is perfectly OK
Unfortunately, I can’t share in public the scenes in question as I am under NDA for this project. I can eventually transmit the scene via the contact form if needed. I have attached a small piece of rendering to illustrate my point.
I can also provide a DEADLINe Log of a bad frame (and a good one also) if needed.
I’ve share a similar post on CHAOS forum in parallel of this one.
We have already made multiple 3DSMAX / VRAY GPU / DEADLINE projects without encountering similar problems.
We have found no echo of our issue on any forum we search on (CHAOS forum, AUTODESK forum, THINKBOX forum)
Thanks in advance for your time and your answers,