AWS Thinkbox Discussion Forums

3dsmax / vray gpu / deadline

I’ve been trying to figure out what’s going on with our production renders on 3DSMAX / VRAY GPU / DEADLINE for the past 2 months.

I have searched, all the options I knew, all the tests I knew how to do and I managed to “circle” the problem but impossible to understand what causes it. There is strong chance that the problem is more on the side of DEADLINE than VRAY / 3DSMAX, but I’ve asked on all side.

Software :
3DSMAX 2023.2.2
VRAY GPU 6.00.20
DEADLINE 10.1.23.6
NVIDIA 512.59

We are working on a project with jewelry (as often in the studio) and the jewels arrive in 3DSMAX in .ABC from animations and rigs with position constraints made in MAYA.
The .ABC are composed of multiple sub-objects corresponding to each element of the jewel (in this case links of chains among others)

The loading of .ABC in 3DSMAX is done without worries, then shading and lighting in 3DSMAX, then we send all that on DEADLINE for the rendering. Classic.

Our concern is that when 3DSMAX will load for the first time the project on deadline order, one of the sub-objects of the ABC can load badly and be positioned at the first frame of the ABC.

Example:
Computer 10 arrives in the DEADLINE and receives the order to render frame 43 because the other 42 frames are already rendered or being rendered by other computers.
Computer 10 loads the scene (without any apparent linked error on DEADLINE) and renders frame 43 (without any apparent linked error on DEADLINE)
In this precise case (first time that the computer loads the scene) it is “possible” that one of the sub-objects of the ABC loads the position of this sub-object as if it renders the frame 00 (we are not 100% sure of this analysis, is it the timing of the ABC, is it the position contriantes that jump …).
And we are talking here about a single sub-object (always a single one, never 2), any one of them (no matter size, polycount …), on any computer (we have 10).

When we restart the task manually to correct the problem on DEADLINE, the frame is restarted by an available computer and the dice are rerun. It is quite possible that this frame has the same problem but on another subobject of the ABC. Still only one object, but any object. It is also possible that the frame renders well and that the problem does not arise.

When the computer moves on to the second frame of its list, there are no more errors of this kind, until it finishes its job, but without ever leaving the queue. If for some reason it has to reload the scene via DEADLINE, the dice are rolled again and the error can return on any of the sub-objects.

Things got even more murky for me when we discovered that this could also affect an animated VRAY light.
The scene can have 10 VRAY lights, just the animated one can have a one frame rollback of a single light when 3DSMAX loads the scene via DEADLINE. According to exactly the same rules as the problem with ABCs.

We have tried a LOT of solutions, none of them worked for the moment.
There is no other third party plugin needed to render these frames.

We were in 3DSMAX 2021, VRAY 5XXXX and DEADLINE 10.1.20.3, we updated everything, it did not change anything.
We made multiple tests with our weak knowledge in DEADLINE. All our tests proved to be unsuccessful for the moment.
We thought of a network problem, but DEADLINE not indicating ANY related error, it seems to me a weak hypothesis.
We made tests in CPU and it seems that the problem does not exist (same scene, same setup, just switch to CPU).
We made tests in GPU with a local render (no DEADLINE involved) and the render is perfectly OK

Unfortunately, I can’t share in public the scenes in question as I am under NDA for this project. I can eventually transmit the scene via the contact form if needed. I have attached a small piece of rendering to illustrate my point.
I can also provide a DEADLINe Log of a bad frame (and a good one also) if needed.

I’ve share a similar post on CHAOS forum in parallel of this one.

We have already made multiple 3DSMAX / VRAY GPU / DEADLINE projects without encountering similar problems.
We have found no echo of our issue on any forum we search on (CHAOS forum, AUTODESK forum, THINKBOX forum)

Thanks in advance for your time and your answers,
Nicolas


Here is the job log of a bad rendered frame, if someone see something that shouldn’t be there (there is a couple of warning but they dont looklike involved in the issue)
Job_2022-11-17_02-03-29_6375885f5b23db1700413ead.zip (7.2 KB)

Hello again,

I try to add as much detail as possible as we research, test and understand the problem internally.
We imported an old ABC from another project to serve as a test more freely and simply than the production files we were using for testing.

I confirm that on the same new simple scene, with the same DEADLINE settings, the same sources and the basic VRAY

  • No worries on the CPU version
  • No problems on an ABC loaded via a VRAY PROXY (unfortunately it’s more complicated for us to use this method, because of our shading and lighting piepline)
  • ABC problems with a ABC imported in the classical way directly via 3DSMAX import.

In the videos attached (very fast test on a sample scene) we ask DEADLINE to reload the scene after each frame in order to generate a maximum of bug (this option is checked in the three renders).

I cannot for the moment provide the basic MAX scene for creative properties reason of the 3D models but I can share the 3D video renders (there is a small normal issue on the chain but it’s not relevant here).
Linked in this post (WeTransfer - Send Large Files & Share Photos Online - Up to 2GB Free) :

  • TEST_ABC_GPU_VRAYPROXY.mp4 (abc file loaded in a Vrayproxy rendered in GPU)
  • TEST_ABC_CPU.mp4 (abc file loaded via 3DSMAX > import > .abc rendered in CPU)
  • TEST_ABC_GPU_V01.mp4 (abc file loaded via 3DSMAX > import > .abc, V01 is exactly the same .max file as V02 rerendered a second time to show that the bugs a not sowing in the same frame with the same objet, it’s totally random. Rendered in GPU)
  • TEST_ABC_GPU_V02.mp4

Has it already happened to someone? I can’t find out if it’s unique to our setup or if the error has already been encountered by someone else.

Any help is welcome,
Thanks in advance,
Nicolas

Privacy | Site terms | Cookie preferences