AWS Thinkbox Discussion Forums

Render nodes occasionally render frames without HDRI map

Deadline
#1

Hi,

This is a really strange problem.

We have 10 PCs/rendernodes on a farm, rendering with 3DS max and V-ray. We often light our scenes with HDRI. Occasionally some rendered scenes have frames that look darker. When we look closer we can see that the frames that are darker simply have the dome light either turned off or missing the HDRI map (not really sure which). If this happens, it will only be one of the PC’s on the farm that does this, so luckily we can tell Deadline (that we are using to manage the farm) to re-render all the frames done by that PC. But I’d still like to resolve the problem as it wastes time.

What is really weird is that this doesn’t happen on every scene, and when it does happen is can be any PC/rendernode, so its not like one of the PC’s is having this problem, it can be any of them.

Because it doesn’t happen on every job and when it does happen its on a different rendernode each time. I’m really struggling to diagnose what could be causing the problem.

I think all this did start when we got a new Qnap NAS drive. But it could be a coincidence. But even if it is that, what specifically could it be?

Here are some thoughts that I have on what could be causing it.

  1. The Scene is taking up too much RAM on the render node and therefore is not loading in the HDRI
  2. The network is being oveloaded with traffic and so the rendernode can’t access the HDRI file
  3. There is some kind of licencing problem with VRAY
  4. There is some abnormally with the network paths or location of the HDRI

Most of the above, I think would not be the problem, as things like licencing issues would simply stop the rendernode from rendering, not make it unable to see a HDRI map?

If anyone has any thoughts suggestions or advice on trying to find out what is going on, that would be great.

It would be great if I could get deadline to fail frames where the node can’t find/see the map/texture. then I could hopefully see an error in the log that says that the frame failed due to a missing map, or maybe something more helpful. Any advice on trying to trouble shoot this would be great. I have occasionally noticed that the nodes miss the odd texture too from frames.

On the v-ray forum they have suggested its because of oplocks, but I’ve tried turning that off and it still does it. I’m really stuck:

Many thanks

Luc

#2

Are there any callouts in the logs that mention the HDRI not getting loaded? To save the manual intervention a standard output handler could be added to fail tasks that don’t load the HDRI for whatever reason, as long as we’ve got some message that we can trigger against.

Usually we point folks at Troubleshooting Render Issues in 3ds Max for Max issues, but that’s only really helpful isolating issues that are happening reliably.

For your theories:

  1. The Scene is taking up too much RAM on the render node and therefore is not loading in the HDRI

Maybe! At the end of the task report will be the peak and average memory usage - are they higher than the frames that render normally?

  1. The network is being oveloaded with traffic and so the rendernode can’t access the HDRI file

I’d expect to see some ‘file not found’ or ‘failed to read’ error in the log in that case.

  1. There is some kind of licencing problem with VRAY

That should kill the whole frame, and it should do it with a really obvious error message.

  1. There is some abnormally with the network paths or location of the HDRI

Also possible, but I don’t even have a first guess on that one.

I’d be curious if running the render in 3dsCMD shows the error - as at least with that you’d have pulled Deadline out of the issue and at least you’d have a simpler problem space.

Privacy | Site terms | Cookie preferences