Hi Folks!
I’m not sure if this is a vray thing or a deadline thing but I notice that in a few cases when I use light cache in single frame mode and network render on some machines in here (mainly my own - the vray license server) light cache calculates for a daft amount of time and generally I cut it off and requeue the frame - I don’t think it ever gets to the end and fo0r example I’ve had 10 minute renders still calcing the lc after 6.5 hours.There’s nothing bonkers in the scenes - static camera moves over regular objects normally and if it’s a moving objects I normally use brute for / single frame lc.
Any thoughts?
Best regards,
John
Hey John,
Sorry for the delayed reply - it was a long weekend here. 
It’s difficult to say if this is a deadline or vray thing. Do you find it happens only on a few particular machines, or is this more of a random occurrence that happens on all of your machines? In the few cases where it does happen, is there anything the scene files have in common that could explain this? Just trying to narrow down the problem.
Deadline does have a timeout option you can set when submitting your job. If you know ahead of time that your frames will take roughly 10 minutes to render, you can set the timeout to something like 20 or 30 minutes. When a timeout occurs, an error is reported and the task is requeued automatically. If we can’t figure out what’s going on, this could be a viable workaround.
Cheers,
Heya Ryan,
I put the same post on the vray forum and though it was initially related to light cache calculation - Vlado though it might be due to dynamic geometry but I just had the same issue with a scene that has no dynamic bits and only uses a single brute force bounce - no light cache or other secondary. As for the workaround it sounds like a plan alright but again if we’ve got a mix of machines that range in speeds having a timeout that suits a slow machine might waste a lot of cycles on a fast machine tied up on a bad frame, likewise too low a timeout might kill frames going on a slow machine that are going fine but don’t meet the time - if a render is left over night and there’s a really nasty frame - say something with heavy motion blur it could be a freak frame in terms of the normal render times so again if unmanned it could time out under the limit and not be done for the morning - not fun if you’ve something due for first thing but you need one frame that takes 2 hours to go through.
Again these are all things you’re aware of since you’ve a far harsher render environment than me but I don’t think I had this issue with backburner.
Cheers,
John
Can you post a simple scene file that we can use to try and reproduce the problem? Also, the next time this happens, can you go to the slave and copy and paste the last 100 lines or so from the slave interface into a post? Finally, can you let us know if this happens on a few particular machines, or if this is more of a random occurrence that happens on all of your machines?
In Deadline 3.0, there will be some improvements to the time out feature. For starters, there will be a Normalized Task Timeout Multiplier setting for the slaves. So you can set the multiplier to different values for machines with different speeds. There will also be an Auto Timeout feature that will dynamically figure out what the timeout value should be based on previously completed tasks and some settings in the repository options. If we can’t solve this problem, Deadline 3.0 (which should be released in the next week or two) will provide better timeout options that you can use.
Cheers,
Heya Ryan!
Yep it’s a bit of a random thing but it seems to happens on some machines more than others - they seem to be the quad core machines in here. Here’s a log file from my own where the lc overran. Also here’s a frame (perhaps the same one) where it got through to rendering correctly.
Hopefully something useful is in the log files, I’ll pop up a simple scene later that caused issues. Hopefully they can get sorted, we’re going to be going into an xsi and max mixed environment so the more solid the better!
Cheers,
John
john_good_frame_log.zip (6.66 KB)
john_bad_frame_log.zip (14.4 KB)
Thanks for the logs! So yes, from the log of the bad frame, it’s obvious it’s getting stuck when building the light cache. So I scrolled up a bit and found that Deadline was printing out this:
0: STDOUT: +lightcache_subdivs set to 1000
0: STDOUT: +lightcache_sampleSize set to 0.02
0: STDOUT: +lightcache_filter_type set to 1
0: STDOUT: +lightcache_filter_size set to 0.04
0: STDOUT: +lightcache_bounces set to 100
0: STDOUT: +lightcache_showCalcPhase set to 1
0: STDOUT: +lightcache_storeDirectLight set to 1
0: STDOUT: +lightcache_scale set to 0
0: STDOUT: +lightcache_mode set to 0
0: STDOUT: --Failed to set lightcache_loadFileName to T:\3D\XSI_DB\ESB\Scenes\sh13_03.vrlmap
0: STDOUT: --Failed to set lightcache_saveFileName to undefined
0: STDOUT: +lightcache_interpSamples set to 10
0: STDOUT: +lightcache_prefilter_on set to false
0: STDOUT: +lightcache_prefilter_samples set to 10
0: STDOUT: +lightcache_dontDelete set to true
0: STDOUT: +lightcache_autoSave set to false
0: STDOUT: --Failed to set lightcache_autoSaveFileName to T:\3D\XSI_DB\ESB\Scenes\sh13_03.vrlmap
0: STDOUT: +lightcache_switchToSavedMap set to true
0: STDOUT: +lightcache_useForGlossyRays set to false
0: STDOUT: +lightcache_numPasses set to 8
Notice that three things failed to be set here. Funny thing though is that this also appears in the good frame’s log. So maybe it’s possible that these failures increase the probability of the light cache getting stuck.
Some background: Deadline exports a lot of renderer specific options during submission so that you can access them from the Deadline Monitor after the job is submitted. If you right-click on your job in the Monitor, select Modify Properties, and then select the 3dsmax tab, you would notice a bunch of VRay options that you can normally access from within 3dsmax.
There is an option when submitting from 3dsmax to not export render options for specific renderers. Before you go to submit your next job, first select the Options tab in the submitter and uncheck the “ChaosGroup V-Ray” box. If this helps the situation, we can go back and try to not set specific options (for example, the three options that failed above) to see if we can narrow down the problem. Let us know how it goes!
Cheers,
Cheers for that, I’ll try it on the next submission. Also as another discovery, the hair and fur modifier causes on of those seh exceptions even if it’s not being used in the scene - the option to use all lights might be the culprit. Again this might be more a vray rather than deadline thing but it’s another to add to the list.
Cheers!
John
Similar problems again to be honest - I actually defaulted back to using backburner which ran the job through fine - again it’s not an option since we’ve a mixed application studio.
Darn! If you can post a simple test scene that reproduces this problem, we’ll do what we can to figure this one out!
Cheers,
Tried running some tests here on a couple of machines with a simple scene we created, but we never had problems with the light cache. I tried replicating some of the options based on the logs you posted earlier and that didn’t help. I know this problem is random, but my two machines finished four 100 frame jobs without a single hiccup. It could just be the settings in the scene I’m using, so if you could post a test scene, that would be a great!
For what it’s worth, I ran my tests with Vray 1.5 SP2 using 3dsmax 2008 64 bit. Here are the specs of the machines I was rendering with:
Machine 1:
XP 64 SP2
Dual-Core AMD Opteron Processor 1218
2.6 GHz
4 GB RAM
Machine 2:
XP 64 SP2
Dual-Core AMD Operton Processor 246
2.01 GHz
2 GB RAM
I should also note I was rendering with our internal beta copy of Deadline 3.0. Not much has changed regarding the internal workings of our 3dsmax plugin, but you never know. If Deadline 3.0 can whip through your test scene a few times without any issues, maybe this issue has already been resolved…
Cheers,
Doing a really simple render for a retail ad at the mo with no gi at all and my own machine is doing it consistently - it’s also the license server since I’m the only max op and the vray license server has a habit of dying on me - I wonder is this a cause?
I wouldn’t expect that the license server dying would cause vray to calculate the light cache indefinitely…
As a last resort, you could try using the 3ds Command plugin instead of the 3dsmax one:
franticfilms.com/software/su … ommand.php
This is a more generic plugin that works with 3dsmax and VIZ, and uses the max command line renderer to do the rendering. It doesn’t have as many options as the standard 3dsmax plugin, but it might provide a workaround until we figure this problem out. I should point out that we use Vray here at Frantic a lot through Deadline, and as far as I know, this problem has never come up.
Maybe it’s a machine configuration issue? Maybe a reinstall of 3dsmax/Vray might help? Just throwing out ideas here…
Cheers,
Hey everyone,
Just an update that this problem should be resolved in the latest/next release of VRay. We were having this problem here at Frantic as well and Vlado was kind enough to implement a fix for us (I think the problem had something to do with how vray communicated with the vray license server).
Cheers,