Hey,
We’ve begun using EXRs in earnest recently and have had a recurring problem. When saving as one type of EXR we get random occurrences where machines will begin to save as other types of EXR. ie. we’ll save as 16bit float and get some frames as 32bit integers and some others as 16bit half floats. This only happens during net rendering. We use 3ds Max 2010 SP1 with Connection Extension, RPManager 4.537 and Deadline 4 on XP64.
It doesn’t seem specific to machines, so I don’t think it’s a configuration issue. A machine that errors one day will be fine the next. Also, it’s across multiple jobs. So different scenes, different assets etc. So in my mind it has to be an issue with either Deadline or 3ds Max or RPManager
Another issue we’ve had might give some ideas, or it may be totally unrelated.
We often get wildly varying file sizes in EXR sequences. This is a result of extra passes randomly being added to some files.
This could be a Max issue rather than Deadline … but it’s really vexing us and we can’t find anyone with any mention of similar problems anywhere on the web, so if anyone has seen something similar we’d really appreciate some help !.
Hey Ben,
Have you been able to narrow down all the dodgy frames to certain machines which may have NOT received the “connection extension” and hence have the ‘new’ Cebas EXR plugin? Also, silly question, but when you go into the EXR saving dialog on the new Cebas EXR extension, you can individually set the bit type for each pass…have you done this? Finally, I would question RPM probably not fully supporting the new EXR connection extension, so to test I would remove RPM from the equation by submitting jobs without it for the time being…(BTW - we haven’t seen this issue either…)
Mike
We’ve tried it with and without RP Manager … with no effect. It’s not limited to any specific machine … it’s as though the connection extension’s dialog settings are local to a machine on some (random) occasions. All machines are on exactly the same spec of 2010 with the connection extension installed.
We’re really stumped, and are a bit lost as to why we’re the only studio having these issues across so many projects. I’m sure Autodesk will say move to 2011, but that’s always a lot of pain … so we tend to skip a version of max to limit the amount of new broken features .
You mention the Cebas plugin, is that similar to the old Splutterfish EXR saver that we had before the connection extension ?. We’ve not seen that.
Which renderer are you using (mental ray, vray, etc)? One thing that’s worth trying (if you haven’t done so already) is to enable the “Restart Renderer Between Frames” option when submitting the job to Deadline. When using vray, this option has been known to fix some strange issues.
We’re using vray … we do the ‘restart renderer between frames’ as a matter of course to fix the elements saving issues. Saving EXR’s via the vray frame buffer fixes things … no doubt as the Connection Extension dialog box isn’t used … however this isn’t a good fix for us as this option will always embed elements into the EXR whether you want to or not (we don’t).
We get this problem too, with and without RPM. max2010sp1. Always with deadline but I haven’t tried backburner. It doesn’t seem to be renderer specific. I think I have an old thread about this from a year ago. My current theory is some machines are workstations that have different defaults just from people using max locally. Then sometimes the render job isn’t allowed to set it’s settings so it uses the machine’s defaults. And since you can’t tell a failure that has the same settings from a failure that has different settings, it’s hard to deduce what triggers it. But I noticed I think I only see the issue on workstations or workstation retired to the farm.
Interesting … we switched back to the Splutterfish EXR saver and seem to still be getting the same issues. It’s very confusing that it’d happen even with this EXR plugin.
Great to know we’re not alone though. It’s a maddening one, as you say it’s almost impossible to predict when and on which machine it’ll happen.
Hi guy’s, I thought I’d just report that I’ve got the same problem. I’m not aware of the bit-depth of the images changing, but I definitely see channels dropping in and out of frames when rendering via deadline. I’ve run quite a few tests and in the end I’m sure it’s deadline causing the problem.
I came to this conclusion by with the following results (All renders are V-Ray)…
Definitely try enabling the option to restart the renderer between frames. This tends to fix a bunch of issues with VRay, and starting in the next Deadline release, we’re actually going to be enabling this setting by default when vray is the selected renderer. It will save a lot of hassle going forward.
Further testing after switching out the ‘new’ connection extension EXR saver (that I think was originally a Cebas plugin) and putting back the old Splutterfish EXR saver that I think came with Brazil has fixed this issue for us. It has to be something to do with Deadline and the Cebas plugin. My reasoning being the amount of people out there that are using Max along with EXRs must be vast … but the only other people on the internet with this issue are all here on the Deadline forum.
Ben - the Cebas plugin I made reference to is basically the NEW Autodesk packaged OpenEXR plugin.(It was developed by Cebas for Uncharted Tet for the film, 2012 and then Autodesk purchased it).
Here’s a few “IDEAS…” to try as I’m not yet convinced its actually Deadline that is causing the issue:
Testing against the latest version of Deadline v4.1, Max2010+connection extension+all recent hotfixes, etc
Idea 1. Compare the modified size/date of this file: “C:\3dsMax_INSTALL\plugcfg\fopenexr.cfg”. Here’s some background from the 3dsMax manual; “The OpenEXR plug-in stores configuration information in a binary CFG file named fopenexr.cfg. The file is generated automatically the first time you edit the OpenEXR configuration settings, and is updated each time you modify settings when you load or save an EXR file.” I’m thinking this file might be getting corrupted in your pipeline. Check the version on your good local machine and then on your slave machine. Copy your copy of this file and overwrite the version on your slave machine and try re-rendering. The settings for the OpenEXR plugin should then be identical.
Idea 2. Check for bitmap plugin corruption; check and compare the OpenEXR *.bmi & *.cfg files between local (good machine) & bad (slave machine)
Idea 3. The bottom of your “3dsmax.ini” file gets an OpenEXR configuration dialog size & position value added to it when you use this dialog. Check it exists in your 3dsmax.ini file. Something like:
Although this in itself won’t cause/fix the problem, what I’m actually looking for is a potential corruption of your 3dsmax.ini file. Does this file look ok? Is it half empty? A known issue is when you have lots of HDRI based images in the Material Editor on your local machine or the slave machine and you carry out some basic manipulation of the “material bitmap”; it can cause the 3dsmax.ini file to become corrupted on the local machine. Simply deleting the file and letting it be re-created at next 3dsMax boot-up fixes this issue.
Idea 4. Write a MaxScript that can be executed at pre-frame or post-frame, etc of the deadline job which checks the OpenEXR settings/configuration before and after a rendered frame to see if it is getting changed from your submitted settings and prints this info to the deadline slave job-frame log. All the OpenEXR settings can be found in the MaxScript - Connection Extension Manual in Max2010 or is automatically included in the Main 2011 MaxScript manual. You might need to just temporarily run the deadline job in “workstation” mode to ensure the script fires ok, but with a bit of luck, you might get away with it, as you’re not directly referencing the active viewport…
Idea 5. Can you post an example basic Max2010 scene file which demonstrates the issue?
Thanks for the detailed response. I’m going to look again at this issue early next year as we have two enormous deadlines looming. We switched back to the Splutterfish EXR saver and it seemed to fix our issues for now.
We looked at scripting a fix for the Cebas OpenEXR plugin but didn’t manage to track down the Connection Extension Maxscript documentation.
Given the information you’ve posted I’m almost certain it’s down to this cfg settings associated with the Connection Extension, but won’t have time now this year to investigate further. I’ll post back next year once we’ve had a look at it.
I’m seeing this with backburner too, but I will soon be loading Deadline on the farm for use. I have tried everything Mike suggested, except a special script callback. My crosshairs are firmly on the cebas exr bmi file causing the issue. Any further developments from anyone?
Interesting. So if the issue occurs in Backburner consistently then we could potentially remove Deadline from the equation.
Does anyone have a simple Max Scene file they can share which always goes wrong at some point when network rendered they can upload?
Sounds like its time for some MAXScript log messages to determine what’s going on…
I think I have fixed the issue, but I haven’t got time to test fully so would appreciate you guys giving it a try!
This is only an issue in Max2010 with the Autodesk Connection Extension installed.
If you look in your render output file type extensions in your render scene dialog you should notice that there is 2 entries for *.exr files!
The original “exrio” bitmap io class and ALSO the new (Cebas) Connection Extension *FOpenEXR" bitmap io class, that gets installed as part of the Connection Extension.
So, to fix:
REMOVE this one file: “C:\maxroot\stdplugs\OpenEXR.bmi” from your copy of 3ds Max (make a backup!) then see if it works ok for you?
My money is on 3dsMax getting confused during network rendering where Max under slave mode initiates an instance of the “?EXR?” bitmap class which may or may not be the correct version and then may or may not render the correct type of EXR back to your file server. NOW, let’s throw a curve-ball into the equation. If you then set your renderer via Deadline to effectively “restart renderer between frames” (which is a really good idea for VRay or MR - as Ryan always tells everyone ), then its my best guess that you effectively cause the bitmap io class within Max to grab a new instance which if it goes by any logic (yeah right Autodesk cough cough!!), say by “alphabetical” order, then in the drop-down list you will see that the OLD “EXR” bitmap io plugin will be selected and used before the NEW “EXR” Cebas plugin has a chance to load and therefore you get the wrong render result in your EXR file OR alternatively 3dsMax is just getting confused by which EXR plugin to load during network rendering.
I’m going to submit an Autodesk support ticket for this potential bug and see what they say…but first, I need a coffee
Hi All,
No word from Autodesk as yet. I somehow think we won’t hear anything anyway
Ben C - Did you try my fix to see if that resolved the issue for you yet?
Regards,
Mike
.
