Bizarre File type issue with Tiles Rendering... Update 03/09

Dave_Wortley · May 28, 2012, 6:17pm

So I’m completely confused as to how this happened but…

Originally the file was set up as a tif render, I changed this before I sent it to deadline, but can’t remember if it was before or after adding my render elements.

Sent a file to render that was a TGA file, and had render elements also TGA files…

Sent the job to deadline as tile render, the job completed and the tiles stitched as saved, all as TGA.

I then re-queued some tile tasks, and the files were saved as a .tga file but internally they were tif format.

Confused…

Bobo · May 28, 2012, 6:52pm

So your tile jobs rendered TGAs the first time, then you requeued some of them and they rendered as TIF content with TGA extension instead WITHOUT you changing anything?

The first thing to check is whether the TGAs rendered in the first attempt were indeed TGAs internally.
Also, did the Tile Assembler fail after you re-rendered the tiles in the second attempt?

Dave_Wortley · May 28, 2012, 6:59pm

Hmm yeah I guess that could have happened, not sure if the tile was a real tga or a .tga saved as tif format, which then when stitched got saved as a .tga in tga format. I’ll test it…

I’ve written a function which strips the render elements and re-builds them before sending to deadline, I’m going to build it into our deadline so this issue should never happen, I’ll post up my code when I’m done, might be worth including in the next Deadline as it should get rid of this persistent issue we’re having, and other have had.

Dave_Wortley · May 29, 2012, 10:42am

EDIT: There was a bug in the code below that meant the render element filenames were getting lost, this is now fixed as of, 14th August 2012.

So I managed to repoduce the bug under controlled circumstances, and as promised here is the code to put a work-around in. It stores all the values and properties of the Render Elements, removes them, adds them back in again, which seems to fix the file format issue.

To reproduce the bug:
Create new scene, add some objects
Set Renderoutput to test_01_.tif
Add some render elements.
Send to Deadline as tiles render but don’t delete the tiles.

Check the stitched Render elements, will be a valid TIF file, check the Render element tiles files, will be valid TIFfile.

Now change the Render Output to test_01_.tga
Send to Deadline again as Tile render and don’t delete the tiles.

Check the stitched render element, will be a valid TGAfile, check the Render Element Tiles files and they will be invalid TGA file, open up in IrfanView and it’ll tell you this file is a TIF file named incorrectly and will prompt you to rename as .tif

To add a fix, follow these steps, let me know if there’s any bugs please.

In SubmitMaxToDeadline.ms

after
checkbox chk_RenderElementsPermanentChanges "Permanent RE Path Changes " pos:[5,45] align:#left

Add

checkbox chk_RebuildRenderElements "Rebuild Render Elements   " pos:[225,45] align:#left

After
chk_RenderElementsPermanentChanges.state = SMTDSettings.RenderElementsPermanentChanges

Add

chk_RebuildRenderElements.state = SMTDSettings.RebuildRenderElements

After
SMTD_MainRollout.Ui_report ("+Rendering: RenderElements Permanent Changes set to [" + state as string + “]”)
)

Add

on chk_RebuildRenderElements changed state do ( setIniSetting SMTDPaths.InIFile "RenderingOptions" "RebuildRenderElements" ((SMTDSettings.RebuildRenderElements= state) as string) SMTD_MainRollout.Ui_report ("+Rendering: ReBuild RenderElements Changes set to [" + state as string + "]") )

In SubmitMaxToDeadline_Functions.ms

After
RenderElementsPermanentChanges = true,

Add:

RebuildRenderElements = true,

After:
SMTDSettings.RenderElementsPermanentChanges = SMTDFunctions.getInI “RenderingOptions” “RenderElementsPermanentChanges” SMTDSettings.RenderElementsPermanentChanges

Add:

SMTDSettings.RebuildRenderElements = SMTDFunctions.getInI "RenderingOptions" "RebuildRenderElements" SMTDSettings.RebuildRenderElements

After:
fn RenderElementsRestorePaths =
(
…
),

Add:

[code]fn RebuildRenderElements =
(
local reManager = maxOps.GetCurRenderElementMgr()
local AR_Elements = #()
local AR_ElementProps = #()
local AR_ElementFilenames = #()
for i = 1 to reManager.numrenderelements() do
(
local theelement = reManager.getrenderelement (i - 1)
append AR_Elements (classof theelement)
local a = #()
for prop in (getPropNames theelement) do
try(append a (getproperty theelement prop))catch(append a “”)
append AR_ElementProps a

			append AR_ElementFilenames (reManager.GetRenderElementFilename (i - 1))
		)
		reManager.RemoveAllRenderElements()
		for i = 1 to AR_Elements.count do
		(
			reManager.addrenderelement (execute((AR_Elements[i] as string) + "()"))
			local theelement = reManager.getrenderelement (i - 1)
			
			for j = 1 to (getPropNames theelement).count do
				try(setproperty theelement (getPropNames theelement)[j] AR_elementProps[i][j])catch()
			
			reManager.SetRenderElementFilename (i - 1) (getFilenamePath AR_ElementFilenames[i] + getFilenameFile AR_ElementFilenames[i] + getFilenameType RendOutputFilename)
		)
	),[/code]

After:
if SMTDSettings.OverrideLowPriorityThread do
(
…
)

Add:

[code]if SMTDSettings.RebuildRenderElements do
(
SMTD_MainRollout.Ui_report “>Rebuilding Render Elements”
SMTDFunctions.RebuildRenderElements()
SMTD_MainRollout.Ui_report “+Rebuilt Render Elements”

		)[/code]

Not sure if this next bit is necessary or not…

In SubmitMaxToDeadline_Defaults.ini

Under [RenderingOptions]
Add:
RebuildRenderElements=true

rrussell · May 30, 2012, 1:12pm

Hey Dave,

Thanks for this! Everything looks in order here, so we’ll include this in the 5.2 release.

Cheers,

Ryan

Dave_Wortley · May 30, 2012, 3:38pm

Cool, Cheers Russell.

The only issue I could see with this is if any properties are animated… can’t think of any reason to animate a render element property, appart from possibly zdepth depth values?

rrussell · May 30, 2012, 4:15pm

Our plan is to just keep the option disabled by default, but allow it to be sticky. That way, if there are any unexpected problems, it would only impact those that chose to enable the feature.

Cheers,

Ryan

MikeOwen · July 20, 2012, 1:55pm

FYI.
This new function doesn’t handle the situation when some of your RE’s are being output to a different image file format on purpose. Unfortunately, it destroys and set’s all RE’s to be identical to the main render output, which may or may not be the intended result that a user is after. Thankfully, this function is disabled by default.
I feel the real crux of this bug is really in the poor initialisation of the particular bitmap image plugins being used within Max2011. I’d be interested to know if this is yet again, just another bug in Max2011. Has anyone had this error in Max2012 or 2013? (In closed circles, Autodesk have now admitted that Max2011 was one of the most bug ridden releases ever!)
Mike

Dave_Wortley · July 20, 2012, 2:04pm

The issue has been in max for years, if you wanted to save a file with different formats for the render elements it would never work, it would always end up being your main render format.

I have thought about re-working the theory behind this hack to force max to change the renderoutput type, add the render element then change the renderoutput type back. So annoying that you can’t access the true render element filetype but this has unfortunately been one of the issues with Max for a while. The render element tab is it’s own dark matter.

MikeOwen · July 20, 2012, 5:21pm

Interesting. So your saying it’s impossible to save RE’s to different file types than that set in the main render output?

Dave_Wortley · July 22, 2012, 11:18pm

Maybe it used to work years ago or maybe it’s something unique to VRay, but it’s never worked for as long as I can remember, would need to double-check with latest Max version.

Dave_Wortley · July 23, 2012, 10:23am

I wouldn’t say impossible Mike but this is 3dsmax we’re dealing with here…

I’m running some test this morning… Vray 2.2 and Max 2011, Deadline 5.1

Test 1:

Set my main render to an Test_01.EXR file, (not adding in render elements)
Added a VRayGI element
Added a VRayRawLighting element
changed the VRayRawLighting element to .tga
Rendered Locally…

Result: Test_01.exr, Test_01_VRayRawGlobalIllumination.exr and Test_01_VRayRawLighting.tga
All files the correct formats for their filenames.

Test 2:
Changed the main Render Output to Test_02.exr
Rendered Locally…

Result: Test_02.exr, Test_02_VRayRawGlobalIllumination.exr and Test_01_VRayRawLighting.tga
All files the correct format… but the tga file didn’t get its name updated. Looking in the Render element list it hadn’t been updated there either.

Test 3:
Deleted all render elements,
Set main output to Test_03_.exr
Added the elements as before changing the RawLighting to a tga file
Submitted to Deadline without the ‘Rebuild Elements’ tickbox (my bit of code)…

Result: Test_03_000.exr, Test_03_VRayRawGlobalIllumination_0000.exr and Test_03_VRayRawLighting_0000.exr
The first two EXRs are correctly formatted but the RawLighting is infact a tga file with a .exr file extension.
Looking at the render element list the RawLighting now has an .EXR extension. (Deadline is set to update render element paths and filenames)

Test 3b:
Deleted the Test_03 renders and rendered locally.

Result: All files saved as EXR files and all in EXR format.

Test 4:
Deleted the Render elements, set the main output to Test_04.exr, added the render elements as before and renamed the RawLighting to .tga
Sent to deadline with all Repathing options turned off, (Update Render Elements’ Paths, Also Update RE’s Filenames).

Result: Test_04_0000.exr as an EXR file, Test_04_VRayRawGI_0000.exr as an EXR and RawLighting_0000.tga as a TGA file! Success!

Test 4b:
Changed the Main Render output to Test_04b_ and the RawGI element changed names to Test_04b_VRayRawGlobalIllumination, but the RawLighting stayed as Test_04_VRayRawLighting.

So, the problems come from Max not updating all render elements and when Deadline updates the render element paths it doesn’t respect the Render Element’s original file-type.

Looking at the Deadline code which does the RenderElementsUpdatePaths() function the reManager.SetRenderElementFilename is getting the filetype from the main output not the render element in question.

Test 5:

I changed this section of code to getFileNameType reFilename instead of getFileNameType OutputFilename

if SMTDSettings.RenderElementsUpdateFilenames then ( reManager.SetRenderElementFilename i (newPath +"\\" + getFileNameFile outputFilename + "_" + ((classof theElement) as string) + getFileNameType reFilename) ) else ( if reFilename != undefined and reFilename != "" then reManager.SetRenderElementFilename i (newPath +"\\" + getFileNameFile reFilename + getFileNameType reFilename) else reManager.SetRenderElementFilename i (newPath +"\\" + theElementFolder + getFileNameType reFilename) )

Set the main output to Test_05_.exr, added the RawGI element, added the RawLighting element and changed to .tga,
I then sent it off to Deadline with the “Update Render Elements’ Paths”, “Also Update RE’s Filenames” turned on.

Result: Test_05_0000.exr saved as EXR, RawGI saved as exr, RawLighting_0000.tga saved as a TGA!

Test 5b:
Just changed the main output to Test_05b_.exr kicked it off through Deadline,

Result: Test_05b_000.exr, Test_05b_VrayRawGlobalIllumination_000.exr and Test_05b_VRayRawLighting_0000.tga all in the correct formats.

Test 6:
Kept the render elements as they were and changed the render output to Test_06_.tif, (thus recreating the initial conditions that this thread was based upon, if you change the main render output type it messes up your render elements.)

Result: Test_06_0000.tif in TIF format, Test_06_VRayGlobalIllumination_0000.exr as an EXR file and Test_06_VRayRawLighting_0000.tga as a tga file.

So the fix above is a much more simple fix than the one I posted earlier in the thread but if you set up all your render elements doing quick render tests with JPEGs for some reason then switch to EXR for your main renders the elements will still be JPEG, but at least now they will save out with a .jpg extension and not be a saved with an incorrect filename.

Test 6b:
Enabled the “Rebuild Render Elements” option in Deadline which I added earlier in the thread…
Submitted to Deadline.

Result:
Main RGB and render elements all save as .tif files in TIF format.

So in conclusion, the fix above should be put in straight away in my opinion, but then it’s probably sensible to put a Sanity check warning message in to warn the artist that the render element formats don’t match their main render format. If you don’t want to render mixed format render elements and want to make sure that the render elements are always the same format as the RGB then keep the ‘Rebuild Render Elements’ option ticked (available in Deadline 5.2 if you haven’t already added it.)

MikeOwen · July 23, 2012, 1:16pm

Good effort on the testing.
I’ll make Bobo aware of your findings.
To keep the SMTD as simple as possible, it may be better to apply your core fix, but remove your RebuildElements function, but instead, insert it as a general RC sanity checker fixer function, based on a sanity check warning message.
The key decision is to decide on what the default behaviour of SMTD should be.

Dave_Wortley · July 23, 2012, 1:48pm

You’d still want access to the Rebuild Render Elements option as if you change the file format of your main renderer and you’ve got loads of MultiMatte’s setup for instance, you don’t want to have to reset them all up again, or go through one-by-one changing the format.

Dave_Wortley · July 23, 2012, 1:49pm

Maybe ‘Rebuild Render Elements’ needs a more descriptive caption, “Match RE format to render format” or something like that?

Bobo · July 24, 2012, 5:16pm

Just a note that the last line may NOT be using ‘getFileNameType reFilename’ because the IF statement has already discovered that reFilename is either undefined or “”. It will have to use the type of the OutputFilename…

I am adding this to the code internally right now.

Dave_Wortley · July 25, 2012, 9:16am

Ok cheers for that Bobo, updated my end as well.

Dave_Wortley · August 14, 2012, 11:09am

Ok so the Saga continues… getting this bug occasionally (inconsistently) .

Took me a little while to track down why it happens and what causes it as it was so inconsistent, but here’s the gist.

Create a test scene, set a render outputfilename, add some render elements.
Close the Render Scene Dialog

Close Max.

Open max, open the file.

Press the submit to deadline button.

Error:

MAXScript Rollout Handler Exception: – Unable to convert: undefined to type: FileName <<

The bit it errors on is, “getFileNameType reFilename”, the bit we changed to fix the other bug.

So tracing back… digging a bit out of the code, I try this…

re = maxOps.GetCurRenderElementMgr() reFilename = re.GetRenderElementFilename 1

returns undefined.

So I think, maybe the Render Elements have lost their paths… open Render Scene Dialog, nope paths still there…

Run this code again…

re = maxOps.GetCurRenderElementMgr() reFilename = re.GetRenderElementFilename 1

and this time it returns the filename of the render element.

And if you try submitting to Deadline again it works!

So it would seem that you cannot access the filename of a render element without first having initialized the Render Scene Dialog, unless I’m missing something?

Bug appears on 3dsmax 2011.

The easiest fix I can see would be on submission to check whether the path of the render element was undefined then open the RenderSceneDialog to initialise if it’s undefined.

Dave_Wortley · August 14, 2012, 11:45am

And just to confuse matters…

I just restarted max, opened the same scene file…

Didn’t press the Submit_to_Deadline button…

Ran this code again…

re = maxOps.GetCurRenderElementMgr() reFilename = re.GetRenderElementFilename 1

and it returned the name as expected…

Press the SubmitMaxtoDeadline button and it errors on the same line as before…

I try that bit of code again… and it now returns undefined.

So it’s something in the submission process that’s breaking this…

Can someone please test this on a vanilla Deadline please, I’ve made a few changes to mine that could potentially have caused this, need to find out if it’s an issue of my own creation or Deadline’s.

Dave_Wortley · August 14, 2012, 11:48am

There’s something in the maxscript help I spotted about using SetSilentMode true to avoid errors with getting render element filenames? Doesn’t say anything more than that.