See the post my Bobo at the bottom of the page for the solution.
There seems to be a bit of a memory leak with the submittodeadline in 3dsmax, the problem get exponetially worse with the more Tiles you submit and with a heavier scene.
On a fairly light scene, I managed to submit 35x35 Tiles, Page File usage went up to 32gb on a 12gb RAM machine.
On a heavier scene with CAD data, Trying to sbmit a scene with 20x20 tiles, and up to 30gb Page File Usage, also with 12gb of RAM, and my computer is almost completely locked up. Infact, I had to force-quit 3dsmax as it had just hit it’s limit of PG file usage and the maxscript listener had listed a load of errors just before max died.
10x10 seems to be the highest number of tiles we can render without massive slow down and max becoming unresponsive, I’ve also noticed that 5x5 is orders of magnatudes faster than 10x10, far faster than 4x.
I’ve been running a garbage collection occasionally after lots of rendering and it’s been knocking a few GB off the page file usage.
Why do Tile jobs need so much more time and memory to submit… The mathmatics behind the tiles isn’t complicated at all, shouldn’t take much longer than a standard animation range sequence?
I looked through the code and I don’t see anything obvious either.
I would insert a ‘gc light:true’ at the end of the nested loops inside the spawnTileJobs() function in SubmitMaxToDeadline_Functions.ms and hope for the best.
Since I don’t have the heavy scene you are using, would you be interested in making the change in your file (located in the Repository\Submission\3dsmax\ folder) and testing it for us?
There are two cases handled in that function - single frame tile job (submits one job where each task is a tile) and animation tile jobs (submits one job per tile with tasks as frames). You would have to add the ‘gc light:true’ to both loops.
)--end if custom tiles
gc light:true --> new line!
)--end x loop
)--end y loop
and then
[code] currentFrame += 1
gc light:true → new line
)
)
In my version of the file, these are around lines 5040 and 5124 respectively.
If this does not help, you can change the light:true to light:false and try again.
I will play with this myself, but I’d rather have someone who has experienced the problem confirm these changes actually help.
So things have definetly improved, but not quite as much as I was hoping. I’ve attached an example of what was happening with a 10x10 submission. You can see the RAM build and drop when I ran a GC() afterwards. This scene wasn’t particularly heavy, all geometry was Xreffed and was quite simple, rendered at 6k.
I then added the gc code and ran a 20x20 tile render and the ram usage stayed around the same amount so didn’t crash out, but it took a very very long time to submit, maybe 3-4 minutes (Everything is being submitted as 1 job)
Below is the section of code where it’s slow… so my thinking is that slowness and memory leak is being caused by accessing the render element information. We have a lot of render elements, Multimaterial masks etc in this render…
It might be more efficient to collect the information about render elements before running the TileX and TileY loops. As really we only need this list once and can populate the tile information pretty simply.
Dave
[code]for y = 1 to SMTDSettings.TilesInY do
(
for x = 1 to SMTDSettings.TilesInX do
(
if not SMTDSettings.UseCustomTiles or findItem SMTDSettings.CustomTiles (Point2 x y) > 0 do
(
local tempRight = DeltaXx
if x == SMTDSettings.TilesInX then
(
tempRight = RenderWidth
if SMTDSettings.TileBlowupMode then
tempRight = (floor (1.0(RenderWidth/SMTDSettings.TilesInX))) as integer
)
local tempBottom = DeltaY*y
if y == SMTDSettings.TilesInY then
(
tempBottom = RenderHeight
if SMTDSettings.TileBlowupMode then
tempBottom = (floor (1.0*(RenderHeight/SMTDSettings.TilesInY))) as integer
)
local currentIndex = currentTile
if SMTDSettings.UseCustomTiles do
currentIndex = currentFrame
SMTDSettings.SingleTileJobLeft = SMTDSettings.SingleTileJobLeft + "RegionLeft" + (currentIndex as string) + "=" + (((DeltaX*(x-1)) as integer) as string) + "\n"
SMTDSettings.SingleTileJobTop = SMTDSettings.SingleTileJobTop + "RegionTop" + (currentIndex as string) + "=" + (((DeltaY*(y-1)) as integer) as string) + "\n"
SMTDSettings.SingleTileJobRight = SMTDSettings.SingleTileJobRight + "RegionRight" + (currentIndex as string) + "=" + ((tempRight as integer) as string) + "\n"
SMTDSettings.SingleTileJobBottom = SMTDSettings.SingleTileJobBottom + "RegionBottom" + (currentIndex as string) + "=" + ((tempBottom as integer) as string) + "\n"
local tileString = "_tile_" + x as string + "x" + y as string + "_" + SMTDSettings.TilesInX as string + "x" + SMTDSettings.TilesInY as string + "_"
local regionFileName = theOutPath + theBaseName + tileString + theOutType
SMTDSettings.SingleTileJobFilename = SMTDSettings.SingleTileJobFilename + "RegionFilename" + (currentIndex as string) + "=" + regionFileName + "\n"
-- If any render elements are specified, include them
local reManager = maxOps.GetCurRenderElementMgr()
local outputFilenameIndex = 0
if reManager.GetElementsActive() then
(
local reCount = reManager.NumRenderElements()
for i = 0 to reCount - 1 do
(
reObject = reManager.GetRenderElement i
if reObject.enabled then
(
reFilename = reManager.GetRenderElementFilename i
if reFilename != undefined and reFilename != "" then
(
local regionReFileName = (getFilenamePath reFilename) + (getFilenameFile reFilename) + tileString + (getFilenameType reFilename)
SMTDSettings.SingleTileJobReFilename = SMTDSettings.SingleTileJobReFilename + "RegionReFilename" + (currentIndex as string) + "_" + (outputFilenameIndex as string) + "=" + regionReFileName + "\n"
outputFilenameIndex = outputFilenameIndex + 1
)
)
)
)
currentTile +=1
if currentFrameList == "" then
currentFrameList = (currentFrame as string)
else
currentFrameList = currentFrameList + "," + (currentFrame as string)
)
currentFrame += 1
gc light:true --> new line
)
That Render Element code only renames all render elements according to the new tile specification. I don’t see how that would take a long time, regardless of how many render elements you have. We could put some timers in the code and see where the time is spent.
Did you try gc light:false instead of true?
When you call GC() yourself, you are technically calling GC LIGHT:FALSE. So it might be a good idea to put it in there.
I would expect it to slow down things even more, but if memory stays low, who knows what would happen.
If you tried it already, then ignore.
I will start doing my own tests here and see if I can reproduce this behavior and debug it.
Ok, I took a very SIMPLE scene first, set it to single frame output, submitted 20x20 tiles at 6400x4800 total resolution and the submission took around 10 seconds. So nothing new here.
Then I created 10 Render Elements and … run out of memory.
So you are right, looks like creating 400 instances of the Render Elements Manager is a bad idea.
Will see if I can fix this quickly…
P.S. I am kind of happy that I did not write that portion of the code
Correction.
Turns out it is not the render elements manager, but the huge string being created by string = string + “bla” which is a big no-no and is the cause for the memory leak.
Each time you add to a string, it copies the original string in memory, and it gets cleaned up only on gc light:true (but that’s slow).
The right way to do it is to format to a stringStream, as explained in the MAXScript Help.
That string (SMTDSettings.SingleTileJobReFilename) grows exponentially as we collect 10 render element file names 400 times.
Here is the fixed portion of the code:
[code] SMTDSettings.SingleTileJobLeft = “”
SMTDSettings.SingleTileJobRight = “”
SMTDSettings.SingleTileJobTop = “”
SMTDSettings.SingleTileJobBottom = “”
SMTDSettings.SingleTileJobFilename = “”
SMTDSettings.SingleTileJobReFilename = “”
local SingleTileJobReFilenameSS = “” as stringStream
local reManager = maxOps.GetCurRenderElementMgr()
local reCount = reManager.NumRenderElements()
local reObjects = for i = 0 to reCount-1 collect
(
local reObject = reManager.GetRenderElement i
#(reObject, reObject.enabled, reManager.GetRenderElementFilename i)
)
local currentFrame = 0
local currentFrameList = ""
for y = 1 to SMTDSettings.TilesInY do
(
for x = 1 to SMTDSettings.TilesInX do
(
if not SMTDSettings.UseCustomTiles or findItem SMTDSettings.CustomTiles (Point2 x y) > 0 do
(
local tempRight = DeltaX*x
if x == SMTDSettings.TilesInX then
(
tempRight = RenderWidth
if SMTDSettings.TileBlowupMode then
tempRight = (floor (1.0*(RenderWidth/SMTDSettings.TilesInX))) as integer
)
local tempBottom = DeltaY*y
if y == SMTDSettings.TilesInY then
(
tempBottom = RenderHeight
if SMTDSettings.TileBlowupMode then
tempBottom = (floor (1.0*(RenderHeight/SMTDSettings.TilesInY))) as integer
)
local currentIndex = currentTile
if SMTDSettings.UseCustomTiles do
currentIndex = currentFrame
SMTDSettings.SingleTileJobLeft = SMTDSettings.SingleTileJobLeft + "RegionLeft" + (currentIndex as string) + "=" + (((DeltaX*(x-1)) as integer) as string) + "\n"
SMTDSettings.SingleTileJobTop = SMTDSettings.SingleTileJobTop + "RegionTop" + (currentIndex as string) + "=" + (((DeltaY*(y-1)) as integer) as string) + "\n"
SMTDSettings.SingleTileJobRight = SMTDSettings.SingleTileJobRight + "RegionRight" + (currentIndex as string) + "=" + ((tempRight as integer) as string) + "\n"
SMTDSettings.SingleTileJobBottom = SMTDSettings.SingleTileJobBottom + "RegionBottom" + (currentIndex as string) + "=" + ((tempBottom as integer) as string) + "\n"
local tileString = "_tile_" + x as string + "x" + y as string + "_" + SMTDSettings.TilesInX as string + "x" + SMTDSettings.TilesInY as string + "_"
local regionFileName = theOutPath + theBaseName + tileString + theOutType
SMTDSettings.SingleTileJobFilename = SMTDSettings.SingleTileJobFilename + "RegionFilename" + (currentIndex as string) + "=" + regionFileName + "\n"
-- If any render elements are specified, include them
local outputFilenameIndex = 0
if reManager.GetElementsActive() then
(
for anElement in reObjects where anElement[2] == true do
(
local reFilename = anElement[3]
if reFilename != undefined and reFilename != "" do
(
local regionReFileName = (getFilenamePath reFilename) + (getFilenameFile reFilename) + tileString + (getFilenameType reFilename)
format "RegionReFilename%_%=%\n" currentIndex outputFilenameIndex regionReFileName to:SingleTileJobReFilenameSS
outputFilenameIndex += 1
)
)
)
currentTile +=1
if currentFrameList == "" then
currentFrameList = (currentFrame as string)
else
currentFrameList = currentFrameList + "," + (currentFrame as string)
)
currentFrame += 1
)
)
SMTDSettings.SingleTileJobReFilename = SingleTileJobReFilenameSS as string[/code]
Perfect! 20x20 Tiles with 28 render elements, submitted in about 15seconds. I recommend anyone who uses Tile Rendering to impleament this code immediately, massive improvement!
