Maxwell Render Error - failed to create a temporary copy of explosion.vdb for delayed loading

Deadline
1

We’re getting an error that we can’t seem to track down a solution to:

2016-05-19 16:14:56: 0: STDOUT: failed to create a temporary copy of N:\renderfarm\Scene_01_Frame_0-1500\explosion.vdb for delayed loading (IoError: failed to open temporary file \s5cc.); will read directly from N:\renderfarm\Scene_01_Frame_0-1500\explosion.vdb instead

The Render works fine when you manually run it from Maxwell locally (from the network drive and from different systems), but trying it from Deadline we get the above error. Only seems to happen with .VDB dependencies.

I’m assuming the error is getting thrown at this point https://github.com/dreamworksanimation/openvdb/blob/master/openvdb/io/TempFile.cc#L111 of the OpenVDB library but I couldn’t trace it any deeper without knowing what the OPENVDB_TEMP_DIR etc is.

The Error log: (removed a bunch of searching for alternative paths for textures which aren’t part of the problem)

[code]

Error

failed to create a temporary copy of N:\renderfarm\Scene_01_Frame_0-1500\explosion.vdb for delayed loading (IoError: failed to open temporary file \s5cc.); will read directly from N:\renderfarm\Scene_01_Frame_0-1500\explosion.vdb instead
at Deadline.Plugins.ScriptPlugin.RenderTasks(String taskId, Int32 startFrame, Int32 endFrame, String& outMessage, AbortLevel& abortLevel)

=======================================================
Type

RenderPluginException

=======================================================
Stack Trace

at Deadline.Plugins.Plugin.RenderTask(String taskId, Int32 startFrame, Int32 endFrame)
at Deadline.Slaves.SlaveRenderThread.a(TaskLogWriter A_0)

=======================================================
Log

016-05-19 16:10:36: 0: INFO: Executable: “C:\Program Files\Next Limit\Maxwell 3\maxwell.exe”
2016-05-19 16:10:36: 0: INFO: Saving output for this job to N:\renderfarm\Rendered_Images\Scene_01_Left.00580.png
2016-05-19 16:10:36: 0: INFO: Saving MXI for this job to N:\renderfarm\Rendered_Images\Scene_01_Left.00580.0593.mxi
2016-05-19 16:10:36: 0: INFO: Argument: -mxs:“N:\renderfarm\Scene_01_Frame_0-1500\Final_Stereo_Renderfarm_Test_Left.0580.mxs” -nowait -hide -animation:593-593 -threads:32 -time:15.0 -sampling:20.0 -node -verbose:4 -output:“N:\renderfarm\Rendered_Images\Scene_01_Left.00580.png” -mxi:“N:\renderfarm\Rendered_Images\Scene_01_Left.00580.0593.mxi”
2016-05-19 16:10:36: 0: INFO: Startup Directory: “C:\Program Files\Next Limit\Maxwell 3”
2016-05-19 16:10:36: 0: STDOUT: maxwell:[INFO]: MAXWELL RENDER (M~R). Engine version 3.2.1.2
2016-05-19 16:10:36: 0: STDOUT: maxwell:[INFO]: © 2004-2015. Licensed to Next Limit Technologies
2016-05-19 16:10:36: 0: STDOUT: maxwell:[INFO]:
2016-05-19 16:10:36: 0: STDOUT: maxwell:[INFO]: C:\Program Files\Next Limit\Maxwell 3\maxwell.exe -mxs:N:\renderfarm\Scene_01_Frame_0-1500\Final_Stereo_Renderfarm_Test_Left.0580.mxs -nowait -hide -animation:593-593 -threads:32 -time:15.0 -sampling:20.0 -node -verbose:4 -output:N:\renderfarm\Rendered_Images\Scene_01_Left.00580.png -mxi:N:\renderfarm\Rendered_Images\Scene_01_Left.00580.0593.mxi
2016-05-19 16:10:36: 0: STDOUT: maxwell:[INFO]:
2016-05-19 16:10:36: 0: STDOUT: maxwell:[INFO]: SYSTEM INFO
2016-05-19 16:10:36: 0: STDOUT: maxwell:[INFO]: Num processors: ( logical: 32, physical: 32 )
2016-05-19 16:10:36: 0: STDOUT: maxwell:[INFO]: Maxwell Root folder :C:\Program Files\Next Limit\Maxwell 3
2016-05-19 16:10:36: 0: STDOUT: maxwell:[INFO]: Host ID: 1803732658f5
2016-05-19 16:10:36: 0: STDOUT: maxwell:[INFO]: Low Priority Enabled
2016-05-19 16:10:36: 0: STDOUT: maxwell:[INFO]:
2016-05-19 16:10:36: 0: STDOUT: maxwell:[INFO]:
2016-05-19 16:10:36: 0: STDOUT: maxwell:[INFO]: Found 5 procedural extensions in C:\Program Files\Next Limit\Maxwell 3\extensions
2016-05-19 16:10:36: 0: STDOUT: maxwell:[INFO]: MaxwellHair
2016-05-19 16:10:36: 0: STDOUT: maxwell:[INFO]: MaxwellParticles
2016-05-19 16:10:36: 0: STDOUT: maxwell:[INFO]: MaxwellVolumetric
2016-05-19 16:10:36: 0: STDOUT: maxwell:[INFO]: MGrassH
2016-05-19 16:10:36: 0: STDOUT: maxwell:[INFO]: MGrassP
2016-05-19 16:10:36: 0: STDOUT: maxwell:[INFO]: Found 6 geometry loader extensions in C:\Program Files\Next Limit\Maxwell 3\extensions
2016-05-19 16:10:36: 0: STDOUT: maxwell:[INFO]: AssetReference
2016-05-19 16:10:36: 0: STDOUT: maxwell:[INFO]: MaxwellMesher
2016-05-19 16:10:36: 0: STDOUT: maxwell:[INFO]: MaxwellSea
2016-05-19 16:10:36: 0: STDOUT: maxwell:[INFO]: MWObjectAlembic
2016-05-19 16:10:36: 0: STDOUT: maxwell:[INFO]: RFMeshes
2016-05-19 16:10:36: 0: STDOUT: maxwell:[INFO]: RWMeshes
2016-05-19 16:10:36: 0: STDOUT: maxwell:[INFO]: Found 4 geometry modifier extensions in C:\Program Files\Next Limit\Maxwell 3\extensions
2016-05-19 16:10:36: 0: STDOUT: maxwell:[INFO]: MaxwellCloner
2016-05-19 16:10:36: 0: STDOUT: maxwell:[INFO]: MaxwellGrass
2016-05-19 16:10:36: 0: STDOUT: maxwell:[INFO]: MaxwellScatter
2016-05-19 16:10:36: 0: STDOUT: maxwell:[INFO]: SubdivisionModifier
2016-05-19 16:10:36: 0: STDOUT: maxwell:[INFO]: Found 12 texture extensions in C:\Program Files\Next Limit\Maxwell 3\extensions
2016-05-19 16:10:36: 0: STDOUT: maxwell:[INFO]: Brick
2016-05-19 16:10:36: 0: STDOUT: maxwell:[INFO]: Checker
2016-05-19 16:10:36: 0: STDOUT: maxwell:[INFO]: Circle
2016-05-19 16:10:36: 0: STDOUT: maxwell:[INFO]: Gradient3
2016-05-19 16:10:36: 0: STDOUT: maxwell:[INFO]: Gradient
2016-05-19 16:10:36: 0: STDOUT: maxwell:[INFO]: Grid
2016-05-19 16:10:36: 0: STDOUT: maxwell:[INFO]: Marble
2016-05-19 16:10:36: 0: STDOUT: maxwell:[INFO]: Noise
2016-05-19 16:10:36: 0: STDOUT: maxwell:[INFO]: Voronoi
2016-05-19 16:10:36: 0: STDOUT: maxwell:[INFO]: MultiPointGradient
2016-05-19 16:10:36: 0: STDOUT: maxwell:[INFO]: TiledTexture
2016-05-19 16:10:36: 0: STDOUT: maxwell:[INFO]: WireframeTexture
2016-05-19 16:10:36: 0: STDOUT: maxwell:[INFO]: Found 7 material modifier extensions in C:\Program Files\Next Limit\Maxwell 3\extensions
2016-05-19 16:10:36: 0: STDOUT: maxwell:[INFO]: AGS
2016-05-19 16:10:36: 0: STDOUT: maxwell:[INFO]: Opaque
2016-05-19 16:10:36: 0: STDOUT: maxwell:[INFO]: Transparent
2016-05-19 16:10:36: 0: STDOUT: maxwell:[INFO]: Metal
2016-05-19 16:10:36: 0: STDOUT: maxwell:[INFO]: Translucent
2016-05-19 16:10:36: 0: STDOUT: maxwell:[INFO]: Car Paint
2016-05-19 16:10:36: 0: STDOUT: maxwell:[INFO]: Hair
2016-05-19 16:10:36: 0: STDOUT: maxwell:[INFO]: Found 2 camera lens extensions in C:\Program Files\Next Limit\Maxwell 3\extensions
2016-05-19 16:10:36: 0: STDOUT: maxwell:[INFO]: Lat-Long Stereo
2016-05-19 16:10:36: 0: STDOUT: maxwell:[INFO]: Fish Stereo
2016-05-19 16:10:36: 0: STDOUT: maxwell:[INFO]:
2016-05-19 16:10:36: 0: STDOUT: maxwell:[INFO]:
2016-05-19 16:10:36: 0: STDOUT: maxwell:[INFO]:
2016-05-19 16:10:36: 0: STDOUT: maxwell:[19/May/2016 16:10:36] [INFO]: Reading MXS:N:\renderfarm\Scene_01_Frame_0-1500\Final_Stereo_Renderfarm_Test_Left.0593.mxs
2016-05-19 16:12:30: 0: STDOUT: maxwell:[19/May/2016 16:12:30] [INFO]: MXS file read successfully
2016-05-19 16:12:30: 0: STDOUT: maxwell:[INFO]:
2016-05-19 16:12:30: 0: STDOUT: maxwell:[INFO]: CPU ID set randomly: 14032
2016-05-19 16:12:30: 0: STDOUT: maxwell:[WARNING]: Preference to force threads to automatic is enabled.
2016-05-19 16:12:30: 0: STDOUT: maxwell:[INFO]:
2016-05-19 16:12:30: 0: STDOUT: maxwell:[19/May/2016 16:12:30] [INFO]: Rendering animation. Loading frame:593.
2016-05-19 16:12:30: 0: STDOUT: maxwell:[INFO]:
2016-05-19 16:12:30: 0: STDOUT: maxwell:[19/May/2016 16:12:30] [INFO]: Reading MXS:N:\renderfarm\Scene_01_Frame_0-1500\Final_Stereo_Renderfarm_Test_Left.0593.mxs
2016-05-19 16:14:49: 0: STDOUT: maxwell:[INFO]: CPU ID set randomly: 2142
2016-05-19 16:14:49: 0: STDOUT: maxwell:[WARNING]: Preference to force threads to automatic is enabled.
2016-05-19 16:14:49: 0: STDOUT: maxwell:[INFO]: CPU ID used: 2142
2016-05-19 16:14:49: 0: STDOUT: maxwell:[INFO]:
2016-05-19 16:14:49: 0: STDOUT: maxwell:[19/May/2016 16:14:49] [INFO]: Checking scene dependencies…
2016-05-19 16:14:51: 0: STDOUT: maxwell:[19/May/2016 16:14:51] [INFO]: Searching in alternative paths: D:/Assessment_2/The_Storm/sourceimages/explosion.vdb
2016-05-19 16:14:51: 0: STDOUT: maxwell:[19/May/2016 16:14:51] [INFO]: File explosion.vdb found in folder: N:\renderfarm\Scene_01_Frame_0-1500\explosion.vdb
2016-05-19 16:14:51: 0: STDOUT: maxwell:[19/May/2016 16:14:51] [INFO]: Searching in alternative paths: D:/Assessment_2/The_Storm/sourceimages/smoke.vdb
2016-05-19 16:14:51: 0: STDOUT: maxwell:[19/May/2016 16:14:51] [INFO]: File smoke.vdb found in folder: N:\renderfarm\Scene_01_Frame_0-1500\smoke.vdb
2016-05-19 16:14:51: 0: STDOUT: maxwell:[19/May/2016 16:14:51] [INFO]: Searching in alternative paths: D:/Assessment_2/The_Storm/sourceimages/fire.vdb
2016-05-19 16:14:51: 0: STDOUT: maxwell:[19/May/2016 16:14:51] [INFO]: File fire.vdb found in folder: N:\renderfarm\Scene_01_Frame_0-1500\fire.vdb
2016-05-19 16:14:51: 0: STDOUT: maxwell:[19/May/2016 16:14:51] [INFO]: Dependency found: N:/renderfarm/Scene_01_Frame_0-1500/explosion.vdb
2016-05-19 16:14:51: 0: STDOUT: maxwell:[19/May/2016 16:14:51] [INFO]: Dependency found: N:/renderfarm/Scene_01_Frame_0-1500/smoke.vdb
2016-05-19 16:14:51: 0: STDOUT: maxwell:[19/May/2016 16:14:51] [INFO]: Dependency found: N:/renderfarm/Scene_01_Frame_0-1500/fire.vdb
2016-05-19 16:14:51: 0: STDOUT: maxwell:[19/May/2016 16:14:51] [INFO]: Searching in alternative paths: h]:
2016-05-19 16:14:51: 0: STDOUT: maxwell:[19/May/2016 16:14:51] [WARNING]: File not found in alternative paths: h]:
2016-05-19 16:14:51: 0: STDOUT: maxwell:[19/May/2016 16:14:51] [INFO]: Checking Data
2016-05-19 16:14:51: 0: STDOUT: maxwell:[19/May/2016 16:14:51] [INFO]: Loading Bitmaps & Preprocessing Data
2016-05-19 16:14:51: 0: STDOUT: maxwell:[19/May/2016 16:14:51] [INFO]: Loading geometry extensions.
2016-05-19 16:14:52: 0: STDOUT: maxwell:[19/May/2016 16:14:52] [INFO]: [Extension MaxwellGrass] Executing Mesh Modifier Extension :“Close_Landscape”
2016-05-19 16:14:52: 0: STDOUT: maxwell:[19/May/2016 16:14:52] [INFO]: [Extension MaxwellGrass] Modifying mesh…
2016-05-19 16:14:52: 0: STDOUT: maxwell:[19/May/2016 16:14:52] [INFO]: [Extension MaxwellGrass] Generating blades…
2016-05-19 16:14:54: 0: STDOUT: maxwell:[19/May/2016 16:14:54] [INFO]: [Extension MaxwellGrass] Exporting blades…
2016-05-19 16:14:54: 0: STDOUT: maxwell:[19/May/2016 16:14:54] [INFO]: [Extension MaxwellGrass] Generating 1700637 blades
2016-05-19 16:14:55: 0: STDOUT: maxwell:[19/May/2016 16:14:55] [INFO]: [Extension MaxwellVolumetric] Initializing Extension Object
2016-05-19 16:14:55: 0: STDOUT: maxwell:[19/May/2016 16:14:55] [INFO]: [Extension MaxwellVolumetric] Opening : N:\renderfarm\Scene_01_Frame_0-1500\explosion.vdb.
2016-05-19 16:14:56: 0: STDOUT: failed to create a temporary copy of N:\renderfarm\Scene_01_Frame_0-1500\explosion.vdb for delayed loading (IoError: failed to open temporary file \s5cc.); will read directly from N:\renderfarm\Scene_01_Frame_0-1500\explosion.vdb instead

=======================================================
Details

Date: 05/19/2016 16:14:57
Frames: 593
Elapsed Time: 00:00:04:22
Job Submit Date: 05/19/2016 14:59:52
Job User: domainuser
Average RAM Usage: 8051458560 (12%)
Peak RAM Usage: 9354391552 (14%)
Average CPU Usage: 2%
Peak CPU Usage: 10%
Used CPU Clocks: 228096
Total CPU Clocks: 11404800

=======================================================
Slave Information

Slave Name: removedname
Version: v7.1.2.1 R (53013d5bd)
Operating System: Windows 7 Enterprise (SP1)
Running As Service: Yes
Machine User: removeduser
IP Address: removedip
MAC Address: removedmac
CPU Architecture: x64
CPUs: 32
CPU Usage: 0%
Memory Usage: 7.9 GB / 63.9 GB (12%)
Free Disk Space: 4.047 TB (52.478 GB on C:, 3.928 TB on D:, 69.385 GB on E:)
Video Card: NVIDIA Quadro 4000[/code]

Edit - wrapping in code display

2

I’m pretty sure it’s not Deadline, and this one’s new to me.

Do you want to try rendering on “removedname” (;)) with the following:

"C:\Program Files\Next Limit\Maxwell 3\maxwell.exe" -mxs:"N:\renderfarm\Scene_01_Frame_0-1500\Final_Stereo_Renderfarm_Test_Left.0580.mxs" -nowait -hide -animation:593-593 -threads:32 -time:15.0 -sampling:20.0 -node -verbose:4 -output:"N:\renderfarm\Rendered_Images\Scene_01_Left.00580.png" -mxi:"N:\renderfarm\Rendered_Images\Scene_01_Left.00580.0593.mxi"

It likely won’t make much difference, but it’s nice to have the issue isolated if you bring it over to Next Limit. Are the paths being remapped at all?

3

Thanks eamsler, i’ll give that a go.

Just spun up a OS X render system and pushed the scene via deadline to it and it’s kicked off nicely, must be something odd with our Windows environment.

We are remapping and it appears to be doing it properly, but i’m now thinking it could be something to do with our Windows Service user (Domain user to add to the complexity) and permission access to the folder it’s attempting to ‘temp’ write too.

4

Yeah permissions can be particularly disruptive to renders. Hopefully they can get sorted out in your farm.