Hi,
We’re using the V-Ray “Restart Renderer Between Frames” option in the SMTD submitter to fix an issue with blank Render Elements being outputted from V-Ray. This works fine, however when using this tweak V-Ray occationally fails to obtain a license and throws an error and thus wastes time and clutters the job report. When rendering without this tweak none of our slaves throws an error.
Any suggestions in fixing this would be greatly appreciated.
Hi Peter,
This is very strange, as I have rendered for years with this setting always on (except in the case of sending IM/LC calculating jobs) to a farm of 200+ machines and never seen this issue. It’s my understanding that we are unloading and re-loading the vray plugins to force a purge of memory from 3dsMax, thereby fixing the blank RE’s. However, VRay holds it license per 3dsmax session, so in theory there shouldn’t be a license issue. You say it happens occasionally ~is there any pattern here? Could you share job reports for both successful and errored jobs? Are you able to see anything on your vray license server which would indicate an issue here? Do your artists or any other machines sometimes have an issue obtaining a vray license during normal use? Can you confirm exact versions of Deadline, 3dsmax, vray, license server you are using?
Regards,
Mike
Hi Mike,
This does indeed happen occationally, after failing to obtain the license and restarting/moving to another task, the slave renders correctly without any license issues. We do not have any license issues on our workstations. I haven’t been able to see a pattern yet… it seems pretty random.
*Edit: Workstations throws same type of error when rendering as part of the farm on the same job.
Please find samples of the job reports included below (from a test job):
Job report - Throwing error:
Error in RenderTasks: RenderTask: Unexpected exception (Failed to open the max renderer.
8] SYSTEM: Production renderer is changed to NVIDIA mental ray. Previous messages are cleared.
2014/09/26 12:17:12 DBG: [09048] [08448] Interface8::LoadFromFile(C:\Users\render\AppData\Local\Thinkbox\Deadline6\slave\RD-SVG-001\plugins\54253cce9856552b58ec5d7c\deadlineStartupMax2014.max)
2014/09/26 12:17:12 INF: [09048] [08448] Starting to load file: C:\Users\render\AppData\Local\Thinkbox\Deadline6\slave\RD-SVG-001\plugins\54253cce9856552b58ec5d7c\deadlineStartupMax2014.max
2014/09/26 12:17:13 INF: [09048] [08448] Done loading file: C:\Users\render\AppData\Local\Thinkbox\Deadline6\slave\RD-SVG-001\plugins\54253cce9856552b58ec5d7c\deadlineStartupMax2014.max
2014/09/26 12:17:13 INF: [09048] [08448] SYSTEM: Production renderer is changed to Default Scanline Renderer. Previous messages are cleared.
2014/09/26 12:17:13 DBG: [09048] [08448] Interface8::LoadFromFile(C:/Users/render/AppData/Local/Thinkbox/Deadline6/slave/RD-SVG-001/jobsData/54253cce9856552b58ec5d7c/Test_av_animasjon_04.max)
2014/09/26 12:17:13 INF: [09048] [08448] Starting to load file: C:/Users/render/AppData/Local/Thinkbox/Deadline6/slave/RD-SVG-001/jobsData/54253cce9856552b58ec5d7c/Test_av_animasjon_04.max
2014/09/26 12:17:14 WRN: [09048] [08448] Gamma Correction Settings are Being Changed
2014/09/26 12:17:38 INF: [09048] [08448] Done loading file: C:/Users/render/AppData/Local/Thinkbox/Deadline6/slave/RD-SVG-001/jobsData/54253cce9856552b58ec5d7c/Test_av_animasjon_04.max
2014/09/26 12:17:38 INF: [09048] [08448] SYSTEM: Production renderer is changed to V-Ray Adv 3.00.07. Previous messages are cleared.
2014/09/26 12:25:53 ERR: [09048] [08448] [V-Ray] Could not obtain a license (0).
2014/09/26 12:25:53 ERR: [09048] [08448] [V-Ray] 1 error(s), 0 warning(s)
)
at Deadline.Plugins.ScriptPlugin.RenderTasks(String taskId, Int32 startFrame, Int32 endFrame, String& outMessage, AbortLevel& abortLevel)
RenderPluginException
at Deadline.Plugins.Plugin.RenderTask(String taskId, Int32 startFrame, Int32 endFrame)
at Deadline.Slaves.SlaveRenderThread.RenderCurrentTask(TaskLogWriter tlw)
0: Task timeout is disabled.
0: Plugin rendering frame(s): 92
0: INFO: Render Tasks called
0: INFO: STARTED
0: INFO: Lightning: Render frame 92
0: INFO: Lightning: Overriding save file option to 1
0: INFO: Lightning: Rendering frame \192.168.10.10\3d$\11_Render_inn\Global Maritime\Wake\Testrender\SLETT\Vraytest\Vraytest_0092.exr
0: INFO: Lightning: Rendering 14 render elements
0: INFO: Lightning: Setting up render parameters
0: INFO: Lightning: Rendering camera node Cam - in sea (template)
0: INFO: Lightning: Checking multipass info for frame 92
0: INFO: Lightning: Opening renderer
Date: 09/26/2014 12:25:55
Frames: 92
Elapsed Time: 00:00:00:04
Job Submit Date: 09/26/2014 12:15:42
Job User: peter
Average RAM Usage: 6414438912 (19%)
Peak RAM Usage: 6452781056 (19%)
Average CPU Usage: 2%
Peak CPU Usage: 5%
Slave Name: RD-SVG-001
Version: v6.1.0.54665 R
Operating System: Windows 7 Enterprise (SP1)
CPU Architecture: x64
CPUs: 32
CPU Usage: 0%
Memory Usage: 6.0 GB / 31.9 GB (18%)
Free Disk Space: 788.094 GB
Video Card: AMD FirePro W5000 (FireGL V) Graphics Adapter
Job report successful job:
0: Task timeout is disabled.
0: Plugin rendering frame(s): 146
0: INFO: Render Tasks called
0: INFO: STARTED
0: INFO: Lightning: Render frame 146
0: INFO: Lightning: Overriding save file option to 1
0: INFO: Lightning: Rendering frame \192.168.10.10\3d$\11_Render_inn\Global Maritime\Wake\Testrender\SLETT\Vraytest\Vraytest_0146.exr
0: INFO: Lightning: Rendering 14 render elements
0: INFO: Lightning: Setting up render parameters
0: INFO: Lightning: Rendering camera node Cam - in sea (template)
0: INFO: Lightning: Checking multipass info for frame 146
0: INFO: Lightning: Opening renderer
0: INFO: Lightning: Multipass disabled
0: INFO: Lightning: Rendering 1 passes
0: INFO: Lightning: Checking output paths
0: INFO: Lightning: Checking default actions
0: INFO: Lightning: TYPE_MISSING_EXTERNAL_FILES = FAIL
0: INFO: Lightning: TYPE_MISSING_UVWS = IGNORE
0: INFO: Lightning: TYPE_MISSING_DLL_FILES = FAIL
0: INFO: Lightning: TYPE_MISSING_XREF_FILES = IGNORE
0: INFO: Lightning: Calling renderer
0: INFO: Preparing tone operator.
0: INFO: Preparing camera sampler…
0: INFO: Preparing objects
0: INFO: Loading bitmaps…
0: INFO: Updating objects
0: INFO: Updating instances
0: INFO: Preparing lights
0: INFO: Preparing lights
0: INFO: Preparing lights (VRayLight - In sea - LIGHT)
0: INFO: Preparing lights (VRayLight - In sea - REFL)
0: INFO: Preparing lights (VRayLight - In sea - REFL) [00:00:00.2] [00:00:00.3 est]
0: INFO: Preparing lights (VRayLight - In sea - REFL): done [00:00:00.2]
0: INFO: Preparing atmospherics
0: INFO: Preparing render effects
0: INFO: Compiling scene geometry
0: INFO: Preparing ray server…
0: INFO: Building static raycast accelerator…
0: INFO: Building static raycast accelerator…: done [00:00:00.1]
0: INFO: Building motion blur raycast accelerator…
0: INFO: Building static hair raycast accelerator…
0: INFO: Building motion blur hair raycast accelerator…
0: INFO: Preparing direct light manager…
0: INFO: Preparing global light manager…
0: INFO: Lightning: CallCurRendererRenderFrame returned code 1
0: INFO: Lightning: Render done
0: INFO: Lightning: Saved image to \192.168.10.10\3d$\11_Render_inn\Global Maritime\Wake\Testrender\SLETT\Vraytest\Vraytest_0146.exr
0: INFO: Lightning: Checking render elements
0: INFO: Lightning: Saved image to \192.168.10.10\3d$\11_Render_inn\Global Maritime\Wake\Testrender\SLETT\Vraytest\VRayDiffuseFilter\VRayDiffuseFilter_0146.exr
0: INFO: Lightning: Saved image to \192.168.10.10\3d$\11_Render_inn\Global Maritime\Wake\Testrender\SLETT\Vraytest\VRayLighting\VRayLighting_0146.exr
0: INFO: Lightning: Saved image to \192.168.10.10\3d$\11_Render_inn\Global Maritime\Wake\Testrender\SLETT\Vraytest\VRayGlobalIllumination\VRayGlobalIllumination_0146.exr
0: INFO: Lightning: Saved image to \192.168.10.10\3d$\11_Render_inn\Global Maritime\Wake\Testrender\SLETT\Vraytest\VRayReflectionFilter\VRayReflectionFilter_0146.exr
0: INFO: Lightning: Saved image to \192.168.10.10\3d$\11_Render_inn\Global Maritime\Wake\Testrender\SLETT\Vraytest\VRayReflection\VRayReflection_0146.exr
0: INFO: Lightning: Saved image to \192.168.10.10\3d$\11_Render_inn\Global Maritime\Wake\Testrender\SLETT\Vraytest\VRayRefractionFilter\VRayRefractionFilter_0146.exr
0: INFO: Lightning: Saved image to \192.168.10.10\3d$\11_Render_inn\Global Maritime\Wake\Testrender\SLETT\Vraytest\VRayRefraction\VRayRefraction_0146.exr
0: INFO: Lightning: Saved image to \192.168.10.10\3d$\11_Render_inn\Global Maritime\Wake\Testrender\SLETT\Vraytest\VRaySpecular\VRaySpecular_0146.exr
0: INFO: Lightning: Saved image to \192.168.10.10\3d$\11_Render_inn\Global Maritime\Wake\Testrender\SLETT\Vraytest\VRayZDepth\VRayZDepth_0146.exr
0: INFO: Lightning: Saved image to \192.168.10.10\3d$\11_Render_inn\Global Maritime\Wake\Testrender\SLETT\Vraytest\VRayAlpha\VRayAlpha_0146.exr
0: INFO: Lightning: Saved image to \192.168.10.10\3d$\11_Render_inn\Global Maritime\Wake\Testrender\SLETT\Vraytest\VRayExtraTex_AOT\VRayExtraTex_AOT_0146.exr
0: INFO: Lightning: Saved image to \192.168.10.10\3d$\11_Render_inn\Global Maritime\Wake\Testrender\SLETT\Vraytest\VRayExtraTex_AOL\VRayExtraTex_AOL_0146.exr
0: INFO: Lightning: Saved image to \192.168.10.10\3d$\11_Render_inn\Global Maritime\Wake\Testrender\SLETT\Vraytest\MultiMatteElementSea\MultiMatteElementSea_0146.exr
0: INFO: Lightning: Saved image to \192.168.10.10\3d$\11_Render_inn\Global Maritime\Wake\Testrender\SLETT\Vraytest\MultiMatteElement01\MultiMatteElement01_0146.exr
Date: 09/26/2014 12:31:14
Frames: 146
Job Submit Date: 09/26/2014 12:15:42
Job User: peter
Average RAM Usage: 0 (0%)
Peak RAM Usage: 0 (0%)
Average CPU Usage: 0%
Peak CPU Usage: 0%
Slave Name: RD-SVG-003
Version: v6.1.0.54665 R
Operating System: Windows 7 Enterprise (SP1)
CPU Architecture: x64
CPUs: 32
CPU Usage: 0%
Memory Usage: 5.6 GB / 31.9 GB (17%)
Free Disk Space: 790.013 GB
Video Card: AMD FirePro W5000 (FireGL V) Graphics Adapter
We are using the following software versions:
Deadline 6.1.0.54665 R
Deadline license manager v11.10
3ds Max Design 2014 SP5
V-Ray 3.00.07 / License server 3.0
RPManager 6.031
Hmmm…tricky.
Few things you could try to see if it narrows down the issue.
Deadline v6.2 is already out and free if on annual support. Needs a newer v6.2 license, please contact sales for the license cut & download link. There is also a v6.2.1 maintenance release which will shortly be released, which includes even more SMTD/3dsmax fixes in it on top of the v6.2 features/fixes. The v6.2.1 is available via our beta forum if you have access or want to try it. Access to beta can be obtained via email to beta@thinkboxsoftware.com
Update VRay License Server to v3.05
Setup a secondary license server incase this is a network connectivity issue which is only seen when a large number of machines make a request for a license at the same time (ie: would only be seen by your farm and not necessarily by workstations).
docs.chaosgroup.com/display/VRAY … figuration
Monitor license server during rendering. Are licenses being released in a timely fashion. Silly question, but as the licensing changed in VRay v3, do you have enough rendernode licenses?!
Chaos Group could confirm if anything can be inferred by the license error message: “[V-Ray] Could not obtain a license (0)”
There was a vray beta issue when running RPManager at the same time. Hopefully, the fix is included for this in v3.00.07 that you are running. (I think so). You could always request access to the Chaos Group nightly build server to get you the latest nightly build, although looking through the release notes for you, I’m not seeing anything interesting there.
I’m still surprised this license error “only” occurs with our “Restart Renderer” checkbox being enabled. Could you go into repo options, application logging and enable Slave verbose logging and see if that helps to get us some more information which we might be able to infer something from.
Apart from the above, I’m struggling to see anything obvious yet. Let us know if anything from above, helps.
Mike
Mike,
Many thanks for all the suggestions. I’ll go through them and get back to you on the result.
I can however confirm we have enough V-Ray licenses, so thats one possibility ruled out.
Mike,
Problem is now solved.
The older slaves had a secondary license server setup from the old days of V-Ray. I assume that as the new license server was busy, the nodes looked to the workstations for licenses, but with the new license system they had none to give, and if they had, the workstation would then fail as the license was occupied by a renderslave. Removing the secondary license server on these machines fixed this issue.
Many thanks for the quick response and pointing me in the right direction!
Glad you got it sorted! I got lucky on a Friday!
Regards,
Mike