Cross platform and backslash fixing (regular expressions)

Deadline
1

Hey guys,

let me first explain the situation.

We own a studio mostly running windows workstations, primary software being Maya, VRay, Fusion and of course Deadline.
Everyone works from a server mounted as Z:\ on windows.
We have just ordered a large batch of render nodes and are testing whether or not Linux (CentOs 6 in our case) is a good solution to the ridiculous costs the OS would bring on those machines should we need to buy Windows.

We have everything running, and all is good and well as long as you stay in either just Linux, or just Windows. The moment we try to mix and match, the infernal backslash directory separator that Microsoft insists on using causes havoc.
Now, we solved the Z:\ drive easily by using the path mapping feature by just mapping to the /mnt/Z/ mount point, but the problem is that when you submit from Deadline on windows, all directories are still separated by a backslash “” (we can fix it by hand, but that gets old real fast, and is a sure way to end up with render batches that failed due to human error.)
At this point we tried mixing the path mapping of the drive letter (Z: to /mnt/Z) together with a second path map, but it seems that whenever a path map is executed, it grabs the original string (correct me if i’m wrong here), or it executes just the first path map that matches its criteria.
We tried using regular expressions for the slashes, which works, but then the drive letter can’t be replaced with a mount point.

My question is, what is the solution to this problem?

Is it possible to use regular expression conditionals/grouping? I can’t really find any real information on the subject in the manual.

Hope i made sense and someone can help us with this problem.

Cheers,
Sven

2

Can you explain the havoc in more detail? More specifically, which applications (Maya, Vray, etc) it’s affecting? In our experience, some are good at dealing with mismatched slashes and some just explode, and we’re trying to handle it as necessary in our plugins. The next 5.1 update will actually include more improvements with regards to slashes for certain plugins.

If you have error messages from jobs that are due to bad slashes, please post them!

Cheers,

  • Ryan
3

Hey Ryan,

i’ll post some of the error logs with details on what we did to get those errors. Won’t be until Tuesday though, holiday and all.

cheers,

Sven

4

Hey Ryan,

back at trying to solve this…

Okay, so at the moment we are trying this with Vray for Maya, but submitted to Vray Standalone (so, i guess it is safe to disregard the whole Maya thing.)

I submit a vrscene file (inside the file i already fixed the paths, making sure paths become relative by doing the …/ directory up name to work relative from the vrscene file location for meshes and textures) to vray standalone.

When i submit the file via Deadline on a windows machine, the path to the vrscene in the job has the backward slashes Z:\Projects\Tests\data\vray_net_render_test.vrscene
This is a problem in that the slashes need to be fixed manually right now and this process is tedious and human error prone.

When i use path mapping set to :

Path To Replace Windows Path Linux Path Mac Path Case Sensitive Regular Expression Z: Z: /mnt/Z /volumes/Z True False
I get the following error:

[code]=======================================================
Error Message

Exception during render: An error occurred in RenderTasks(): error: Error in file “/mnt/Z\Projects\Tests\data\vray_net_render_test.vrscene”, line 0: No file “/mnt/Z\Projects\Tests\data\vray_net_render_test.vrscene” exists
at Deadline.Plugins.ScriptPlugin.RenderTasks (System.String taskId, Int32 startFrame, Int32 endFrame, System.String& outMessage) [0x00000] in :0

=======================================================
Slave Log

0: Task timeout is disabled.
0: Loaded job: vray net render test (999_050_999_30e1dae5)
0: INFO: StartJob: initializing script plugin Vray
0: INFO: Any stdout that matches the regular expression “error:." will be handled as appropriate
0: INFO: Any stdout that matches the regular expression "Rendering image…: ([0-9].[0-9])%.” will be handled as appropriate
0: INFO: Any stdout that matches the regular expression “Rendering image…: done." will be handled as appropriate
0: INFO: Any stdout that matches the regular expression "Starting frame ([0-9])." will be handled as appropriate
0: INFO: Any stdout that matches the regular expression "Closing log.” will be handled as appropriate
0: INFO: About: VRay Plugin for Deadline, 2007-1-16
0: Plugin rendering frame(s): 1
0: INFO: Starting VRay Task
0: INFO: Stdout Handling Enabled: True
0: INFO: Popup Handling Enabled: False
0: INFO: Using Process Tree: True
0: INFO: Hiding DOS Window: True
0: INFO: Creating New Console: False
0: INFO: Render Executable: “/usr/autodesk/maya2012-x64/vray/bin/vray”
0: INFO: CheckPathMapping: Swapped “Z:\Projects\Tests\data\vray_net_render_test.vrscene” with “/mnt/Z\Projects\Tests\data\vray_net_render_test.vrscene”
0: INFO: Render Argument: -scenefile="/mnt/Z\Projects\Tests\data\vray_net_render_test.vrscene" -interactive=0 -display=0 -frames=1 -numThreads=0
0: INFO: Startup Directory: “/usr/autodesk/maya2012-x64/vray/bin”
0: INFO: Process Priority: BelowNormal
0: INFO: Process Affinity: default
0: INFO: Process is now running
0: STDOUT: [2012/Apr/11|12:27:35] Reading configuration from “/usr/autodesk/maya2012-x64/vray/bin//vrayconfig.xml”
0: STDOUT: [2012/Apr/11|12:27:35] ===============================================
0: STDOUT: [2012/Apr/11|12:27:35] V-Ray Standalone, version 2.20.01 for x64
0: STDOUT: [2012/Apr/11|12:27:35] Copyright © 2000-2010 Chaos Group Ltd. All rights reserved.
0: STDOUT: [2012/Apr/11|12:27:35] Use -credits option for additional third-party copyright notices.
0: STDOUT: [2012/Apr/11|12:27:35]
0: STDOUT: [2012/Apr/11|12:27:35] Build from Feb 1 2012, 17:00:56
0: STDOUT: [2012/Apr/11|12:27:35] Compiled with GCC-compatible compiler, version 4.1
0: STDOUT: [2012/Apr/11|12:27:35] Operating system is Linux
0: STDOUT: [2012/Apr/11|12:27:35]
0: STDOUT: [2012/Apr/11|12:27:35] V-Ray core version is 2.00.01
0: STDOUT: [2012/Apr/11|12:27:35] ===============================================
0: STDOUT: [2012/Apr/11|12:27:35]
0: STDOUT: [2012/Apr/11|12:27:35] Loading plugins from “/usr/autodesk/maya2012-x64/vray/vrayplugins/libvray_*.so”
0: STDOUT: [2012/Apr/11|12:27:35] 102 plugin(s) loaded successfully
0: STDOUT: [2012/Apr/11|12:27:35] Finished loading plugins.
0: STDOUT: [2012/Apr/11|12:27:35] Reading scene file “/mnt/Z\Projects\Tests\data\vray_net_render_test.vrscene”
0: STDOUT: [2012/Apr/11|12:27:35] error: Error in file “/mnt/Z\Projects\Tests\data\vray_net_render_test.vrscene”, line 0: No file “/mnt/Z\Projects\Tests\data\vray_net_render_test.vrscene” exists
0: STDOUT: [2012/Apr/11|12:27:35] Closing log - 1 error(s), 0 warning(s).

=======================================================
Error Type

RenderPluginException

=======================================================
Error Stack Trace

at Deadline.Plugins.Plugin.RenderTask (System.String taskId, Int32 startFrame, Int32 endFrame) [0x00000] in :0
at Deadline.Slaves.SlaveRenderThread.RenderCurrentTask (Deadline.IO.TaskLogWriter tlw) [0x00000] in :0
[/code]

When i leave it as is (no path mapping), it gets even worse on Linux:

[code]=======================================================
Error Message

Exception during render: An error occurred in RenderTasks(): error: Error in file “/usr/autodesk/maya2012-x64/vray/bin/Z:\Projects\Tests\data\vray_net_render_test.vrscene”, line 0: No file “/usr/autodesk/maya2012-x64/vray/bin/Z:\Projects\Tests\data\vray_net_render_test.vrscene” exists
at Deadline.Plugins.ScriptPlugin.RenderTasks (System.String taskId, Int32 startFrame, Int32 endFrame, System.String& outMessage) [0x00000] in :0

=======================================================
Slave Log

0: Task timeout is disabled.
0: Loaded job: vray net render test (999_050_999_30e1dae5)
0: INFO: StartJob: initializing script plugin Vray
0: INFO: Any stdout that matches the regular expression “error:." will be handled as appropriate
0: INFO: Any stdout that matches the regular expression "Rendering image…: ([0-9].[0-9])%.” will be handled as appropriate
0: INFO: Any stdout that matches the regular expression “Rendering image…: done." will be handled as appropriate
0: INFO: Any stdout that matches the regular expression "Starting frame ([0-9])." will be handled as appropriate
0: INFO: Any stdout that matches the regular expression "Closing log.” will be handled as appropriate
0: INFO: About: VRay Plugin for Deadline, 2007-1-16
0: Plugin rendering frame(s): 1
0: INFO: Starting VRay Task
0: INFO: Stdout Handling Enabled: True
0: INFO: Popup Handling Enabled: False
0: INFO: Using Process Tree: True
0: INFO: Hiding DOS Window: True
0: INFO: Creating New Console: False
0: INFO: Render Executable: “/usr/autodesk/maya2012-x64/vray/bin/vray”
0: INFO: Render Argument: -scenefile=“Z:\Projects\Tests\data\vray_net_render_test.vrscene” -interactive=0 -display=0 -frames=1 -numThreads=0
0: INFO: Startup Directory: “/usr/autodesk/maya2012-x64/vray/bin”
0: INFO: Process Priority: BelowNormal
0: INFO: Process Affinity: default
0: INFO: Process is now running
0: STDOUT: [2012/Apr/11|12:26:06] Reading configuration from “/usr/autodesk/maya2012-x64/vray/bin//vrayconfig.xml”
0: STDOUT: [2012/Apr/11|12:26:06] ===============================================
0: STDOUT: [2012/Apr/11|12:26:06] V-Ray Standalone, version 2.20.01 for x64
0: STDOUT: [2012/Apr/11|12:26:06] Copyright © 2000-2010 Chaos Group Ltd. All rights reserved.
0: STDOUT: [2012/Apr/11|12:26:06] Use -credits option for additional third-party copyright notices.
0: STDOUT: [2012/Apr/11|12:26:06]
0: STDOUT: [2012/Apr/11|12:26:06] Build from Feb 1 2012, 17:00:56
0: STDOUT: [2012/Apr/11|12:26:06] Compiled with GCC-compatible compiler, version 4.1
0: STDOUT: [2012/Apr/11|12:26:06] Operating system is Linux
0: STDOUT: [2012/Apr/11|12:26:06]
0: STDOUT: [2012/Apr/11|12:26:06] V-Ray core version is 2.00.01
0: STDOUT: [2012/Apr/11|12:26:06] ===============================================
0: STDOUT: [2012/Apr/11|12:26:06]
0: STDOUT: [2012/Apr/11|12:26:06] Loading plugins from “/usr/autodesk/maya2012-x64/vray/vrayplugins/libvray_*.so”
0: STDOUT: [2012/Apr/11|12:26:06] 102 plugin(s) loaded successfully
0: STDOUT: [2012/Apr/11|12:26:06] Finished loading plugins.
0: STDOUT: [2012/Apr/11|12:26:06] Reading scene file “/usr/autodesk/maya2012-x64/vray/bin/Z:\Projects\Tests\data\vray_net_render_test.vrscene”
0: STDOUT: [2012/Apr/11|12:26:06] error: Error in file “/usr/autodesk/maya2012-x64/vray/bin/Z:\Projects\Tests\data\vray_net_render_test.vrscene”, line 0: No file “/usr/autodesk/maya2012-x64/vray/bin/Z:\Projects\Tests\data\vray_net_render_test.vrscene” exists
0: STDOUT: [2012/Apr/11|12:26:06] Closing log - 1 error(s), 0 warning(s).

=======================================================
Error Type

RenderPluginException

=======================================================
Error Stack Trace

at Deadline.Plugins.Plugin.RenderTask (System.String taskId, Int32 startFrame, Int32 endFrame) [0x00000] in :0
at Deadline.Slaves.SlaveRenderThread.RenderCurrentTask (Deadline.IO.TaskLogWriter tlw) [0x00000] in :0
[/code]

When i add a regex to replace the slashes:

Path To Replace Windows Path Linux Path Mac Path Case Sensitive Regular Expression ([\\]) / / / True True
It replaces the slashes but of course the mount point is wrong:

[code]=======================================================
Error Message

Exception during render: An error occurred in RenderTasks(): error: Error in file “/usr/autodesk/maya2012-x64/vray/bin/Z:/Projects/Tests/data/vray_net_render_test.vrscene”, line 0: No file “/usr/autodesk/maya2012-x64/vray/bin/Z:/Projects/Tests/data/vray_net_render_test.vrscene” exists
at Deadline.Plugins.ScriptPlugin.RenderTasks (System.String taskId, Int32 startFrame, Int32 endFrame, System.String& outMessage) [0x00000] in :0

=======================================================
Slave Log

0: Task timeout is disabled.
0: Loaded job: vray net render test (999_050_999_30e1dae5)
0: INFO: StartJob: initializing script plugin Vray
0: INFO: Any stdout that matches the regular expression “error:." will be handled as appropriate
0: INFO: Any stdout that matches the regular expression "Rendering image…: ([0-9].[0-9])%.” will be handled as appropriate
0: INFO: Any stdout that matches the regular expression “Rendering image…: done." will be handled as appropriate
0: INFO: Any stdout that matches the regular expression "Starting frame ([0-9])." will be handled as appropriate
0: INFO: Any stdout that matches the regular expression "Closing log.” will be handled as appropriate
0: INFO: About: VRay Plugin for Deadline, 2007-1-16
0: Plugin rendering frame(s): 1
0: INFO: Starting VRay Task
0: INFO: Stdout Handling Enabled: True
0: INFO: Popup Handling Enabled: False
0: INFO: Using Process Tree: True
0: INFO: Hiding DOS Window: True
0: INFO: Creating New Console: False
0: INFO: Render Executable: “/usr/autodesk/maya2012-x64/vray/bin/vray”
0: INFO: CheckPathMapping: Swapped “Z:\Projects\Tests\data\vray_net_render_test.vrscene” with “Z:/Projects/Tests/data/vray_net_render_test.vrscene”
0: INFO: Render Argument: -scenefile=“Z:/Projects/Tests/data/vray_net_render_test.vrscene” -interactive=0 -display=0 -frames=1 -numThreads=0
0: INFO: Startup Directory: “/usr/autodesk/maya2012-x64/vray/bin”
0: INFO: Process Priority: BelowNormal
0: INFO: Process Affinity: default
0: INFO: Process is now running
0: STDOUT: [2012/Apr/11|12:24:48] Reading configuration from “/usr/autodesk/maya2012-x64/vray/bin//vrayconfig.xml”
0: STDOUT: [2012/Apr/11|12:24:48] ===============================================
0: STDOUT: [2012/Apr/11|12:24:48] V-Ray Standalone, version 2.20.01 for x64
0: STDOUT: [2012/Apr/11|12:24:48] Copyright © 2000-2010 Chaos Group Ltd. All rights reserved.
0: STDOUT: [2012/Apr/11|12:24:48] Use -credits option for additional third-party copyright notices.
0: STDOUT: [2012/Apr/11|12:24:48]
0: STDOUT: [2012/Apr/11|12:24:48] Build from Feb 1 2012, 17:00:56
0: STDOUT: [2012/Apr/11|12:24:48] Compiled with GCC-compatible compiler, version 4.1
0: STDOUT: [2012/Apr/11|12:24:48] Operating system is Linux
0: STDOUT: [2012/Apr/11|12:24:48]
0: STDOUT: [2012/Apr/11|12:24:48] V-Ray core version is 2.00.01
0: STDOUT: [2012/Apr/11|12:24:48] ===============================================
0: STDOUT: [2012/Apr/11|12:24:48]
0: STDOUT: [2012/Apr/11|12:24:48] Loading plugins from “/usr/autodesk/maya2012-x64/vray/vrayplugins/libvray_*.so”
0: STDOUT: [2012/Apr/11|12:24:48] 102 plugin(s) loaded successfully
0: STDOUT: [2012/Apr/11|12:24:48] Finished loading plugins.
0: STDOUT: [2012/Apr/11|12:24:48] Reading scene file “/usr/autodesk/maya2012-x64/vray/bin/Z:/Projects/Tests/data/vray_net_render_test.vrscene”
0: STDOUT: [2012/Apr/11|12:24:48] error: Error in file “/usr/autodesk/maya2012-x64/vray/bin/Z:/Projects/Tests/data/vray_net_render_test.vrscene”, line 0: No file “/usr/autodesk/maya2012-x64/vray/bin/Z:/Projects/Tests/data/vray_net_render_test.vrscene” exists
0: STDOUT: [2012/Apr/11|12:24:48] Closing log - 1 error(s), 0 warning(s).

=======================================================
Error Type

RenderPluginException

=======================================================
Error Stack Trace

at Deadline.Plugins.Plugin.RenderTask (System.String taskId, Int32 startFrame, Int32 endFrame) [0x00000] in :0
at Deadline.Slaves.SlaveRenderThread.RenderCurrentTask (Deadline.IO.TaskLogWriter tlw) [0x00000] in :0
[/code]

When i try to mix any of the above regardless the order the are placed in the path mapping, say:

Path To Replace Windows Path Linux Path Mac Path Case Sensitive Regular Expression ([\\]) / / / True True Z: Z: /mnt/Z /volumes/Z True False
I’ll get:

[code]=======================================================
Error Message

Exception during render: An error occurred in RenderTasks(): error: Error in file “/mnt/Z\Projects\Tests\data\vray_net_render_test.vrscene”, line 0: No file “/mnt/Z\Projects\Tests\data\vray_net_render_test.vrscene” exists
at Deadline.Plugins.ScriptPlugin.RenderTasks (System.String taskId, Int32 startFrame, Int32 endFrame, System.String& outMessage) [0x00000] in :0

=======================================================
Slave Log

0: Task timeout is disabled.
0: Loaded job: vray net render test (999_050_999_30e1dae5)
0: INFO: StartJob: initializing script plugin Vray
0: INFO: Any stdout that matches the regular expression “error:." will be handled as appropriate
0: INFO: Any stdout that matches the regular expression "Rendering image…: ([0-9].[0-9])%.” will be handled as appropriate
0: INFO: Any stdout that matches the regular expression “Rendering image…: done." will be handled as appropriate
0: INFO: Any stdout that matches the regular expression "Starting frame ([0-9])." will be handled as appropriate
0: INFO: Any stdout that matches the regular expression "Closing log.” will be handled as appropriate
0: INFO: About: VRay Plugin for Deadline, 2007-1-16
0: Plugin rendering frame(s): 1
0: INFO: Starting VRay Task
0: INFO: Stdout Handling Enabled: True
0: INFO: Popup Handling Enabled: False
0: INFO: Using Process Tree: True
0: INFO: Hiding DOS Window: True
0: INFO: Creating New Console: False
0: INFO: Render Executable: “/usr/autodesk/maya2012-x64/vray/bin/vray”
0: INFO: CheckPathMapping: Swapped “Z:\Projects\Tests\data\vray_net_render_test.vrscene” with “/mnt/Z\Projects\Tests\data\vray_net_render_test.vrscene”
0: INFO: Render Argument: -scenefile="/mnt/Z\Projects\Tests\data\vray_net_render_test.vrscene" -interactive=0 -display=0 -frames=1 -numThreads=0
0: INFO: Startup Directory: “/usr/autodesk/maya2012-x64/vray/bin”
0: INFO: Process Priority: BelowNormal
0: INFO: Process Affinity: default
0: INFO: Process is now running
0: STDOUT: [2012/Apr/11|12:29:17] Reading configuration from “/usr/autodesk/maya2012-x64/vray/bin//vrayconfig.xml”
0: STDOUT: [2012/Apr/11|12:29:17] ===============================================
0: STDOUT: [2012/Apr/11|12:29:17] V-Ray Standalone, version 2.20.01 for x64
0: STDOUT: [2012/Apr/11|12:29:17] Copyright © 2000-2010 Chaos Group Ltd. All rights reserved.
0: STDOUT: [2012/Apr/11|12:29:17] Use -credits option for additional third-party copyright notices.
0: STDOUT: [2012/Apr/11|12:29:17]
0: STDOUT: [2012/Apr/11|12:29:17] Build from Feb 1 2012, 17:00:56
0: STDOUT: [2012/Apr/11|12:29:17] Compiled with GCC-compatible compiler, version 4.1
0: STDOUT: [2012/Apr/11|12:29:17] Operating system is Linux
0: STDOUT: [2012/Apr/11|12:29:17]
0: STDOUT: [2012/Apr/11|12:29:17] V-Ray core version is 2.00.01
0: STDOUT: [2012/Apr/11|12:29:17] ===============================================
0: STDOUT: [2012/Apr/11|12:29:17]
0: STDOUT: [2012/Apr/11|12:29:17] Loading plugins from “/usr/autodesk/maya2012-x64/vray/vrayplugins/libvray_*.so”
0: STDOUT: [2012/Apr/11|12:29:17] 102 plugin(s) loaded successfully
0: STDOUT: [2012/Apr/11|12:29:17] Finished loading plugins.
0: STDOUT: [2012/Apr/11|12:29:17] Reading scene file “/mnt/Z\Projects\Tests\data\vray_net_render_test.vrscene”
0: STDOUT: [2012/Apr/11|12:29:17] error: Error in file “/mnt/Z\Projects\Tests\data\vray_net_render_test.vrscene”, line 0: No file “/mnt/Z\Projects\Tests\data\vray_net_render_test.vrscene” exists
0: STDOUT: [2012/Apr/11|12:29:17] Closing log - 1 error(s), 0 warning(s).

=======================================================
Error Type

RenderPluginException

=======================================================
Error Stack Trace

at Deadline.Plugins.Plugin.RenderTask (System.String taskId, Int32 startFrame, Int32 endFrame) [0x00000] in :0
at Deadline.Slaves.SlaveRenderThread.RenderCurrentTask (Deadline.IO.TaskLogWriter tlw) [0x00000] in :0
[/code]

So, i was wondering we could use something like grouping in regex to replace certain parts with one string, and other parts with another.
Say: (?[Z:])|(?[\]) with /mnt/Z /, or ([Z:])|([\]) with \1/mnt/Z \2/ … or something in that direction.

Or if it doesn’t exist already the abilty to have regex path mappings to work cumulative (so regex works after the drive path map has set the proper mount point)

Hope this made sense…

5

Thanks for the description! I’ve uploaded a modified VRay.py script file which you can unzip to \your\repository\plugins\VRay. You might want to back up the original, just in case.

The VRay plugin now adjusts the slashes in the path as necessary based on the OS after performing the path mapping. So it should now work with your original path mapping settings:

Give it a try and let me know if it helps.

Cheers,

6

My god you guys rock, works like a charm.

Only small cosmetic thing is that the Linux slaves report the following, not sure this was a quick fix just for our farm or that this is something you guys plan on adding to the trunk of your code:

0: INFO: CheckPathMapping: Swapped "Z:\Projects\Tests\data\vray_net_render_test.vrscene" with "/mnt/Z\Projects\Tests\data\vray_net_render_test.vrscene" 0: INFO: Render Argument: -scenefile="/mnt/Z/Projects/Tests/data/vray_net_render_test.vrscene" -interactive=0 -display=0 -frames=1 -numThreads=0

Again, thanks, your support keeps amazing us.

7

Thanks for confirming that it works! We’ll be including this fix in the upcoming 5.1 maintenance release, which should go out next week providing that nothing major comes up.

That output you’re seeing is a result of the path mapping being done before the slashes are converted. It’s nothing to worry about. :slight_smile:

Cheers,

  • Ryan