GPU Affinity on Linux Centos 6.8

[RyanIG]

Hey,

Having some issues rendering on a Fresh Linux system, the system has 8 Nvidia 980Tis in it, and i’ve setup the GPU Affinity so each frame will be allocated two GPUs, So I have 4 slave instances running at a time.

Ex:
GPU 1 - 0,1
GPU 2 - 2,3
GPU 3 - 4,5
GPU 4 - 6,7

I’m used to looking at Deadline slave logs on windows, not so much on Linux, it appears as if the slaves are using all 8 gpus across 4 frames concurrently.

This seems to be causing slaves to hang and it also crashes MayaBatch occasionally, I’m not sure if this log is related to the issues but using GPU affinity on Windows before did resolve many errors for us.

Screenshot of GPU Affinity / Slave instance Names.
http://i67.tinypic.com/2i1yr8k.png

Here’s the log of a Slave, I can attach the full log if needed.

2016-10-06 15:22:57: 0: STDOUT: mel: mel: [Redshift] Cache path: /root/redshift/cache
2016-10-06 15:22:57: 0: STDOUT: [Redshift] Redshift Initialized
2016-10-06 15:22:57: 0: STDOUT: [Redshift] Linux Platform
2016-10-06 15:22:57: 0: STDOUT: [Redshift] Release Build
2016-10-06 15:22:57: 0: STDOUT: [Redshift] Number of CPU HW threads: 24
2016-10-06 15:22:57: 0: STDOUT: [Redshift] Total system memory: 78.59 GB
2016-10-06 15:22:57: 0: STDOUT: [Redshift] Creating CUDA contexts
2016-10-06 15:22:57: 0: STDOUT: [Redshift] CUDA init ok
2016-10-06 15:22:57: 0: STDOUT: [Redshift] Ordinals: { 0 1 2 3 4 5 6 7 }
2016-10-06 15:23:04: 0: STDOUT: [Redshift] Initializing GPUComputing module (CUDA). Ordinal 0
2016-10-06 15:23:04: 0: STDOUT: [Redshift] CUDA Ver: 8000

2016-10-06 15:23:04: 0: STDOUT: [Redshift] Device 1/8 : GeForce GTX 980 Ti
2016-10-06 15:23:04: 0: STDOUT: [Redshift] Compute capability: 5.2
2016-10-06 15:23:04: 0: STDOUT: [Redshift] Num multiprocessors: 22
2016-10-06 15:23:04: 0: STDOUT: [Redshift] PCI busID: 4, deviceID: 0, domainID: 0
2016-10-06 15:23:04: 0: STDOUT: [Redshift] Theoretical memory bandwidth: 336.480011 GB/Sec
2016-10-06 15:23:04: 0: STDOUT: [Redshift] Measured PCIe bandwidth (pinned CPU->GPU): 3.734444 GB/s
2016-10-06 15:23:04: 0: STDOUT: [Redshift] Measured PCIe bandwidth (pinned GPU->CPU): 3.164830 GB/s
2016-10-06 15:23:04: 0: STDOUT: [Redshift] Measured PCIe bandwidth (paged CPU->GPU): 3.587930 GB/s
2016-10-06 15:23:04: 0: STDOUT: [Redshift] Measured PCIe bandwidth (paged GPU->CPU): 5.033083 GB/s
2016-10-06 15:23:04: 0: STDOUT: [Redshift] Estimated GPU->CPU latency (0): 0.013842 ms
2016-10-06 15:23:04: 0: STDOUT: [Redshift] Estimated GPU->CPU latency (1): 0.013315 ms
2016-10-06 15:23:04: 0: STDOUT: [Redshift] Estimated GPU->CPU latency (2): 0.008581 ms
2016-10-06 15:23:04: 0: STDOUT: [Redshift] Estimated GPU->CPU latency (3): 0.008449 ms
2016-10-06 15:23:04: 0: STDOUT: [Redshift] New CUDA context created
2016-10-06 15:23:04: 0: STDOUT: [Redshift] Available memory: 5699.3125 MB out of 6077.3750 MB
2016-10-06 15:23:04: 0: STDOUT: [Redshift] Initializing GPUComputing module (CUDA). Ordinal 1
2016-10-06 15:23:04: 0: STDOUT: [Redshift] CUDA Ver: 8000

2016-10-06 15:23:04: 0: STDOUT: [Redshift] Device 2/8 : GeForce GTX 980 Ti
2016-10-06 15:23:04: 0: STDOUT: [Redshift] Compute capability: 5.2
2016-10-06 15:23:04: 0: STDOUT: [Redshift] Num multiprocessors: 22
2016-10-06 15:23:04: 0: STDOUT: [Redshift] PCI busID: 5, deviceID: 0, domainID: 0
2016-10-06 15:23:04: 0: STDOUT: [Redshift] Theoretical memory bandwidth: 336.480011 GB/Sec
2016-10-06 15:23:04: 0: STDOUT: [Redshift] Measured PCIe bandwidth (pinned CPU->GPU): 5.265391 GB/s
2016-10-06 15:23:04: 0: STDOUT: [Redshift] Measured PCIe bandwidth (pinned GPU->CPU): 3.529710 GB/s
2016-10-06 15:23:04: 0: STDOUT: [Redshift] Measured PCIe bandwidth (paged CPU->GPU): 3.286990 GB/s
2016-10-06 15:23:04: 0: STDOUT: [Redshift] Measured PCIe bandwidth (paged GPU->CPU): 5.071782 GB/s
2016-10-06 15:23:04: 0: STDOUT: [Redshift] Estimated GPU->CPU latency (0): 0.014596 ms
2016-10-06 15:23:04: 0: STDOUT: [Redshift] Estimated GPU->CPU latency (1): 0.014719 ms
2016-10-06 15:23:04: 0: STDOUT: [Redshift] Estimated GPU->CPU latency (2): 0.016304 ms
2016-10-06 15:23:04: 0: STDOUT: [Redshift] Estimated GPU->CPU latency (3): 0.014752 ms
2016-10-06 15:23:04: 0: STDOUT: [Redshift] New CUDA context created
2016-10-06 15:23:04: 0: STDOUT: [Redshift] Available memory: 5767.4375 MB out of 6077.7500 MB
2016-10-06 15:23:05: 0: STDOUT: [Redshift] Initializing GPUComputing module (CUDA). Ordinal 2
2016-10-06 15:23:05: 0: STDOUT: [Redshift] CUDA Ver: 8000

2016-10-06 15:23:05: 0: STDOUT: [Redshift] Device 3/8 : GeForce GTX 980 Ti
2016-10-06 15:23:05: 0: STDOUT: [Redshift] Compute capability: 5.2
2016-10-06 15:23:05: 0: STDOUT: [Redshift] Num multiprocessors: 22
2016-10-06 15:23:05: 0: STDOUT: [Redshift] PCI busID: 8, deviceID: 0, domainID: 0
2016-10-06 15:23:05: 0: STDOUT: [Redshift] Theoretical memory bandwidth: 336.480011 GB/Sec
2016-10-06 15:23:05: 0: STDOUT: [Redshift] Measured PCIe bandwidth (pinned CPU->GPU): 10.262206 GB/s
2016-10-06 15:23:05: 0: STDOUT: [Redshift] Measured PCIe bandwidth (pinned GPU->CPU): 7.233344 GB/s
2016-10-06 15:23:05: 0: STDOUT: [Redshift] Measured PCIe bandwidth (paged CPU->GPU): 2.917630 GB/s
2016-10-06 15:23:05: 0: STDOUT: [Redshift] Measured PCIe bandwidth (paged GPU->CPU): 1.262746 GB/s
2016-10-06 15:23:05: 0: STDOUT: [Redshift] Estimated GPU->CPU latency (0): 0.030813 ms
2016-10-06 15:23:05: 0: STDOUT: [Redshift] Estimated GPU->CPU latency (1): 0.018031 ms
2016-10-06 15:23:05: 0: STDOUT: [Redshift] Estimated GPU->CPU latency (2): 0.017355 ms
2016-10-06 15:23:05: 0: STDOUT: [Redshift] Estimated GPU->CPU latency (3): 0.014718 ms
2016-10-06 15:23:05: 0: STDOUT: [Redshift] New CUDA context created
2016-10-06 15:23:05: 0: STDOUT: [Redshift] Available memory: 5767.4375 MB out of 6077.7500 MB
2016-10-06 15:23:05: 0: STDOUT: [Redshift] Initializing GPUComputing module (CUDA). Ordinal 3
2016-10-06 15:23:05: 0: STDOUT: [Redshift] CUDA Ver: 8000
2016-10-06 15:23:05: 0: STDOUT: [Redshift] Device 4/8 : GeForce GTX 980 Ti

2016-10-06 15:23:05: 0: STDOUT: [Redshift] Compute capability: 5.2
2016-10-06 15:23:05: 0: STDOUT: [Redshift] Num multiprocessors: 22
2016-10-06 15:23:05: 0: STDOUT: [Redshift] PCI busID: 9, deviceID: 0, domainID: 0
2016-10-06 15:23:05: 0: STDOUT: [Redshift] Theoretical memory bandwidth: 336.480011 GB/Sec
2016-10-06 15:23:05: 0: STDOUT: [Redshift] Measured PCIe bandwidth (pinned CPU->GPU): 4.354003 GB/s
2016-10-06 15:23:05: 0: STDOUT: [Redshift] Measured PCIe bandwidth (pinned GPU->CPU): 3.055894 GB/s
2016-10-06 15:23:06: 0: STDOUT: [Redshift] Measured PCIe bandwidth (paged CPU->GPU): 1.430131 GB/s
2016-10-06 15:23:06: 0: STDOUT: [Redshift] Measured PCIe bandwidth (paged GPU->CPU): 1.638408 GB/s
2016-10-06 15:23:06: 0: STDOUT: [Redshift] Estimated GPU->CPU latency (0): 0.023706 ms
2016-10-06 15:23:06: 0: STDOUT: [Redshift] Estimated GPU->CPU latency (1): 0.024728 ms
2016-10-06 15:23:06: 0: STDOUT: [Redshift] Estimated GPU->CPU latency (2): 0.026575 ms
2016-10-06 15:23:06: 0: STDOUT: [Redshift] Estimated GPU->CPU latency (3): 0.025886 ms
2016-10-06 15:23:06: 0: STDOUT: [Redshift] New CUDA context created
2016-10-06 15:23:06: 0: STDOUT: [Redshift] Available memory: 5767.4375 MB out of 6077.7500 MB
2016-10-06 15:23:06: 0: STDOUT: [Redshift] Initializing GPUComputing module (CUDA). Ordinal 4
2016-10-06 15:23:06: 0: STDOUT: [Redshift] CUDA Ver: 8000
2016-10-06 15:23:06: 0: STDOUT: [Redshift] Device 5/8 : GeForce GTX 980 Ti

2016-10-06 15:23:06: 0: STDOUT: [Redshift] Compute capability: 5.2
2016-10-06 15:23:06: 0: STDOUT: [Redshift] Num multiprocessors: 22
2016-10-06 15:23:06: 0: STDOUT: [Redshift] PCI busID: 131, deviceID: 0, domainID: 0
2016-10-06 15:23:06: 0: STDOUT: [Redshift] Theoretical memory bandwidth: 336.480011 GB/Sec
2016-10-06 15:23:06: 0: STDOUT: [Redshift] Measured PCIe bandwidth (pinned CPU->GPU): 10.882689 GB/s
2016-10-06 15:23:06: 0: STDOUT: [Redshift] Measured PCIe bandwidth (pinned GPU->CPU): 6.997245 GB/s
2016-10-06 15:23:06: 0: STDOUT: [Redshift] Measured PCIe bandwidth (paged CPU->GPU): 1.953577 GB/s
2016-10-06 15:23:06: 0: STDOUT: [Redshift] Measured PCIe bandwidth (paged GPU->CPU): 2.272115 GB/s
2016-10-06 15:23:06: 0: STDOUT: [Redshift] Estimated GPU->CPU latency (0): 0.037778 ms
2016-10-06 15:23:06: 0: STDOUT: [Redshift] Estimated GPU->CPU latency (1): 0.030086 ms
2016-10-06 15:23:06: 0: STDOUT: [Redshift] Estimated GPU->CPU latency (2): 0.019099 ms
2016-10-06 15:23:06: 0: STDOUT: [Redshift] Estimated GPU->CPU latency (3): 0.012297 ms
2016-10-06 15:23:06: 0: STDOUT: [Redshift] New CUDA context created
2016-10-06 15:23:06: 0: STDOUT: [Redshift] Available memory: 5767.4375 MB out of 6077.7500 MB
2016-10-06 15:23:06: 0: STDOUT: [Redshift] Initializing GPUComputing module (CUDA). Ordinal 5
2016-10-06 15:23:06: 0: STDOUT: [Redshift] CUDA Ver: 8000
2016-10-06 15:23:06: 0: STDOUT: [Redshift] Device 6/8 : GeForce GTX 980 Ti

2016-10-06 15:23:06: 0: STDOUT: [Redshift] Compute capability: 5.2
2016-10-06 15:23:06: 0: STDOUT: [Redshift] Num multiprocessors: 22
2016-10-06 15:23:06: 0: STDOUT: [Redshift] PCI busID: 132, deviceID: 0, domainID: 0
2016-10-06 15:23:06: 0: STDOUT: [Redshift] Theoretical memory bandwidth: 336.480011 GB/Sec
2016-10-06 15:23:07: 0: STDOUT: [Redshift] Measured PCIe bandwidth (pinned CPU->GPU): 11.277424 GB/s
2016-10-06 15:23:07: 0: STDOUT: [Redshift] Measured PCIe bandwidth (pinned GPU->CPU): 6.180534 GB/s
2016-10-06 15:23:07: 0: STDOUT: [Redshift] Measured PCIe bandwidth (paged CPU->GPU): 1.533541 GB/s
2016-10-06 15:23:07: 0: STDOUT: [Redshift] Measured PCIe bandwidth (paged GPU->CPU): 4.110238 GB/s
2016-10-06 15:23:07: 0: STDOUT: [Redshift] Estimated GPU->CPU latency (0): 0.015370 ms
2016-10-06 15:23:07: 0: STDOUT: [Redshift] Estimated GPU->CPU latency (1): 0.015822 ms
2016-10-06 15:23:07: 0: STDOUT: [Redshift] Estimated GPU->CPU latency (2): 0.017610 ms
2016-10-06 15:23:07: 0: STDOUT: [Redshift] Estimated GPU->CPU latency (3): 0.013428 ms
2016-10-06 15:23:07: 0: STDOUT: [Redshift] New CUDA context created
2016-10-06 15:23:07: 0: STDOUT: [Redshift] Available memory: 5767.4375 MB out of 6077.7500 MB
2016-10-06 15:23:07: 0: STDOUT: [Redshift] Initializing GPUComputing module (CUDA). Ordinal 6
2016-10-06 15:23:07: 0: STDOUT: [Redshift] CUDA Ver: 8000
2016-10-06 15:23:07: 0: STDOUT: [Redshift] Device 7/8 : GeForce GTX 980 Ti

2016-10-06 15:23:07: 0: STDOUT: [Redshift] Compute capability: 5.2
2016-10-06 15:23:07: 0: STDOUT: [Redshift] Num multiprocessors: 22
2016-10-06 15:23:07: 0: STDOUT: [Redshift] PCI busID: 135, deviceID: 0, domainID: 0
2016-10-06 15:23:07: 0: STDOUT: [Redshift] Theoretical memory bandwidth: 336.480011 GB/Sec
2016-10-06 15:23:07: 0: STDOUT: [Redshift] Measured PCIe bandwidth (pinned CPU->GPU): 10.746902 GB/s
2016-10-06 15:23:07: 0: STDOUT: [Redshift] Measured PCIe bandwidth (pinned GPU->CPU): 7.103100 GB/s
2016-10-06 15:23:07: 0: STDOUT: [Redshift] Measured PCIe bandwidth (paged CPU->GPU): 3.847854 GB/s
2016-10-06 15:23:07: 0: STDOUT: [Redshift] Measured PCIe bandwidth (paged GPU->CPU): 1.555237 GB/s
2016-10-06 15:23:07: 0: STDOUT: [Redshift] Estimated GPU->CPU latency (0): 0.134678 ms
2016-10-06 15:23:08: 0: STDOUT: [Redshift] Estimated GPU->CPU latency (1): 0.046141 ms
2016-10-06 15:23:08: 0: STDOUT: [Redshift] Estimated GPU->CPU latency (2): 0.012746 ms
2016-10-06 15:23:08: 0: STDOUT: [Redshift] Estimated GPU->CPU latency (3): 0.013795 ms
2016-10-06 15:23:08: 0: STDOUT: [Redshift] New CUDA context created
2016-10-06 15:23:08: 0: STDOUT: [Redshift] Available memory: 5767.4375 MB out of 6077.7500 MB
2016-10-06 15:23:08: 0: STDOUT: [Redshift] Initializing GPUComputing module (CUDA). Ordinal 7
2016-10-06 15:23:08: 0: STDOUT: [Redshift] CUDA Ver: 8000
2016-10-06 15:23:08: 0: STDOUT: [Redshift] Device 8/8 : GeForce GTX 980 Ti

2016-10-06 15:23:08: 0: STDOUT: [Redshift] Compute capability: 5.2
2016-10-06 15:23:08: 0: STDOUT: [Redshift] Num multiprocessors: 22
2016-10-06 15:23:08: 0: STDOUT: [Redshift] PCI busID: 136, deviceID: 0, domainID: 0
2016-10-06 15:23:08: 0: STDOUT: [Redshift] Theoretical memory bandwidth: 336.480011 GB/Sec
2016-10-06 15:23:08: 0: STDOUT: [Redshift] Measured PCIe bandwidth (pinned CPU->GPU): 9.768453 GB/s
2016-10-06 15:23:08: 0: STDOUT: [Redshift] Measured PCIe bandwidth (pinned GPU->CPU): 4.299793 GB/s
2016-10-06 15:23:08: 0: STDOUT: [Redshift] Measured PCIe bandwidth (paged CPU->GPU): 1.596275 GB/s
2016-10-06 15:23:08: 0: STDOUT: [Redshift] Measured PCIe bandwidth (paged GPU->CPU): 1.582923 GB/s
2016-10-06 15:23:08: 0: STDOUT: [Redshift] Estimated GPU->CPU latency (0): 0.036990 ms
2016-10-06 15:23:08: 0: STDOUT: [Redshift] Estimated GPU->CPU latency (1): 0.010945 ms
2016-10-06 15:23:08: 0: STDOUT: [Redshift] Estimated GPU->CPU latency (2): 0.012484 ms
2016-10-06 15:23:08: 0: STDOUT: [Redshift] Estimated GPU->CPU latency (3): 0.013389 ms
2016-10-06 15:23:08: 0: STDOUT: [Redshift] New CUDA context created
2016-10-06 15:23:08: 0: STDOUT: [Redshift] Available memory: 5871.9375 MB out of 6077.7500 MB
2016-10-06 15:23:13: 0: STDOUT: [Redshift] Loading Redshift procedural extensions…
2016-10-06 15:23:13: 0: STDOUT: [Redshift] Done!
2016-10-06 15:23:13: 0: STDOUT: [Redshift] Redshift for Maya 2016
2016-10-06 15:23:13: 0: STDOUT: [Redshift] Version 2.0.50, Jul 12 2016
2016-10-06 15:23:13: 0: STDOUT: [Redshift] renderable camera = |persp
2016-10-06 15:23:13: 0: STDOUT: [Redshift] Rendering frame 2 (1/1)
2016-10-06 15:23:13: 0: STDOUT: [Redshift] Maya evaluation manager mode: parallel

Edit1:

Deadline Client & Repo Version - 8.1.4.6
http://i66.tinypic.com/2cdzpg9.png

The logs do not contain

This slave is overriding its GPU affinity, so the following GPUs will be used by RedShift:…
http://i64.tinypic.com/orjaf8.png

[MikeOwen]

Could you save out the full job log report from Deadline Monitor and post it here please?

[RyanIG]

Hi Mike,

Sorry for the late reply, Here is the output of the full log

=======================================================
Error

Error: Monitored managed process “MayaBatch” has exited or been terminated.
at Deadline.Plugins.PluginWrapper.RenderTasks (System.String taskId, Int32 startFrame, Int32 endFrame, System.String& outMessage, FranticX.Processes.AbortLevel& abortLevel) <0x41343e60 + 0x00d87> in :0

=======================================================
Type

RenderPluginException

=======================================================
Stack Trace

at Deadline.Plugins.SandboxedPlugin.a (Deadline.Net.DeadlineMessage A_0) <0x41f15380 + 0x005ef> in :0
at Deadline.Plugins.SandboxedPlugin.RenderTask (System.String taskId, Int32 startFrame, Int32 endFrame) <0x41f30a10 + 0x001a3> in :0
at Deadline.Slaves.SlaveRenderThread.a (Deadline.IO.TaskLogWriter A_0) <0x41f29000 + 0x0223a> in :0

=======================================================
Log

2016-10-06 15:21:49: 0: Loading Job’s Plugin timeout is Disabled
2016-10-06 15:21:51: 0: Executing plugin command of type ‘Sync Files for Job’
2016-10-06 15:21:51: 0: Synchronization time for job files: 27.604 ms
2016-10-06 15:21:51: 0: Synchronizing Plugin MayaBatch from /mnt/Deadline8/plugins/MayaBatch took: 0 seconds
2016-10-06 15:21:51: 0: Done executing plugin command of type ‘Sync Files for Job’
2016-10-06 15:21:51: 0: Executing plugin command of type ‘Initialize Plugin’
2016-10-06 15:21:52: 0: INFO: Executing plugin script ‘/tmp/Thinkbox/Deadline8/root/slave/chinook2-gpu4/plugins/57f65df4a0d4413849daacce/MayaBatch.py’
2016-10-06 15:21:52: 0: INFO: About: Maya Batch Plugin for Deadline
2016-10-06 15:21:52: 0: INFO: Render Job As User disabled, running as current user ‘root’
2016-10-06 15:21:52: 0: INFO: The job’s environment will be merged with the current environment before rendering
2016-10-06 15:21:52: 0: Done executing plugin command of type ‘Initialize Plugin’
2016-10-06 15:21:52: 0: Start Job timeout is disabled.
2016-10-06 15:21:52: 0: Task timeout is disabled.
2016-10-06 15:21:52: 0: Loaded job: Chinook2 | Linux Render Test | Centos 6.8 (57f65df4a0d4413849daacce)
2016-10-06 15:21:52: 0: Executing plugin command of type ‘Start Job’
2016-10-06 15:21:52: 0: INFO: Rendering with Maya Version 2016.0
2016-10-06 15:21:52: 0: INFO: Setting MAYA_DISABLE_CIP (ADSK Customer Involvement Program) environment variable to 1 for this session
2016-10-06 15:21:52: 0: INFO: Setting MAYA_DISABLE_CER (ADSK Customer Error Reporting) environment variable to 1 for this session
2016-10-06 15:21:52: 0: INFO: Setting MAYA_DISABLE_CLIC_IPM (ADSK In Product Messaging) environment variable to 1 for this session
2016-10-06 15:21:52: 0: INFO: Starting monitored managed process MayaBatch
2016-10-06 15:21:52: 0: INFO: Stdout Redirection Enabled: True
2016-10-06 15:21:52: 0: INFO: Asynchronous Stdout Enabled: False
2016-10-06 15:21:52: 0: INFO: Stdout Handling Enabled: True
2016-10-06 15:21:52: 0: INFO: Popup Handling Enabled: True
2016-10-06 15:21:52: 0: INFO: QT Popup Handling Enabled: False
2016-10-06 15:21:52: 0: INFO: WindowsForms10.Window.8.app.* Popup Handling Enabled: False
2016-10-06 15:21:52: 0: INFO: Using Process Tree: True
2016-10-06 15:21:52: 0: INFO: Hiding DOS Window: True
2016-10-06 15:21:52: 0: INFO: Creating New Console: False
2016-10-06 15:21:52: 0: INFO: Running as user: root
2016-10-06 15:21:52: 0: INFO: Not enforcing a build of Maya
2016-10-06 15:21:52: 0: INFO: Executable: “/usr/autodesk/maya2016/bin/maya”
2016-10-06 15:21:52: 0: INFO: Argument: -prompt -proj “/mnt/Isilon/data/PROJECTS/project/VFX/reference/Crew/John_H/Archive/TestProject”
2016-10-06 15:21:52: 0: INFO: Full Command: “/usr/autodesk/maya2016/bin/maya” -prompt -proj “/mnt/Isilon/data/PROJECTS/project/VFX/reference/Crew/John_H/Archive/TestProject”
2016-10-06 15:21:52: 0: INFO: Startup Directory: “/usr/autodesk/maya2016/bin”
2016-10-06 15:21:52: 0: INFO: Process Priority: BelowNormal
2016-10-06 15:21:52: 0: INFO: Process Affinity: default
2016-10-06 15:21:52: 0: INFO: Process is now running
2016-10-06 15:21:52: 0: Done executing plugin command of type ‘Start Job’
2016-10-06 15:21:52: 0: Plugin rendering frame(s): 4
2016-10-06 15:21:52: 0: Executing plugin command of type ‘Render Task’
2016-10-06 15:21:52: 0: INFO: Waiting until maya is ready to go
2016-10-06 15:21:55: 0: STDOUT: [Redshift] Redshift for Maya 2016
2016-10-06 15:21:55: 0: STDOUT: [Redshift] Version 2.0.50, Jul 12 2016
2016-10-06 15:22:59: 0: STDOUT: Running submission script “//mnt/Deadline8/submission/Maya/Main/SubmitMayaToDeadline.mel”
2016-10-06 15:22:59: 0: STDOUT: Result: untitled
2016-10-06 15:22:59: 0: STDOUT: READY FOR INPUT
2016-10-06 15:23:00: 0: INFO: >This is a Render Job
2016-10-06 15:23:00: 0: INFO: Rendering with file
2016-10-06 15:23:00: 0: INFO: Rendering to network drive
2016-10-06 15:23:00: 0: INFO: Creating melscript to execute render
2016-10-06 15:23:00: 0: INFO: Adding dirmap command to map “//192.168.168.205/ifs/” to “/mnt/Isilon/”
2016-10-06 15:23:00: 0: INFO: Adding dirmap command to map “//192.168.168.204/ifs/” to “/mnt/Isilon/”
2016-10-06 15:23:00: 0: INFO: Adding dirmap command to map “//192.168.168.203/ifs/” to “/mnt/Isilon/”
2016-10-06 15:23:00: 0: INFO: Adding dirmap command to map “//192.168.168.202/ifs/” to “/mnt/Isilon/”
2016-10-06 15:23:00: 0: INFO: Adding dirmap command to map “//192.168.168.201/ifs/” to “/mnt/Isilon/”
2016-10-06 15:23:00: 0: INFO: Adding dirmap command to map “//192.168.168.200/ifs/” to “/mnt/Isilon/”
2016-10-06 15:23:00: 0: INFO: Executing script: /tmp/tmp7a22ebdf.tmp
2016-10-06 15:23:00: 0: INFO: Waiting for script to finish
2016-10-06 15:23:00: 0: STDOUT: Loading scene: /tmp/Thinkbox/Deadline8/root/slave/chinook2-gpu4/jobsData/57f65df4a0d4413849daacce/Chinook2-Linux-Test.mb
2016-10-06 15:23:02: 0: STDOUT: Initialized VP2.0 renderer {
2016-10-06 15:23:02: 0: STDOUT: Version : 6.3.1.0. Feature Level 4.
2016-10-06 15:23:02: 0: STDOUT: Adapter : GeForce GTX 980 Ti/PCIe/SSE2
2016-10-06 15:23:02: 0: STDOUT: Vendor ID: 4318. Device ID :
2016-10-06 15:23:02: 0: STDOUT: Driver : 4.5.0 NVIDIA 367.44.
2016-10-06 15:23:02: 0: STDOUT: API : OpenGL V.4.5.
2016-10-06 15:23:02: 0: STDOUT: Max texture size : 16384 * 16384.
2016-10-06 15:23:02: 0: STDOUT: Max tex coords : 8
2016-10-06 15:23:02: 0: STDOUT: Shader versions supported (Vertex: 4, Geometry: 4, Pixel 4).
2016-10-06 15:23:02: 0: STDOUT: Shader compiler profile : (Best card profile)
2016-10-06 15:23:02: 0: STDOUT: Active stereo support available : 0
2016-10-06 15:23:02: 0: STDOUT: GPU Memory Limit : 6144 MB.
2016-10-06 15:23:02: 0: STDOUT: CPU Memory Limit: 76451.5 MB.
2016-10-06 15:23:02: 0: STDOUT: }
2016-10-06 15:23:03: 0: STDOUT: OpenCL evaluator is attempting to initialize OpenCL.
2016-10-06 15:23:04: 0: STDOUT: Detected 1 OpenCL Platforms:
2016-10-06 15:23:04: 0: STDOUT: 0: NVIDIA Corporation. NVIDIA CUDA. OpenCL 1.2 CUDA 8.0.0.
2016-10-06 15:23:04: 0: STDOUT: Supported extensions: cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics cl_khr_fp64 cl_khr_byte_addressable_store cl_khr_icd cl_khr_gl_sharing cl_nv_compiler_options cl_nv_device_attribute_query cl_nv_pragma_unroll cl_nv_copy_opts
2016-10-06 15:23:04: 0: STDOUT: OpenCL evaluator choosing OpenCL platform NVIDIA Corporation.
2016-10-06 15:23:04: 0: STDOUT: OpenCL evaluator is unable to find a GPU device that can share VBOs with OpenGL.
2016-10-06 15:23:04: 0: STDOUT: File read in 4.1 seconds.
2016-10-06 15:23:04: 0: STDOUT: Importing XGen Collections…
2016-10-06 15:23:04: 0: STDOUT: Done.
2016-10-06 15:23:04: 0: STDOUT: mel: mel: [Redshift] Cache path: /root/redshift/cache
2016-10-06 15:23:04: 0: STDOUT: [Redshift] Redshift Initialized
2016-10-06 15:23:04: 0: STDOUT: [Redshift] Linux Platform
2016-10-06 15:23:04: 0: STDOUT: [Redshift] Release Build
2016-10-06 15:23:04: 0: STDOUT: [Redshift] Number of CPU HW threads: 24
2016-10-06 15:23:04: 0: STDOUT: [Redshift] Total system memory: 78.59 GB
2016-10-06 15:23:04: 0: STDOUT: [Redshift] Creating CUDA contexts
2016-10-06 15:23:04: 0: STDOUT: [Redshift] CUDA init ok
2016-10-06 15:23:04: 0: STDOUT: [Redshift] Ordinals: { 0 1 2 3 4 5 6 7 }
2016-10-06 15:23:11: 0: STDOUT: [Redshift] Initializing GPUComputing module (CUDA). Ordinal 0
2016-10-06 15:23:11: 0: STDOUT: [Redshift] CUDA Ver: 8000
2016-10-06 15:23:11: 0: STDOUT: [Redshift] Device 1/8 : GeForce GTX 980 Ti
2016-10-06 15:23:11: 0: STDOUT: [Redshift] Compute capability: 5.2
2016-10-06 15:23:11: 0: STDOUT: [Redshift] Num multiprocessors: 22
2016-10-06 15:23:11: 0: STDOUT: [Redshift] PCI busID: 4, deviceID: 0, domainID: 0
2016-10-06 15:23:11: 0: STDOUT: [Redshift] Theoretical memory bandwidth: 336.480011 GB/Sec
2016-10-06 15:23:11: 0: STDOUT: [Redshift] Measured PCIe bandwidth (pinned CPU->GPU): 4.115958 GB/s
2016-10-06 15:23:11: 0: STDOUT: [Redshift] Measured PCIe bandwidth (pinned GPU->CPU): 2.701534 GB/s
2016-10-06 15:23:11: 0: STDOUT: [Redshift] Measured PCIe bandwidth (paged CPU->GPU): 1.880550 GB/s
2016-10-06 15:23:11: 0: STDOUT: [Redshift] Measured PCIe bandwidth (paged GPU->CPU): 3.459764 GB/s
2016-10-06 15:23:11: 0: STDOUT: [Redshift] Estimated GPU->CPU latency (0): 0.008806 ms
2016-10-06 15:23:11: 0: STDOUT: [Redshift] Estimated GPU->CPU latency (1): 0.008586 ms
2016-10-06 15:23:11: 0: STDOUT: [Redshift] Estimated GPU->CPU latency (2): 0.009334 ms
2016-10-06 15:23:11: 0: STDOUT: [Redshift] Estimated GPU->CPU latency (3): 0.009009 ms
2016-10-06 15:23:11: 0: STDOUT: [Redshift] New CUDA context created
2016-10-06 15:23:11: 0: STDOUT: [Redshift] Available memory: 5517.3125 MB out of 6077.3750 MB
2016-10-06 15:23:11: 0: STDOUT: [Redshift] Initializing GPUComputing module (CUDA). Ordinal 1
2016-10-06 15:23:11: 0: STDOUT: [Redshift] CUDA Ver: 8000
2016-10-06 15:23:11: 0: STDOUT: [Redshift] Device 2/8 : GeForce GTX 980 Ti
2016-10-06 15:23:11: 0: STDOUT: [Redshift] Compute capability: 5.2
2016-10-06 15:23:11: 0: STDOUT: [Redshift] Num multiprocessors: 22
2016-10-06 15:23:11: 0: STDOUT: [Redshift] PCI busID: 5, deviceID: 0, domainID: 0
2016-10-06 15:23:11: 0: STDOUT: [Redshift] Theoretical memory bandwidth: 336.480011 GB/Sec
2016-10-06 15:23:11: 0: STDOUT: [Redshift] Measured PCIe bandwidth (pinned CPU->GPU): 8.322531 GB/s
2016-10-06 15:23:11: 0: STDOUT: [Redshift] Measured PCIe bandwidth (pinned GPU->CPU): 6.246152 GB/s
2016-10-06 15:23:11: 0: STDOUT: [Redshift] Measured PCIe bandwidth (paged CPU->GPU): 4.466307 GB/s
2016-10-06 15:23:11: 0: STDOUT: [Redshift] Measured PCIe bandwidth (paged GPU->CPU): 6.645273 GB/s
2016-10-06 15:23:11: 0: STDOUT: [Redshift] Estimated GPU->CPU latency (0): 0.008740 ms
2016-10-06 15:23:11: 0: STDOUT: [Redshift] Estimated GPU->CPU latency (1): 0.008655 ms
2016-10-06 15:23:11: 0: STDOUT: [Redshift] Estimated GPU->CPU latency (2): 0.008700 ms
2016-10-06 15:23:11: 0: STDOUT: [Redshift] Estimated GPU->CPU latency (3): 0.008707 ms
2016-10-06 15:23:11: 0: STDOUT: [Redshift] New CUDA context created
2016-10-06 15:23:11: 0: STDOUT: [Redshift] Available memory: 5592.4375 MB out of 6077.7500 MB
2016-10-06 15:23:11: 0: STDOUT: [Redshift] Initializing GPUComputing module (CUDA). Ordinal 2
2016-10-06 15:23:11: 0: STDOUT: [Redshift] CUDA Ver: 8000
2016-10-06 15:23:11: 0: STDOUT: [Redshift] Device 3/8 : GeForce GTX 980 Ti
2016-10-06 15:23:11: 0: STDOUT: [Redshift] Compute capability: 5.2
2016-10-06 15:23:11: 0: STDOUT: [Redshift] Num multiprocessors: 22
2016-10-06 15:23:11: 0: STDOUT: [Redshift] PCI busID: 8, deviceID: 0, domainID: 0
2016-10-06 15:23:11: 0: STDOUT: [Redshift] Theoretical memory bandwidth: 336.480011 GB/Sec
2016-10-06 15:23:11: 0: STDOUT: [Redshift] Measured PCIe bandwidth (pinned CPU->GPU): 4.888330 GB/s
2016-10-06 15:23:11: 0: STDOUT: [Redshift] Measured PCIe bandwidth (pinned GPU->CPU): 2.627784 GB/s
2016-10-06 15:23:12: 0: STDOUT: [Redshift] Measured PCIe bandwidth (paged CPU->GPU): 1.419169 GB/s
2016-10-06 15:23:12: 0: STDOUT: [Redshift] Measured PCIe bandwidth (paged GPU->CPU): 2.683157 GB/s
2016-10-06 15:23:12: 0: STDOUT: [Redshift] Estimated GPU->CPU latency (0): 0.008761 ms
2016-10-06 15:23:12: 0: STDOUT: [Redshift] Estimated GPU->CPU latency (1): 0.008657 ms
2016-10-06 15:23:12: 0: STDOUT: [Redshift] Estimated GPU->CPU latency (2): 0.008656 ms
2016-10-06 15:23:12: 0: STDOUT: [Redshift] Estimated GPU->CPU latency (3): 0.008868 ms
2016-10-06 15:23:12: 0: STDOUT: [Redshift] New CUDA context created
2016-10-06 15:23:12: 0: STDOUT: [Redshift] Available memory: 5592.4375 MB out of 6077.7500 MB
2016-10-06 15:23:12: 0: STDOUT: [Redshift] Initializing GPUComputing module (CUDA). Ordinal 3
2016-10-06 15:23:12: 0: STDOUT: [Redshift] CUDA Ver: 8000
2016-10-06 15:23:12: 0: STDOUT: [Redshift] Device 4/8 : GeForce GTX 980 Ti
2016-10-06 15:23:12: 0: STDOUT: [Redshift] Compute capability: 5.2
2016-10-06 15:23:12: 0: STDOUT: [Redshift] Num multiprocessors: 22
2016-10-06 15:23:12: 0: STDOUT: [Redshift] PCI busID: 9, deviceID: 0, domainID: 0
2016-10-06 15:23:12: 0: STDOUT: [Redshift] Theoretical memory bandwidth: 336.480011 GB/Sec
2016-10-06 15:23:12: 0: STDOUT: [Redshift] Measured PCIe bandwidth (pinned CPU->GPU): 3.907875 GB/s
2016-10-06 15:23:12: 0: STDOUT: [Redshift] Measured PCIe bandwidth (pinned GPU->CPU): 3.667439 GB/s
2016-10-06 15:23:12: 0: STDOUT: [Redshift] Measured PCIe bandwidth (paged CPU->GPU): 3.233031 GB/s
2016-10-06 15:23:12: 0: STDOUT: [Redshift] Measured PCIe bandwidth (paged GPU->CPU): 2.951101 GB/s
2016-10-06 15:23:12: 0: STDOUT: [Redshift] Estimated GPU->CPU latency (0): 0.008937 ms
2016-10-06 15:23:12: 0: STDOUT: [Redshift] Estimated GPU->CPU latency (1): 0.008894 ms
2016-10-06 15:23:12: 0: STDOUT: [Redshift] Estimated GPU->CPU latency (2): 0.009014 ms
2016-10-06 15:23:12: 0: STDOUT: [Redshift] Estimated GPU->CPU latency (3): 0.008823 ms
2016-10-06 15:23:12: 0: STDOUT: [Redshift] New CUDA context created
2016-10-06 15:23:12: 0: STDOUT: [Redshift] Available memory: 5590.4375 MB out of 6077.7500 MB
2016-10-06 15:23:12: 0: STDOUT: [Redshift] Initializing GPUComputing module (CUDA). Ordinal 4
2016-10-06 15:23:12: 0: STDOUT: [Redshift] CUDA Ver: 8000
2016-10-06 15:23:12: 0: STDOUT: [Redshift] Device 5/8 : GeForce GTX 980 Ti
2016-10-06 15:23:12: 0: STDOUT: [Redshift] Compute capability: 5.2
2016-10-06 15:23:12: 0: STDOUT: [Redshift] Num multiprocessors: 22
2016-10-06 15:23:12: 0: STDOUT: [Redshift] PCI busID: 131, deviceID: 0, domainID: 0
2016-10-06 15:23:12: 0: STDOUT: [Redshift] Theoretical memory bandwidth: 336.480011 GB/Sec
2016-10-06 15:23:12: 0: STDOUT: [Redshift] Measured PCIe bandwidth (pinned CPU->GPU): 8.924276 GB/s
2016-10-06 15:23:12: 0: STDOUT: [Redshift] Measured PCIe bandwidth (pinned GPU->CPU): 7.405812 GB/s
2016-10-06 15:23:12: 0: STDOUT: [Redshift] Measured PCIe bandwidth (paged CPU->GPU): 1.974471 GB/s
2016-10-06 15:23:12: 0: STDOUT: [Redshift] Measured PCIe bandwidth (paged GPU->CPU): 6.722584 GB/s
2016-10-06 15:23:12: 0: STDOUT: [Redshift] Estimated GPU->CPU latency (0): 0.008874 ms
2016-10-06 15:23:12: 0: STDOUT: [Redshift] Estimated GPU->CPU latency (1): 0.008903 ms
2016-10-06 15:23:12: 0: STDOUT: [Redshift] Estimated GPU->CPU latency (2): 0.008940 ms
2016-10-06 15:23:12: 0: STDOUT: [Redshift] Estimated GPU->CPU latency (3): 0.008812 ms
2016-10-06 15:23:12: 0: STDOUT: [Redshift] New CUDA context created
2016-10-06 15:23:12: 0: STDOUT: [Redshift] Available memory: 5588.4375 MB out of 6077.7500 MB
2016-10-06 15:23:12: 0: STDOUT: [Redshift] Initializing GPUComputing module (CUDA). Ordinal 5
2016-10-06 15:23:12: 0: STDOUT: [Redshift] CUDA Ver: 8000
2016-10-06 15:23:12: 0: STDOUT: [Redshift] Device 6/8 : GeForce GTX 980 Ti
2016-10-06 15:23:12: 0: STDOUT: [Redshift] Compute capability: 5.2
2016-10-06 15:23:12: 0: STDOUT: [Redshift] Num multiprocessors: 22
2016-10-06 15:23:12: 0: STDOUT: [Redshift] PCI busID: 132, deviceID: 0, domainID: 0
2016-10-06 15:23:12: 0: STDOUT: [Redshift] Theoretical memory bandwidth: 336.480011 GB/Sec
2016-10-06 15:23:12: 0: STDOUT: [Redshift] Measured PCIe bandwidth (pinned CPU->GPU): 10.905223 GB/s
2016-10-06 15:23:12: 0: STDOUT: [Redshift] Measured PCIe bandwidth (pinned GPU->CPU): 5.762272 GB/s
2016-10-06 15:23:13: 0: STDOUT: [Redshift] Measured PCIe bandwidth (paged CPU->GPU): 1.803028 GB/s
2016-10-06 15:23:13: 0: STDOUT: [Redshift] Measured PCIe bandwidth (paged GPU->CPU): 6.118744 GB/s
2016-10-06 15:23:13: 0: STDOUT: [Redshift] Estimated GPU->CPU latency (0): 0.009031 ms
2016-10-06 15:23:13: 0: STDOUT: [Redshift] Estimated GPU->CPU latency (1): 0.008965 ms
2016-10-06 15:23:13: 0: STDOUT: [Redshift] Estimated GPU->CPU latency (2): 0.008741 ms
2016-10-06 15:23:13: 0: STDOUT: [Redshift] Estimated GPU->CPU latency (3): 0.008769 ms
2016-10-06 15:23:13: 0: STDOUT: [Redshift] New CUDA context created
2016-10-06 15:23:13: 0: STDOUT: [Redshift] Available memory: 5588.4375 MB out of 6077.7500 MB
2016-10-06 15:23:13: 0: STDOUT: [Redshift] Initializing GPUComputing module (CUDA). Ordinal 6
2016-10-06 15:23:13: 0: STDOUT: [Redshift] CUDA Ver: 8000
2016-10-06 15:23:13: 0: STDOUT: [Redshift] Device 7/8 : GeForce GTX 980 Ti
2016-10-06 15:23:13: 0: STDOUT: [Redshift] Compute capability: 5.2
2016-10-06 15:23:13: 0: STDOUT: [Redshift] Num multiprocessors: 22
2016-10-06 15:23:13: 0: STDOUT: [Redshift] PCI busID: 135, deviceID: 0, domainID: 0
2016-10-06 15:23:13: 0: STDOUT: [Redshift] Theoretical memory bandwidth: 336.480011 GB/Sec
2016-10-06 15:23:13: 0: STDOUT: [Redshift] Measured PCIe bandwidth (pinned CPU->GPU): 10.521413 GB/s
2016-10-06 15:23:13: 0: STDOUT: [Redshift] Measured PCIe bandwidth (pinned GPU->CPU): 9.949578 GB/s
2016-10-06 15:23:13: 0: STDOUT: [Redshift] Measured PCIe bandwidth (paged CPU->GPU): 4.378646 GB/s
2016-10-06 15:23:13: 0: STDOUT: [Redshift] Measured PCIe bandwidth (paged GPU->CPU): 6.451344 GB/s
2016-10-06 15:23:13: 0: STDOUT: [Redshift] Estimated GPU->CPU latency (0): 0.008798 ms
2016-10-06 15:23:13: 0: STDOUT: [Redshift] Estimated GPU->CPU latency (1): 0.008826 ms
2016-10-06 15:23:13: 0: STDOUT: [Redshift] Estimated GPU->CPU latency (2): 0.008791 ms
2016-10-06 15:23:13: 0: STDOUT: [Redshift] Estimated GPU->CPU latency (3): 0.009372 ms
2016-10-06 15:23:13: 0: STDOUT: [Redshift] New CUDA context created
2016-10-06 15:23:13: 0: STDOUT: [Redshift] Available memory: 5588.4375 MB out of 6077.7500 MB
2016-10-06 15:23:13: 0: STDOUT: [Redshift] Initializing GPUComputing module (CUDA). Ordinal 7
2016-10-06 15:23:13: 0: STDOUT: [Redshift] CUDA Ver: 8000
2016-10-06 15:23:13: 0: STDOUT: [Redshift] Device 8/8 : GeForce GTX 980 Ti
2016-10-06 15:23:13: 0: STDOUT: [Redshift] Compute capability: 5.2
2016-10-06 15:23:13: 0: STDOUT: [Redshift] Num multiprocessors: 22
2016-10-06 15:23:13: 0: STDOUT: [Redshift] PCI busID: 136, deviceID: 0, domainID: 0
2016-10-06 15:23:13: 0: STDOUT: [Redshift] Theoretical memory bandwidth: 336.480011 GB/Sec
2016-10-06 15:23:13: 0: STDOUT: [Redshift] Measured PCIe bandwidth (pinned CPU->GPU): 9.121967 GB/s
2016-10-06 15:23:13: 0: STDOUT: [Redshift] Measured PCIe bandwidth (pinned GPU->CPU): 7.116389 GB/s
2016-10-06 15:23:13: 0: STDOUT: [Redshift] Measured PCIe bandwidth (paged CPU->GPU): 5.195380 GB/s
2016-10-06 15:23:13: 0: STDOUT: [Redshift] Measured PCIe bandwidth (paged GPU->CPU): 6.327258 GB/s
2016-10-06 15:23:13: 0: STDOUT: [Redshift] Estimated GPU->CPU latency (0): 0.008688 ms
2016-10-06 15:23:13: 0: STDOUT: [Redshift] Estimated GPU->CPU latency (1): 0.011023 ms
2016-10-06 15:23:13: 0: STDOUT: [Redshift] Estimated GPU->CPU latency (2): 0.008642 ms
2016-10-06 15:23:13: 0: STDOUT: [Redshift] Estimated GPU->CPU latency (3): 0.009500 ms
2016-10-06 15:23:13: 0: STDOUT: [Redshift] New CUDA context created
2016-10-06 15:23:13: 0: STDOUT: [Redshift] Available memory: 5588.4375 MB out of 6077.7500 MB
2016-10-06 15:23:18: 0: STDOUT: [Redshift] ======================================================================================================
2016-10-06 15:23:18: 0: STDOUT: ASSERT FAILED
2016-10-06 15:23:18: 0: STDOUT: File …/GPUComputing_CUDA.cpp
2016-10-06 15:23:18: 0: STDOUT: Line 1847
2016-10-06 15:23:18: 0: STDOUT: Failed to create 2D texture! (1024 1 - 4096 1) (CUDA_ERROR_OUT_OF_MEMORY)
2016-10-06 15:23:18: 0: STDOUT: ======================================================================================================
2016-10-06 15:23:18: 0: STDOUT: [Redshift] Call stack from …/GPUComputing_CUDA.cpp:1847:
2016-10-06 15:23:18: 0: STDOUT: [Redshift] /usr/redshift/redshift4maya/2016/…/…/bin/libredshift-core.so(+0x68f6a2) [0x7f9fa17b76a2]
2016-10-06 15:23:18: 0: STDOUT: [Redshift] /usr/redshift/redshift4maya/2016/…/…/bin/libredshift-core.so(+0xc6e2f) [0x7f9fa11eee2f]
2016-10-06 15:23:18: 0: STDOUT: [Redshift] /usr/redshift/redshift4maya/2016/…/…/bin/libredshift-core.so(+0x5c2e4b) [0x7f9fa16eae4b]
2016-10-06 15:23:18: 0: STDOUT: [Redshift] /usr/redshift/redshift4maya/2016/…/…/bin/libredshift-core.so(+0x5c50e1) [0x7f9fa16ed0e1]
2016-10-06 15:23:18: 0: STDOUT: [Redshift] /usr/redshift/redshift4maya/2016/…/…/bin/libredshift-core.so(+0x16022d) [0x7f9fa128822d]
2016-10-06 15:23:18: 0: STDOUT: [Redshift] /usr/redshift/redshift4maya/2016/…/…/bin/libredshift-core.so(+0x413f59) [0x7f9fa153bf59]
2016-10-06 15:23:18: 0: STDOUT: [Redshift] /usr/redshift/redshift4maya/2016/…/…/bin/libredshift-core.so(+0x4346c9) [0x7f9fa155c6c9]
2016-10-06 15:23:18: 0: STDOUT: [Redshift] /usr/redshift/redshift4maya/2016/…/…/bin/libredshift-core.so(_Z18RS_Renderer_CreatePKcjPi+0x75) [0x7f9fa1194b35]
2016-10-06 15:23:18: 0: STDOUT: [Redshift] /usr/redshift/redshift4maya/2016/redshift4maya.so(+0x159b26) [0x7f9fa3393b26]
2016-10-06 15:23:18: 0: STDOUT: [Redshift] /lib64/libpthread.so.0(+0x3d75e07aa1) [0x7fa026884aa1]
2016-10-06 15:23:18: 0: STDOUT: [Redshift] /lib64/libc.so.6(clone+0x6d) [0x3d75ae8aad]
2016-10-06 15:23:18: 0: STDOUT: terminate called after throwing an instance of ‘char*’
2016-10-06 15:23:18: 0: STDOUT: Stack trace:
2016-10-06 15:23:18: 0: STDOUT: /lib64/libc.so.6() [0x3d75a32660]
2016-10-06 15:23:18: 0: STDOUT: gsignal
2016-10-06 15:23:18: 0: STDOUT: abort
2016-10-06 15:23:18: 0: STDOUT: __gnu_cxx::__verbose_terminate_handler()
2016-10-06 15:23:18: 0: STDOUT: /usr/lib64/libstdc++.so.6() [0x3d80ebcbd6]
2016-10-06 15:23:18: 0: STDOUT: /usr/lib64/libstdc++.so.6() [0x3d80ebcc03]
2016-10-06 15:23:18: 0: STDOUT: /usr/lib64/libstdc++.so.6() [0x3d80ebcd22]
2016-10-06 15:23:18: 0: STDOUT: /usr/redshift/redshift4maya/2016/…/…/bin/libredshift-core.so(+0x68f704) [0x7f9fa17b7704]
2016-10-06 15:23:18: 0: STDOUT: /usr/redshift/redshift4maya/2016/…/…/bin/libredshift-core.so(+0xc6e2f) [0x7f9fa11eee2f]
2016-10-06 15:23:18: 0: STDOUT: /usr/redshift/redshift4maya/2016/…/…/bin/libredshift-core.so(+0x5c2e4b) [0x7f9fa16eae4b]
2016-10-06 15:23:18: 0: STDOUT: /usr/redshift/redshift4maya/2016/…/…/bin/libredshift-core.so(+0x5c50e1) [0x7f9fa16ed0e1]
2016-10-06 15:23:18: 0: STDOUT: /usr/redshift/redshift4maya/2016/…/…/bin/libredshift-core.so(+0x16022d) [0x7f9fa128822d]
2016-10-06 15:23:18: 0: STDOUT: /usr/redshift/redshift4maya/2016/…/…/bin/libredshift-core.so(+0x413f59) [0x7f9fa153bf59]
2016-10-06 15:23:18: 0: STDOUT: /usr/redshift/redshift4maya/2016/…/…/bin/libredshift-core.so(+0x4346c9) [0x7f9fa155c6c9]
2016-10-06 15:23:18: 0: STDOUT: RS_Renderer_Create(char const*, unsigned int, int*)
2016-10-06 15:23:18: 0: STDOUT: /usr/redshift/redshift4maya/2016/redshift4maya.so(+0x159b26) [0x7f9fa3393b26]
2016-10-06 15:23:18: 0: STDOUT: /lib64/libpthread.so.0(+0x3d75e07aa1) [0x7fa026884aa1]
2016-10-06 15:23:18: 0: STDOUT: clone
2016-10-06 15:23:18: 0: STDOUT: Result: /tmp/Thinkbox/Deadline8/root/slave/chinook2-gpu4/jobsData/57f65df4a0d4413849daacce/Chinook2-Linux-Test.ma
2016-10-06 15:23:18: 0: STDOUT: Fatal Error. Attempting to save in /usr/tmp/root.20161006.1523.ma
2016-10-06 15:23:18: 0: STDOUT: Writing crash report in /usr/tmp/root.20161006.1523.crash
2016-10-06 15:23:20: 0: WARNING: Monitored managed process MayaBatch is no longer running
2016-10-06 15:23:20: 0: Done executing plugin command of type ‘Render Task’

=======================================================
Details

Date: 10/06/2016 15:22:04
Frames: 4
Elapsed Time: 00:00:01:33
Job Submit Date: 10/06/2016 15:20:22
Job User: root
Average RAM Usage: 4314768621 (6%)
Peak RAM Usage: 10250014720 (13%)
Average CPU Usage: 13%
Peak CPU Usage: 56%
Used CPU Clocks (x10^6 cycles): 333474
Total CPU Clocks (x10^6 cycles): 2565181

=======================================================
Slave Information

Slave Name: Chinook2-GPU4
Version: v8.1.4.6 Release (e3b95748b)
Operating System: CentOS release 6.8 (Final)
Machine User: root
IP Address: 192.168.94.118
MAC Address: A0:42:3F:32:F0:04
CPU Architecture: x86_64
CPUs: 24
CPU Usage: 17%
Memory Usage: 7.6 GB / 78.6 GB (9%)
Free Disk Space: 32.090 GB
Video Card: NVIDIA Corporation GM200 [GeForce GTX 980 Ti] (rev a1)

[MikeOwen]

Hi,

Thanks for the log. Can you try this updated MayaBatch.py plugin file for me?

[Backup first in-case]

Unzip and overwrite the attached file, to this location, overwriting the same named file there already:

“<your_repo>/plugins/MayaBatch/MayaBatch.py”

and give it another whirl? (The job log should print something this time about setting GPU affinity…?)

MayaBatch.py.zip (24.1 KB)

[RyanIG]

Hi Mike,

I’ve swapped out the MayaBatch.py files and rebooted but the GPU Affinity still isn’t working,

Heres another Error log form a slave, it does look like redshift is outputting more information than the last log.

pastebin.com/kL6CMmaf

Cheers

Ryan

[MikeOwen]

Something isn’t right here. There should have been a message either way, reporting what GPU’s are or are not being used, if the GPU override is enabled and configured as per your first screen-grab you posted.

Did the file copy process go ok, copying the MayaBatch.py file onto your Deadline Repository, plugins directory, “MayaBatch” directory? Right?

Please find attached another version, which has many DEBUG print statements in it. If we don’t see at least one of these debug statements when the job next runs, then the py script is not getting copied over or being used for some other reason, which we will need to investigate.

So, same instructions as before:

Unzip and overwrite the attached file, to this location, overwriting the same named file there already:

“<your_repo>/plugins/MayaBatch/MayaBatch.py”

MayaBatch.py.zip (24.2 KB)

[RyanIG]

Hey Mike,

It seems like it’s only happening when I manually try and submit a job through Deadline, when submitting through Maya using the plugin the GPU Affinity works without issues. This will work for us as we don’t normally submit manually through Deadline and were only doing it because we were having issues submitting through Maya.

There where no errors when copying the MayaBatch.Py file, unzipped and copied to -> \LICENSE-SERVER3\Deadline8-Repo\plugins\MayaBatch which is our repository.

Chances are it is something I’m missing or doing wrong when submitting manually through Deadline.

Here are some screenshots of my manual Submission settings.
i66.tinypic.com/2uzdr35.jpg
i66.tinypic.com/2qbvsk2.png
i67.tinypic.com/29qi1yf.png
i66.tinypic.com/33o5n5i.png

Submission through Maya
pastebin.com/0Xxd3t3E

Manual Submission with Latest MayaBatch.py Script
pastebin.com/kh7dE21r

[MikeOwen]

Ah, I see. I think you just need to make sure in the Monitor submitter UI, in the first tab, make sure you select the correct “Renderer = Redshift”, then GPU rendering will be enabled. Submitting a Maya job to Deadline outside of Maya, we have no way to detect what renderer is being used, so we need the user to tell us what render engine to use. With “Redshift” enabled, then we can enable various GPU settings and we know to respect any GPU affinity override settings that might have been configured per Slave.

[RyanIG]

Hey Mike,

Had to wait for a render to finish on the system before testing again, after choosing redshift when submitting it worked without any issues, I assumed it would of picked up the information you set in Maya so that’s why I wasn’t setting it.

Everything seems to be working away happily now!

2016-10-10 17:09:03: 0: INFO: DEBUG: MIKE: redshift is the renderer
2016-10-10 17:09:03: 0: INFO: DEBUG: MIKE: Entering GetGpuOverrides function
2016-10-10 17:09:03: 0: INFO: DEBUG: MIKE: Entering Override GetGpuOverrides if statement
2016-10-10 17:09:03: 0: INFO: DEBUG: MIKE: resultGPUs is: [0, 1]
2016-10-10 17:09:03: 0: INFO: DEBUG: MIKE: selectedGPUs: [0, 1]
2016-10-10 17:09:03: 0: INFO: DEBUG: MIKE: gpus: 0,1

Cheers for your time and help!

Ryan

[MikeOwen]

Cool! Glad its all sorted for you now. Feel free to revert that MayaBatch file, as I’m sure you don’t want my name splattered all over your log reports!