RS on FARM ERROR - ASSERT FAILED

Cihan_Kenar · September 6, 2021, 1:10am

Can anyone make sense of this error message?

=======================================================

Error

=======================================================

Error: Renderer returned non-zero error code, -536870911. Check the log for more information.

at Deadline.Plugins.PluginWrapper.RenderTasks(Task task, String& outMessage, AbortLevel& abortLevel)

=======================================================

Type

=======================================================

RenderPluginException

=======================================================

Stack Trace

=======================================================

at Deadline.Plugins.SandboxedPlugin.d(DeadlineMessage brn, CancellationToken bro)

at Deadline.Plugins.SandboxedPlugin.RenderTask(Task task, CancellationToken cancellationToken)

at Deadline.Slaves.SlaveRenderThread.c(TaskLogWriter aiw, CancellationToken aix)

=======================================================

Log

=======================================================

2021-09-06 11:04:35: 0: Loading Job’s Plugin timeout is Disabled

2021-09-06 11:04:35: 0: SandboxedPlugin: Render Job As User disabled, running as current user ‘administrator’

2021-09-06 11:04:37: 0: nogui was not added to the CommandLineParser.

2021-09-06 11:04:37: 0: Executing plugin command of type ‘Initialize Plugin’

2021-09-06 11:04:37: 0: INFO: Executing plugin script ‘C:\ProgramData\Thinkbox\Deadline10\workers\wks024-gpu-2\plugins\613568e85d99370778ad4fb4\Redshift.py’

2021-09-06 11:04:37: 0: INFO: About: Redshift Plugin for Deadline

2021-09-06 11:04:37: 0: INFO: The job’s environment will be merged with the current environment before rendering

2021-09-06 11:04:37: 0: Done executing plugin command of type ‘Initialize Plugin’

2021-09-06 11:04:37: 0: Start Job timeout is disabled.

2021-09-06 11:04:37: 0: Task timeout is disabled.

2021-09-06 11:04:37: 0: Loaded job: TURNTABLE.v051- Redshift- /out/VISTALINER (613568e85d99370778ad4fb4)

2021-09-06 11:04:38: 0: Executing plugin command of type ‘Start Job’

2021-09-06 11:04:38: 0: DEBUG: S3BackedCache Client is not installed.

2021-09-06 11:04:38: 0: INFO: Executing global asset transfer preload script ‘C:\ProgramData\Thinkbox\Deadline10\workers\wks024-gpu-2\plugins\613568e85d99370778ad4fb4\GlobalAssetTransferPreLoad.py’

2021-09-06 11:04:38: 0: INFO: Looking for legacy (pre-10.0.26) AWS Portal File Transfer…

2021-09-06 11:04:38: 0: INFO: Looking for legacy (pre-10.0.26) File Transfer controller in C:/Program Files/Thinkbox/S3BackedCache/bin/task.py…

2021-09-06 11:04:38: 0: INFO: Could not find legacy (pre-10.0.26) AWS Portal File Transfer.

2021-09-06 11:04:38: 0: INFO: Legacy (pre-10.0.26) AWS Portal File Transfer is not installed on the system.

2021-09-06 11:04:38: 0: Done executing plugin command of type ‘Start Job’

2021-09-06 11:04:38: 0: Plugin rendering frame(s): 74

2021-09-06 11:04:38: 0: Executing plugin command of type ‘Render Task’

2021-09-06 11:04:38: 0: INFO: Stdout Redirection Enabled: True

2021-09-06 11:04:38: 0: INFO: Stdout Handling Enabled: True

2021-09-06 11:04:38: 0: INFO: Popup Handling Enabled: True

2021-09-06 11:04:38: 0: INFO: QT Popup Handling Enabled: False

2021-09-06 11:04:38: 0: INFO: WindowsForms10.Window.8.app.* Popup Handling Enabled: False

2021-09-06 11:04:38: 0: INFO: Using Process Tree: True

2021-09-06 11:04:38: 0: INFO: Hiding DOS Window: True

2021-09-06 11:04:38: 0: INFO: Creating New Console: False

2021-09-06 11:04:38: 0: INFO: Running as user: administrator

2021-09-06 11:04:38: 0: INFO: Executable: “Y:\pipeline\apps\redshift\3.0.50\bin\redshiftCmdLine.exe”

2021-09-06 11:04:38: 0: INFO: The Worker is overriding the GPUs to render, so the following GPUs will be used: 2

2021-09-06 11:04:38: 0: INFO: Argument: “Y:\projects\kat_king\assets\env\MGM_day\lyt\work\houdini\hip…\temp\rs_render_proxies\TURNTABLE\VISTALINER\v051\TURNTABLE__VISTALINER.v051.0074.rs” -oip “Y:\projects\kat_king\assets\env\MGM_day\lyt\work\houdini\hip…\out\render\TURNTABLE\VISTALINER\v051”

2021-09-06 11:04:38: 0: INFO: Full Command: “Y:\pipeline\apps\redshift\3.0.50\bin\redshiftCmdLine.exe” “Y:\projects\kat_king\assets\env\MGM_day\lyt\work\houdini\hip…\temp\rs_render_proxies\TURNTABLE\VISTALINER\v051\TURNTABLE__VISTALINER.v051.0074.rs” -oip “Y:\projects\kat_king\assets\env\MGM_day\lyt\work\houdini\hip…\out\render\TURNTABLE\VISTALINER\v051”

2021-09-06 11:04:38: 0: INFO: Startup Directory: “Y:\pipeline\apps\redshift\3.0.50\bin”

2021-09-06 11:04:38: 0: INFO: Process Priority: BelowNormal

2021-09-06 11:04:38: 0: INFO: Process Affinity: default

2021-09-06 11:04:38: 0: INFO: Process is now running

2021-09-06 11:04:42: 0: STDOUT: Redshift Command-Line Renderer (version 3.0.50 - API: 3028)

2021-09-06 11:04:42: 0: STDOUT: Querying texture cache buget from preferences.xml: 32 GB

2021-09-06 11:04:42: 0: STDOUT: Querying cache path from preferences.xml: %LOCALAPPDATA%\Redshift\Cache

2021-09-06 11:04:42: 0: STDOUT: No GPUs were selected in the command line, using selected compute devices from preferences.

2021-09-06 11:04:42: 0: STDOUT: Creating cache path C:\Users\administrator\AppData\Local\Redshift\Cache

2021-09-06 11:04:42: 0: STDOUT: Enforcing shader cache budget…

2021-09-06 11:04:42: 0: STDOUT: Enforcing texture cache budget…

2021-09-06 11:04:42: 0: STDOUT: Collecting files…

2021-09-06 11:04:42: 0: STDOUT: Total size for 707 files 14558.92MB (budget 32768.00MB)

2021-09-06 11:04:42: 0: STDOUT: Under budget. Done.

2021-09-06 11:04:42: 0: STDOUT: Creating mesh cache…

2021-09-06 11:04:42: 0: STDOUT: Done

2021-09-06 11:04:42: 0: STDOUT: Overriding GPU devices due to REDSHIFT_GPUDEVICES (2)

2021-09-06 11:04:42: 0: STDOUT: Redshift Initialized

2021-09-06 11:04:42: 0: STDOUT: Version: 3.0.50, Jul 15 2021

2021-09-06 11:04:42: 0: STDOUT: Windows Platform (Windows 10 Pro)

2021-09-06 11:04:42: 0: STDOUT: Release Build

2021-09-06 11:04:42: 0: STDOUT: Number of CPU HW threads: 32

2021-09-06 11:04:42: 0: STDOUT: CPU speed: 3.40 GHz

2021-09-06 11:04:42: 0: STDOUT: Total system memory: 127.94 GB

2021-09-06 11:04:42: 0: STDOUT: TDR delay: 60s

2021-09-06 11:04:42: 0: STDOUT: Hardware-accelerated GPU scheduling enabled

2021-09-06 11:04:42: 0: STDOUT: Driver version: 471.41

2021-09-06 11:04:42: 0: STDOUT: Current working dir: Y:\pipeline\apps\redshift\3.0.50\bin

2021-09-06 11:04:42: 0: STDOUT: redshift_LICENSE=5055@license.future.associate

2021-09-06 11:04:42: 0: STDOUT: Creating CUDA contexts

2021-09-06 11:04:42: 0: STDOUT: CUDA init ok

2021-09-06 11:04:42: 0: STDOUT: Ordinals: { 2 }

2021-09-06 11:04:42: 0: STDOUT: ======================================================================================================

2021-09-06 11:04:42: 0: STDOUT: ASSERT FAILED

2021-09-06 11:04:42: 0: STDOUT: File GPUComputing_CUDA.cpp

2021-09-06 11:04:42: 0: STDOUT: Line 695

2021-09-06 11:04:42: 0: STDOUT: Failed to create auxilliary CUDA context (CUDA_ERROR_INVALID_DEVICE)

2021-09-06 11:04:42: 0: STDOUT: ======================================================================================================

2021-09-06 11:04:42: 0: INFO: Process exit code: -536870911

2021-09-06 11:04:42: 0: Done executing plugin command of type ‘Render Task’

=======================================================

Details

=======================================================

Date: 09/06/2021 11:02:54

Frames: 74

Elapsed Time: 00:00:01:19

Job Submit Date: 09/06/2021 11:01:31

Job User: cihan.kenar

Average RAM Usage: 5842608128 (5%)

Peak RAM Usage: 5842608128 (5%)

Average CPU Usage: 4%

Peak CPU Usage: 7%

Used CPU Clocks (x10^6 cycles): 37014

Total CPU Clocks (x10^6 cycles): 925335

=======================================================

Worker Information

=======================================================

Worker Name: wks024-gpu-2

Version: v10.1.11.5 Release (a9bdcb38a)

Operating System: Windows 10 Pro

Running As Service: No

Machine User: administrator

IP Address: 10.0.4.83

MAC Address: 18:C0:4D:3A:77:67

CPU Architecture: x64

CPUs: 32

CPU Usage: 6%

Memory Usage: 5.4 GB / 127.9 GB (4%)

Free Disk Space: 774.679 GB

Video Card: Microsoft Remote Display Adapter

antoinedurr · September 6, 2021, 2:04am

I don’t think that’s a Deadline issue, that’s a Reshift/nVidia/CUDA issue. When assertions fail, that’s the programmers’ way of saying “hey, some input I was expecting isn’t as expected, I give up!”.

That’s not to say that there isn’t anything you can do about it, but without knowing what line 695 of GPUComputing_CUDA.cpp is asserting, it’s gonna be a bit tricky to figure out without Maxon’s help.

The other part of the error is

2021-09-06 11:04:42: 0: STDOUT: Failed to create auxilliary CUDA context (CUDA_ERROR_INVALID_DEVICE)

Googling ‘CUDA_ERROR_INVALID_DEVICE’ shows that maybe reinstalling the Nvidia drivers helps. That’s about as much as I can offer.

– Antoine

Cihan_Kenar · September 6, 2021, 4:40am

Hi Antoine,

Thanks for the speedy reply,

this will give me some actions I can take to investigate further.

Regards,

Cihan