Deadline Worker + Octane Render plugin for 3ds Max results in Windows 11 OS freeze

Deadline
1

We’re in the process of migrating to Windows 11 on our systems and unfortunately that’s where we began experiencing an issue where rendering with Octane through Deadline causes Windows 11 OS to freeze and stop rendering. Closing the DL Worker unfreezes the OS immediately. This only occurs during rendering jobs dispatched through Deadline — not when rendering natively through 3ds Max with the same scene and Octane plugin.

My leading suspicion is WDDM 3.x. and how that could interface with GPU accelerated processes via Deadline. As Win 10 is largely the same environment as far as I can tell, except for running WDDM 2.7 and that has no problems at all.

But my knowledge is lacking here.

We have exhausted our options and switched back to Windows 10 for the time being while we try to find a solution.

Here is a summary of the troubleshooting and mitigation steps we have taken so far:

Issue Summary

  • When rendering with 3ds Max + Octane through Deadline Worker on Windows 11, the system freezes shortly after geometry processing finishes and rendering begins.

  • Only the Deadline Worker window remains interactive; the rest of the OS becomes unresponsive, including remote access (RDP).

  • When closing the Worker window, the system immediately returns to normal.

  • Same Issue occurs across 5x tested workstations and servers.

  • This issue does not occur on Windows 10.

Troubleshooting Steps Taken

General Tests:

  • Verified issue only happens with Octane, not with Scanline or other render engines.

  • Confirmed rendering works fine outside Deadline (native 3ds Max + Octane).

  • Updated and rolled back NVIDIA drivers across several versions.

  • Disabled Windows Update services and PushToInstall to rule out interference.

  • Monitored logs and system behavior during render freezes.

  • Windows Event Viewer shows no relevant data either.

  • Logs do not show any error, they merely freeze in the middle of a task.

Deadline Configuration:

  • Switched between running Worker in standard mode and as a Windows Service.

  • Set the OCTANE_NOGUI=1 environment variable to avoid GUI-related hangs in Octane.

  • Checked “Restart Worker if it stalls” setting — though it does not recover from the stall/freeze condition.

  • Compared stability across different Deadline versions including 10.4.0.10 and 10.4.1.9.

OS & Driver Considerations:

  • Rolled machines back to Windows 10 (WDDM 2.7) and confirmed stable rendering with Deadline, 3ds Max + Octane.

  • Noted that instability correlates with systems running WDDM 3.x (Windows 11).

  • Currently using Windows 10 as a temporary mitigation while evaluating root cause.

Any ideas on if there are known solutions to this issue? Or things to try next?

Apologies for the long message, really hope someone can help find the missing piece to the puzzle here!

Thank you in advance!