The crashes were indeed in the cases where jobs are submitted in threads.
I debugged this for a while today, and found that the crash happens around line 1435, where get_filename(node) is called.
This might be because the nuke api is called concurrently in this function from multiple callers.
First tried acquiring the semaphore before this call, which did not help.
A sleep(0.5) before the call seems to fix it as far as I have tested. This solution relies on luck though, so not that good.
Yes, it is random, because it depends on how the separate threads run, and whether they happen to concurrently use the nuke api.
By the way there is a way to force the submission to run on the main thread, by checking “Set Dependencies Based on Write Node Render Order”. This should reduce the crashes greatly.
Otherwise I do not know if this is being worked on.
Hello Nico,
We are struggling against this problem as well. Would it be possible to share your solution/ .py file with the multithreading stripped out?
Thankyou,
-Jake
Hi @JakeS . Happy to help out. I haven’t looked at this code in several years so I’m afraid I’ll be pretty ineffective at fielding questions, but please find our solution attached.SubmitNukeToDeadline.py (121.2 KB)
Best of luck, let us know if it solves your issue.
Note that Nico’s version will not submit for certain parameters, e.g. if submitting as separate jobs, but without job dependencies, it will do nothing.
The main cause for crashes is the multithreaded submission, which is triggered for certain combinations of parameters:
if separate jobs, but separate jobs dependencies is off
if separate tasks
if not separate jobs
(I hope I got these right)
Here is my take, attached. It tries to fix the threading issues (while still using threads), and also contains some fixes for Python 3 (Nuke 13 onward) SubmitNukeToDeadline.py (122.1 KB)
I came across this thread while experiencing a similar issue in Nuke.
Turns out logging to stdout was the culprit. Use py-spy dump -p <pid> once Nuke locks up and you’ll likely see it stuck waiting on:
write (hiero/FnRedirect.py:33)
Nuke redirects sys.stdout to FnRedirect and I guess it’s not thread-safe. I’m not sure of a good work around, passing sys.__stdout__ to a StreamHandler still gets redirected to FnRedirect.
Hi Mois,
This seems to be working very nicely!
One thing I noticed, and wondering if you have the same behavior on your end: The submission dialogue no longer shows the progress of the submitted jobs. I get a popup at the end when they’ve all been submitted successfully, but it used to show the incremental progress?
Thanks,
-Jake
Does anyone have an updated script for this. I had this working once upon a time and when updating to the latest deadline I guess it got wiped out. Would anyone be able to tell me where SubmitNukeToDeadline.py goes? Or maybe I do have this right and its just not working. My guess is this submitter doesn’t work with current versions of deadline.
There must be a comparability issue. This is the returned error…
Traceback (most recent call last):
File “C:\Users/Chris/.nuke\DeadlineNukeClient.py”, line 91, in main
import SubmitNukeToDeadline
File “C:\Program Files\Nuke14.1v3\pythonextensions\site-packages\shiboken2\files.dir\shibokensupport_feature_.py”, line 142, in import
return original_import(name, *args, **kwargs)
File “//DSVFXSERVE/DeadlineRepository10\custom/submission/Nuke/Main\SubmitNukeToDeadline.py”, line 29, in
import DeadlineGlobals
File "C:\Program Files\Nuke14.1v3\pythonextensions\site-packages\shiboken2\files.dir\shibokensupport_feature.py", line 142, in import
return original_import(name, *args, **kwargs)
ModuleNotFoundError: No module named ‘DeadlineGlobals’
The SubmitNukeToDeadline.py failed with the following error message. Please make sure that the Deadline Client has been installed on this machine, that the Deadline Client bin folder is set in the DEADLINE_PATH environment variable, and that the Deadline Client has been configured to point to a valid Repository.
Traceback (most recent call last):
File “C:\Users/Chris/.nuke\DeadlineNukeClient.py”, line 91, in main
import SubmitNukeToDeadline
File "C:\Program Files\Nuke14.1v3\pythonextensions\site-packages\shiboken2\files.dir\shibokensupport_feature.py", line 142, in import
return original_import(name, *args, **kwargs)
File “//DSVFXSERVE/DeadlineRepository10\custom/submission/Nuke/Main\SubmitNukeToDeadline.py”, line 29, in
import DeadlineGlobals
File "C:\Program Files\Nuke14.1v3\pythonextensions\site-packages\shiboken2\files.dir\shibokensupport_feature.py", line 142, in _import
return original_import(name, *args, **kwargs)
ModuleNotFoundError: No module named ‘DeadlineGlobals’
Which version of Deadline are you using? The default SubmitNukeToDeadline.py that comes with DL 10.3.0.09 (or 15) incorporated a lot of the fixes that users had with earlier versions.
You can re-name the custom script to SubmitNukeToDeadline.py.bak and Deadline should ignore it and use the vanilla/default installed one – does that work?
And maybe check that env var DEADLINE_PATH is resolve-able – as per your error message.
That’s the version. It’s the same issue we’ve always had on any deadline version. The patched version from @mois.moshev worked great and now I just can’t get it to work here.
The vanilla Deadline submitter works fine, it just crashes 1/4 times you use it. It only seems to crash when multiple sessions of nuke are open. Might be the localization happening and the fact that other nuke sessions are constantly doing calls to check for updated localization files. Maybe it’s another multithreading issue.
My bad. In addition to creating the folder, you should also copy the \\your-repo\submission\Nuke\Main\DeadlineGlobals.py file into the custom\submission\Nuke\Main\ folder.