I came across this thread while experiencing a similar issue in Nuke.
Turns out logging to stdout was the culprit. Use py-spy dump -p <pid> once Nuke locks up and you’ll likely see it stuck waiting on:
write (hiero/FnRedirect.py:33)
Nuke redirects sys.stdout to FnRedirect and I guess it’s not thread-safe. I’m not sure of a good work around, passing sys.__stdout__ to a StreamHandler still gets redirected to FnRedirect.
Hi Mois,
This seems to be working very nicely!
One thing I noticed, and wondering if you have the same behavior on your end: The submission dialogue no longer shows the progress of the submitted jobs. I get a popup at the end when they’ve all been submitted successfully, but it used to show the incremental progress?
Thanks,
-Jake
I think there are two scenarios, and in one there is no proper notification. It should be fixed, but I guess it’s not worth the time investment.
@mois.moshev
Curious if you are still using the same fixes in the latest iteration of the nuke submitter? I haven’t updated since 10.2.1.1.
Does anyone have an updated script for this. I had this working once upon a time and when updating to the latest deadline I guess it got wiped out. Would anyone be able to tell me where SubmitNukeToDeadline.py goes? Or maybe I do have this right and its just not working. My guess is this submitter doesn’t work with current versions of deadline.
The customized scripts usually go into the custom folders, so this script would be located:
\\your-deadline-repo\custom\submission\Nuke\Main\SubmitNukeToDeadline.py
and if the folders “submission, Nuke, and Main” don’t exist, just make them and then drop that py file in there.
Anything in the “custom” folder area should not be touched when upgrading deadline.
What version of Deadline are you using?
There must be a comparability issue. This is the returned error…
Traceback (most recent call last):
File “C:\Users/Chris/.nuke\DeadlineNukeClient.py”, line 91, in main
import SubmitNukeToDeadline
File “C:\Program Files\Nuke14.1v3\pythonextensions\site-packages\shiboken2\files.dir\shibokensupport_feature_.py”, line 142, in import
return original_import(name, *args, **kwargs)
File “//DSVFXSERVE/DeadlineRepository10\custom/submission/Nuke/Main\SubmitNukeToDeadline.py”, line 29, in
import DeadlineGlobals
File "C:\Program Files\Nuke14.1v3\pythonextensions\site-packages\shiboken2\files.dir\shibokensupport_feature.py", line 142, in import
return original_import(name, *args, **kwargs)
ModuleNotFoundError: No module named ‘DeadlineGlobals’
The SubmitNukeToDeadline.py failed with the following error message. Please make sure that the Deadline Client has been installed on this machine, that the Deadline Client bin folder is set in the DEADLINE_PATH environment variable, and that the Deadline Client has been configured to point to a valid Repository.
Traceback (most recent call last):
File “C:\Users/Chris/.nuke\DeadlineNukeClient.py”, line 91, in main
import SubmitNukeToDeadline
File "C:\Program Files\Nuke14.1v3\pythonextensions\site-packages\shiboken2\files.dir\shibokensupport_feature.py", line 142, in import
return original_import(name, *args, **kwargs)
File “//DSVFXSERVE/DeadlineRepository10\custom/submission/Nuke/Main\SubmitNukeToDeadline.py”, line 29, in
import DeadlineGlobals
File "C:\Program Files\Nuke14.1v3\pythonextensions\site-packages\shiboken2\files.dir\shibokensupport_feature.py", line 142, in _import
return original_import(name, *args, **kwargs)
ModuleNotFoundError: No module named ‘DeadlineGlobals’
Which version of Deadline are you using? The default SubmitNukeToDeadline.py that comes with DL 10.3.0.09 (or 15) incorporated a lot of the fixes that users had with earlier versions.
You can re-name the custom script to SubmitNukeToDeadline.py.bak and Deadline should ignore it and use the vanilla/default installed one – does that work?
And maybe check that env var DEADLINE_PATH is resolve-able – as per your error message.
Repository Version: 10.3.2.1
That’s the version. It’s the same issue we’ve always had on any deadline version. The patched version from @mois.moshev worked great and now I just can’t get it to work here.
The vanilla Deadline submitter works fine, it just crashes 1/4 times you use it. It only seems to crash when multiple sessions of nuke are open. Might be the localization happening and the fact that other nuke sessions are constantly doing calls to check for updated localization files. Maybe it’s another multithreading issue.
My bad. In addition to creating the folder, you should also copy the \\your-repo\submission\Nuke\Main\DeadlineGlobals.py file into the custom\submission\Nuke\Main\ folder.
No beuno sadly, same errors.
Here is the version we use - it contains my patches and newer changes from the Deadline repo.
SubmitNukeToDeadline.py (125.4 KB)
The tldr is that the vanilla submitter has its threading all wrong.
@mois.moshev that makes such a difference. Thank you sooo much. There’s still a crash occasionally, but we’re down from 25% to 5% of the time. Again I greatly appreciate this.
If anyone from Deadline is following this thread I’m pretty sure Nuke’s localization going on in the background is the source of the issue. Multiple nuke sessions have to be open and I’m sure nuke is doing calls to check for new files.
I think we have locked it down to an OFX plugin called reduce noise. This is one of the most common OFX nodes out there. Disabling it has reduced crashing by a very large %. I’m not sure if the plugin itself was causing the crash which was 1 in 3 tries or whether the license for reduce noise is floating and there’s a call back to the license server.
Hello!
The Nuke runs the DeadlineNukeClient.py file when you click the Submit Nuke To Deadline button. It try finds the path the submission/Nuke/Main running deadlinecommand -GetRepositoryPath to return the path to the Nuke submitter folder. The -GetRepositoryPath files find the files in the /custom/submission folder first and then look for the default folder under the repository submission/Nuke. So if you have the submitter file for Nuke under the custom folder will get prioritization. You can run the below command to print the submitter path:
deadlinecommand -GetRepositoryPath submission\Nuke\Main
It add that to the system paths of the Workstation and load the SubmitNukeToDeadline module from the Main folder. This is the breakdown of how the Nuke submitter works.
Thank you for your reply. 90% of the submissions to the farm things work great. Once the first submit to deadline crash happens, which is after we hit the submit button, the rate of future crash starts to go up. We are testing now if closing all nuke operations helps this. Each computers system variables has C:\Program Files\Thinkbox\Deadline10\bin mapped as the DEADLINE_PATH. We’ve put the custom submitter in the repository custom folder on the license server we have at C:\DeadlineRepository10\custom\submission\Submission\Nuke\Main. Is the above correct or does each workstation require us to put the custom submitter on each workstation manually or is having it on the main repository server enough?