AWS Thinkbox Discussion Forums

Nuke Submitter Crashing

I’m not sure what we’ll be able to add on the call other than that it seems to be happening more and more. We do use Shotguns Toolkit, whether that has an impact but we’re on a serious deadline at the moment and the lead compositor is having to submit jobs manually through the Monitor because he can’t get it to work at all at the moment. Does the submitter put out any logs that I can look through?

Thanks

Nick

It’s supposed to log to Nuke’s standard scripting console, but I’ve already tested that and it hasn’t been any help. I’ll add the Shotgun angle to the dev issue.

So, here’s an interesting twist: I had a call this morning and the client mentioned that he disabled most of the plugins in his Menu.py file and that got Nuke working correctly again. He’s going to try to re-enable and test and get back to me. Unfortunately he had more pressing issues so we weren’t able to do it on the call.

Since the issue’s been building up, is there some new tool or maybe an upgrade that’s been making the rounds there?

Hi Edwin,

I don’t think there’s anything specific we’ve added. We certainly run a few plugins and set some variables in the menu.py and init.py. The shotgun toolkit write node stopped working when we upgraded to deadline 9, we couldn’t resolve it so stopped using it but I wonder if that’s linked.

Nick

Well, this customer isn’t using Shotgun but there may be a common link between whatever plugin they have loading and Shotgun. Would you be willing to try and clean Menu.py just to see if it makes a difference? We’re really just trying to focus our search at this point so we can get this reproduced in-house.

Hi Edwin,

I’m the head of comp who works with Weedy, thanks for looking into this for us.

Its very starange that the problems seem so intermitten. If it was all the time and simply not working, that would be one thing, but it just seems so random.

If I do a full machine restart it will usually (although not always) work for a few submissions, but then at some point it will fail and Nuke will hang. once its hung once, it will normally hang everytime I try a submission, untill i do a full restart.

We are running a few custom tools and scripts but nothing particularly clever, and nothing that I can imagine would cause this issue. Everything seems to be working completely normally from a user/tools point of view, its just the submissions thats failing. I can send you the menu.py and init.py if you want to check what were doing?

I’ll try and clear out the menu.py on monday morning and see if that at least gets it back to working.

Thanks

Matt

Thanks guys! I’m not sure it’s intermittent for most people, but it’s good to know.

I’m wondering if it has something to do with the thread we’re spawning. I need to get some time to read over what the submit button is actually calling, but I do remember us having problems a few years ago trying to interact with the progress bar in the UI from our Python thread. That locked things up as well.

I am looking into very similar symptoms right now. Has anyone tried testing Administrator privileges specifically?

Artist with No Local Admin submits == Nuke freeze
Me with Local Admin submits == No Nuke freeze

Let me know if you have?
Thanks!

I’ve not tried that but none of the artists have admin privileges here either.

Nick

Our studio is experiencing the same issue described in this old thread. In our case it’s intermittently crashing Nuke 11.3 on Centos 7. Best I can tell the issue seems to be related to the way the submission script is submitting each job in its own thread (as hinted by @eamsler) above. In an effort to isolated the problem, we tested a hacked copy (not in production) of the submission script which strips out all multithreading and runs submissions in the main nuke thread and we’ve not been able to repro the crash with the change.

The crashes were indeed in the cases where jobs are submitted in threads.
I debugged this for a while today, and found that the crash happens around line 1435, where
get_filename(node) is called.
This might be because the nuke api is called concurrently in this function from multiple callers.

First tried acquiring the semaphore before this call, which did not help.
A sleep(0.5) before the call seems to fix it as far as I have tested. This solution relies on luck though, so not that good.

has there been an update on this issue ?
we cannot submit more than one write node at a time in Nuke because of this… it is very frustating.

btw the issue is very random : sometimes for the same scene all writes node will be submitted, or only one, or a couple before crashing

Yes, it is random, because it depends on how the separate threads run, and whether they happen to concurrently use the nuke api.
By the way there is a way to force the submission to run on the main thread, by checking “Set Dependencies Based on Write Node Render Order”. This should reduce the crashes greatly.

Otherwise I do not know if this is being worked on.

Hello Nico,
We are struggling against this problem as well. Would it be possible to share your solution/ .py file with the multithreading stripped out?
Thankyou,
-Jake

Hi @JakeS . Happy to help out. I haven’t looked at this code in several years so I’m afraid I’ll be pretty ineffective at fielding questions, but please find our solution attached.SubmitNukeToDeadline.py (121.2 KB)
Best of luck, let us know if it solves your issue.

1 Like

Much Thanks @Nico_B_Vandenbosch !
I updated the latest submitter from 10.1.20.2 with the bits that you had modified and it appears all is good!
-Jake

Note that Nico’s version will not submit for certain parameters, e.g. if submitting as separate jobs, but without job dependencies, it will do nothing.

The main cause for crashes is the multithreaded submission, which is triggered for certain combinations of parameters:

  • if separate jobs, but separate jobs dependencies is off
  • if separate tasks
  • if not separate jobs
    (I hope I got these right)

Here is my take, attached. It tries to fix the threading issues (while still using threads), and also contains some fixes for Python 3 (Nuke 13 onward)
SubmitNukeToDeadline.py (122.1 KB)

Thanks Mois!
Much appreciated. I’ll give this method a try as well and see how it goes.
-Jake

I came across this thread while experiencing a similar issue in Nuke.

Turns out logging to stdout was the culprit. Use py-spy dump -p <pid> once Nuke locks up and you’ll likely see it stuck waiting on:

write (hiero/FnRedirect.py:33)

Nuke redirects sys.stdout to FnRedirect and I guess it’s not thread-safe. I’m not sure of a good work around, passing sys.__stdout__ to a StreamHandler still gets redirected to FnRedirect.

1 Like

Hi Mois,
This seems to be working very nicely!
One thing I noticed, and wondering if you have the same behavior on your end: The submission dialogue no longer shows the progress of the submitted jobs. I get a popup at the end when they’ve all been submitted successfully, but it used to show the incremental progress?
Thanks,
-Jake

Privacy | Site terms | Cookie preferences