Nuke Submitter Crashing

Hi,

We’re having some issues with the Deadline submitter, it seems to intermittently crash Nuke on submission. There doesn’t seem to be an obvious trigger and it’s hard to debug so I’m wondering if there’s a log from the submitter that we could see after it’s crashed or if there are any known issues? We’re using Deadline 9.0.3 and Nuke 10.0v3.

Thanks

Nick

We’ve been trying to figure that out over here actually.

Want to give us a call today and we can take a look? We were thinking it was a 10.0v5 thing, but if you’re on v3 then that’s very interesting. I’ll update the guy working on this one.

I’m not sure what we’ll be able to add on the call other than that it seems to be happening more and more. We do use Shotguns Toolkit, whether that has an impact but we’re on a serious deadline at the moment and the lead compositor is having to submit jobs manually through the Monitor because he can’t get it to work at all at the moment. Does the submitter put out any logs that I can look through?

Thanks

Nick

It’s supposed to log to Nuke’s standard scripting console, but I’ve already tested that and it hasn’t been any help. I’ll add the Shotgun angle to the dev issue.

So, here’s an interesting twist: I had a call this morning and the client mentioned that he disabled most of the plugins in his Menu.py file and that got Nuke working correctly again. He’s going to try to re-enable and test and get back to me. Unfortunately he had more pressing issues so we weren’t able to do it on the call.

Since the issue’s been building up, is there some new tool or maybe an upgrade that’s been making the rounds there?

Hi Edwin,

I don’t think there’s anything specific we’ve added. We certainly run a few plugins and set some variables in the menu.py and init.py. The shotgun toolkit write node stopped working when we upgraded to deadline 9, we couldn’t resolve it so stopped using it but I wonder if that’s linked.

Nick

Well, this customer isn’t using Shotgun but there may be a common link between whatever plugin they have loading and Shotgun. Would you be willing to try and clean Menu.py just to see if it makes a difference? We’re really just trying to focus our search at this point so we can get this reproduced in-house.

Hi Edwin,

I’m the head of comp who works with Weedy, thanks for looking into this for us.

Its very starange that the problems seem so intermitten. If it was all the time and simply not working, that would be one thing, but it just seems so random.

If I do a full machine restart it will usually (although not always) work for a few submissions, but then at some point it will fail and Nuke will hang. once its hung once, it will normally hang everytime I try a submission, untill i do a full restart.

We are running a few custom tools and scripts but nothing particularly clever, and nothing that I can imagine would cause this issue. Everything seems to be working completely normally from a user/tools point of view, its just the submissions thats failing. I can send you the menu.py and init.py if you want to check what were doing?

I’ll try and clear out the menu.py on monday morning and see if that at least gets it back to working.

Thanks

Matt

Thanks guys! I’m not sure it’s intermittent for most people, but it’s good to know.

I’m wondering if it has something to do with the thread we’re spawning. I need to get some time to read over what the submit button is actually calling, but I do remember us having problems a few years ago trying to interact with the progress bar in the UI from our Python thread. That locked things up as well.

I am looking into very similar symptoms right now. Has anyone tried testing Administrator privileges specifically?

Artist with No Local Admin submits == Nuke freeze
Me with Local Admin submits == No Nuke freeze

Let me know if you have?
Thanks!

I’ve not tried that but none of the artists have admin privileges here either.

Nick

Our studio is experiencing the same issue described in this old thread. In our case it’s intermittently crashing Nuke 11.3 on Centos 7. Best I can tell the issue seems to be related to the way the submission script is submitting each job in its own thread (as hinted by @eamsler) above. In an effort to isolated the problem, we tested a hacked copy (not in production) of the submission script which strips out all multithreading and runs submissions in the main nuke thread and we’ve not been able to repro the crash with the change.

The crashes were indeed in the cases where jobs are submitted in threads.
I debugged this for a while today, and found that the crash happens around line 1435, where
get_filename(node) is called.
This might be because the nuke api is called concurrently in this function from multiple callers.

First tried acquiring the semaphore before this call, which did not help.
A sleep(0.5) before the call seems to fix it as far as I have tested. This solution relies on luck though, so not that good.

Privacy | Site terms | Cookie preferences