Nuke submission broken with deadline 8 upgrade

christianl · June 14, 2016, 3:48am

Hey guys, I just upgraded to deadline 8 and it appears that my nuke submission is now broken. when I go to launch the submitter I get this error now:

And here is the traceback from the python console:

Appending “C:/ProgramData/Thinkbox/Deadline8/cache/EPMuvp1PEQO1QZhdxjdZUtdqJDI/submission/Nuke/Main” to system path to import SubmitNukeToDeadline module
Reading sticky settings from C:\Users\Administrator\AppData\Local\Thinkbox\Deadline8/settings/nuke_py_submission.ini
Found a total of 1 write nodes
Found a total of 0 write nodes within precomp nodes
Creating submission dialog…
Failed to read in configuration file Error: One or more errors occurred. (System.AggregateException)\formats.txt.
Traceback (most recent call last):
File “C:\Users\Administrator.nuke\DeadlineNukeClient.py”, line 62, in main
SubmitNukeToDeadline.SubmitToDeadline( )
File “C:/ProgramData/Thinkbox/Deadline8/cache/EPMuvp1PEQO1QZhdxjdZUtdqJDI/submission/Nuke/Main\SubmitNukeToDeadline.py”, line 2907, in SubmitToDeadline
dialog = DeadlineDialog( maximumPriority, pools, secondaryPools, groups )
File “C:/ProgramData/Thinkbox/Deadline8/cache/EPMuvp1PEQO1QZhdxjdZUtdqJDI/submission/Nuke/Main\SubmitNukeToDeadline.py”, line 66, in init
self.ReadInDraftOptions()
File “C:/ProgramData/Thinkbox/Deadline8/cache/EPMuvp1PEQO1QZhdxjdZUtdqJDI/submission/Nuke/Main\SubmitNukeToDeadline.py”, line 535, in ReadInDraftOptions
Formats = self.ReadInFormatsFile( os.path.join( mainDraftFolder, “formats.txt” ) )
File “C:/ProgramData/Thinkbox/Deadline8/cache/EPMuvp1PEQO1QZhdxjdZUtdqJDI/submission/Nuke/Main\SubmitNukeToDeadline.py”, line 557, in ReadInFormatsFile
raise Exception(errorMsg)
Exception: Failed to read in configuration file Error: One or more errors occurred. (System.AggregateException)\formats.txt.

“C:/ProgramData/Thinkbox/Deadline8/cache/EPMuvp1PEQO1QZhdxjdZUtdqJDI/submission/Nuke/Main” is already in the system path
Reading sticky settings from C:\Users\Administrator\AppData\Local\Thinkbox\Deadline8/settings/nuke_py_submission.ini
Found a total of 1 write nodes
Found a total of 0 write nodes within precomp nodes
Creating submission dialog…
Failed to read in configuration file Error: One or more errors occurred. (System.AggregateException)\formats.txt.
Traceback (most recent call last):
File “C:\Users\Administrator.nuke\DeadlineNukeClient.py”, line 62, in main
SubmitNukeToDeadline.SubmitToDeadline( )
File “C:/ProgramData/Thinkbox/Deadline8/cache/EPMuvp1PEQO1QZhdxjdZUtdqJDI/submission/Nuke/Main\SubmitNukeToDeadline.py”, line 2907, in SubmitToDeadline
dialog = DeadlineDialog( maximumPriority, pools, secondaryPools, groups )
File “C:/ProgramData/Thinkbox/Deadline8/cache/EPMuvp1PEQO1QZhdxjdZUtdqJDI/submission/Nuke/Main\SubmitNukeToDeadline.py”, line 66, in init
self.ReadInDraftOptions()
File “C:/ProgramData/Thinkbox/Deadline8/cache/EPMuvp1PEQO1QZhdxjdZUtdqJDI/submission/Nuke/Main\SubmitNukeToDeadline.py”, line 535, in ReadInDraftOptions
Formats = self.ReadInFormatsFile( os.path.join( mainDraftFolder, “formats.txt” ) )
File “C:/ProgramData/Thinkbox/Deadline8/cache/EPMuvp1PEQO1QZhdxjdZUtdqJDI/submission/Nuke/Main\SubmitNukeToDeadline.py”, line 557, in ReadInFormatsFile
raise Exception(errorMsg)
Exception: Failed to read in configuration file Error: One or more errors occurred. (System.AggregateException)\formats.txt.

I checked out this text file called formats in the draft folder of the repository but I did not see anything wrong with it or any reason why python can’t read it in, so I thought I would defer this to you guys to see if you had any ideas.

Thanks for checking into this.

Cheers,
Christian

kdronez · June 14, 2016, 7:01am

Hi,

we had the same problem which we managed to fix by changing manually the envirement to dl8 bin .We have our init pointing to folder on the server where all our custom scripts are and where dl8 nuke client should be (for us) so that could be another fix to the problem if you are using simillar setup :>.

Hope this helps.

Veselin Georgiev Gyurov.

dwallbridge · June 14, 2016, 7:19pm

Hello,

Can I have you verify that you ran the Deadline 8 integrated submitter installer since installing the Deadline 8 client apps on the machine? That should hopefully give you some better results.

christianl · June 14, 2016, 7:54pm

Hey guys, Thanks for all the answers. I solved the problem by upgrading the machines from python 2.7 to 3.4. As soon as I did that all the submission scripts worked (after effects and Nuke specifically).

-Christian

christianl · June 17, 2016, 6:17am

ok, looks like I spoke too soon. Once I did the Python upgrade this seemed to fix it temporarily but then the submission error for Nuke and After Effects comes back. A temporary fix seems to work when you log off and log back on again but then after submitting a certain number of jobs it breaks again. Not sure what is happening exactly. I guess it has something to do with the environment setup but I don’t know what exactly and I can’t have artists continually logging off and logging on again. I did do all the installations with the installers from the repository so I am not sure what else I can do here.

Thanks.

Cheers,
Christian

kdronez · June 17, 2016, 7:33am

Hi,
you can try until you find better solution ,not so elegant solution :>.To submit job to deadline from nuke the submiter is using the deadlinecommand.exe
which it executes with the parameteres gathered (I think xD) or atleast thats how I was doing it when I did custom submiter for Natron here xD.What you can do is go to the DeadlineNukeClient.py and hardcode the path to that exe there which is usually at “C:\Program Files\Thinkbox\Deadline8\bin”.

def GetRepositoryPath(subdir = None):
# On OSX, we look for the DEADLINE_PATH file. On other platforms, we use the environment variable.
if os.path.exists( “/Users/Shared/Thinkbox/DEADLINE_PATH” ):
with open( “/Users/Shared/Thinkbox/DEADLINE_PATH” ) as f: deadlineBin = f.read().strip()
deadlineCommand = deadlineBin + “/deadlinecommand”
else:
try:
deadlineBin = os.environ[‘DEADLINE_PATH’]
except KeyError:
return “”

    if os.name == 'nt':
        deadlineCommand = deadlineBin + "\\deadlinecommand.exe"
    else:
        deadlineCommand = deadlineBin + "/deadlinecommand"

this is the function that is searching for the env which for whatever reason it cant find.It could be that you need to maybe uninstall completly dl7 or check the env in system settings.

dwallbridge · June 17, 2016, 8:49pm

Hello,

Can you go to “C:/ProgramData/Thinkbox/Deadline8/” and delete the cache folder, then open up Nuke and try to run the submitter again? This should hopefully recreate the cache folder properly.

christianl · June 18, 2016, 12:48am

Thanks, I deleted the cache folder and this worked temporarily. After I launched the SMTD in 3ds Max 2016 the Nuke submitter stopped working again with the same error. So I closed max deleted the cache folder again and Nuke submitter worked again. So there seems to be some kind of conflict here.

Maybe I am missing a permissions setting somewhere?

Cheers,
Christian

dwallbridge · June 20, 2016, 6:27pm

Can you verify the full version of Deadline here? I thought these issues were resolved in a service pack, so I want to make sure we know of any ongoing issues. Thanks

christianl · June 20, 2016, 11:56pm

The full version is 8.0.3.0.

christianl · June 22, 2016, 2:58am

Another question. If you guys knew about these issues earlier, was there a fix for it? Its becoming a real pain here for the artists in the studio and I am heavily considering rolling back to deadline 7 so I can get some submitters that work properly.

Thanks for your attention on this.

cheers,

Christian

dwallbridge · June 23, 2016, 8:42pm

Hello,

I am told there are still some permission issues, but I think the fix will be out in either 8.0.5 or 8.0.6. In the meantime, can you upgrade to 8.0.4.1 and see if that helps on this problem? Thanks.

christianl · June 30, 2016, 3:13am

Thanks, I upgraded to 8.0.4.1 and I still have this same permissions issue. If I have a 3ds max open and Nuke open at the same time then one of the submitters will not open unless I close one of the programs and then it works again. Its a real pain in the a**. If you guys have a fix for it, can I get it now instead of waiting for the next release?

I noticed that even when I close the smtd script in max, the cache folder is still locked until I completely exit out of Max, so something must keep it locked even after the submitter dialog is closed. I also noticed that after I try to send with Nuke and it errors, I then try to go to the integration tab of the SMTD and it errors on not being able to get the template formats for Draft. Not sure if that’s related but thought that I would mention it.

Thanks.

Cheers, Christian

eamsler · July 4, 2016, 9:04pm

Hey Christian,

I just fought with Nuke at a different studio on Friday. It seemed that there was an issue with how we’re reading the output from DeadlineCommand. We use that for things like finding where the submission script lives or the pools and groups. Could I give you a call tomorrow to fight with this? Maybe at 1:00pm PST?

We have a quick test script that might help (we changed the call we use for output gathering from process.stdout.read() to process.communicate()). Can you try making a backup of “[repo]/submission/Nuke/SubmitNukeToDeadline.py” and copy this in its place.

SubmitNukeToDeadline.py.zip (27.6 KB)

eamsler · July 6, 2016, 10:22pm

I’ve got good news and bad news. After banging against this awhile, I’ve found that running this is the problem when the SMTD is open:

c:\Program Files\Thinkbox\Deadline8\bin>deadlinecommand getrepositorypath submission/Draft/Main
Error: One or more errors occurred. (System.AggregateException)

I’m going to try SP5 first thing tomorrow to see if it makes a difference. If not, I’ll hand it to the core team to hack on as well. We might be able to work around it in the SMTD, but at this point I’m not sure what’s triggering it.

eamsler · July 7, 2016, 2:15pm

While I’m waiting for SP5 to extract and install, I thought I’d detail what we tested.

In the 8.1 beta, we changed the way we loaded the Draft info files (“format.txt” and friends). I created a diff from 8.1 and back-ported it to 8.0.4.1 in hopes that the new loader would change things here. No dice there, and that took a fair amount of time. It seems like the SMTD is holding those files or another open, and I did try looking for those specific files. Also, the SMTD fails to load them if you close and re-open claiming that they’re missing, but the paths it claims their missing from are empty (a side effect of whatever’s killing DeadlineCommand I’m assuming).

Then I decided to attack things from the Nuke side to figure out why the error had an exception in the file name ("(System.AggregateException)\formats.txt"). That’s usually a tell-tale sign that DeadlineCommand is having trouble as we pull the first line from it to get the Repository path.

I brought the problem over to the integration team, and on the 8.1 branch, the problem is gone. I wonder if it’s because of the changes in SP5, which is where I’m sitting now, about to test things.

Sorry this description took so long, had an appointment. I have another tonight, but if this isn’t resolved by Friday I’m going to stay here until it is.

Update:

Great news! It looks like upgrading to SP5 is all that’s needed to work around it. I thought the problem might have something to do with the Repository cache, but I re-enabled it (it’s properly disabled by default in SP5) and it also still comes through.

This means that my action plan now becomes filing bugs against Nuke and AE so they can survive those Draft settings being missing. I believe that code is shared everywhere, so it’s affecting most of our Python-based submitters.

In conclusion: This forums thread has taken far too long, and it’s entirely on our team here for not testing deeply enough to find the problem. If you ever need anything or need to vent about problems, harassing me by phone is always appreciated. The same goes for anyone else. These problems are my responsibility either direct or not, and I need to see them resolved.

Chris, can you try the upgrade and see if it resolves for you as well?