AWS Thinkbox Discussion Forums

Deadline 9.0.5 - Job is completed even if there is an error

Hi everyone,

Since updating to Deadline 9.0 we’re having problems on the farm where it seems like Deadline considers a job to be complete even if it errors out.
So instead of trying to re render it, it just stops the render which leads to renders not being ready in the morning which leads to rush renders in the morning to meet the deadlines :wink:

We just updated all our farm to Deadline 9.0.5 a few days ago but it seems like it still happens…
I’m hoping that it’s just a setting we can change either in the repository options or by changing a few lines of code in the repository.

FYI we’re using 3dsmax and Vray. We send jobs with distributed rendering on the farm (1 blade becomes the master and the other 9 ones are Vrayspawners)

Anyways, I attached the job report here.

Here is the error (which happens once in a while when the camera gets corrupted in some ways)
2017-07-18 18:53:56: 0: STDOUT: 7/18/2017 18:53:56 PM; Camera Cam_Entrance_ defined but not found
2017-07-18 18:53:56: 0: STDOUT: 7/18/2017 18:53:56 PM; Error rendering frame 0: Camera Cam_Entrance_ defined but not found

And here are the lines which in my mind are part of the problem
2017-07-18 18:54:05: 0: STDOUT: 7/18/2017 18:54:04 PM; Job Completed with Error(s) - see above
2017-07-18 18:54:05: 0: STDOUT: 7/18/2017 18:54:04 PM; Scene C:\Users\renderone\AppData\Local\Thinkbox\Deadline9\slave\qcore167\jobsData\596e9163d5a521a6304eac85\EXT_Entrance_06.max completed.
2017-07-18 18:54:15: 0: STDOUT: 7/18/2017 18:54:14 PM; Error occured while rendering job.
2017-07-18 18:54:15: 0: INFO: V-Ray DBR: Restoring backup config file: C:\Users\renderone\AppData\Local\Thinkbox\Deadline9\slave\qcore167\jobsData\596e9163d5a521a6304eac85\vray_dr.cfg to original location: C:\Users\renderone\AppData\Local\Autodesk\3dsmax\2016 - 64bit\ENU\en-US\plugcfg\vray_dr.cfg
2017-07-18 18:54:15: 0: INFO: V-Ray DBR: Marking other incomplete tasks as complete

The job should not be completed if the task had some errors.

Hopefully it’s an easy fix…

Thanks!

Chris
Job_2017-07-18_18-54-16_596e91983dff78070c90bd41.txt (15.2 KB)

In your “3dsCmd.py” file in your “…/plugins/3dsCmd/” directory in your “<your_repo_path>”, open it in a good text editor and locate this line (#1271):

        self.AddStdoutHandlerCallback( "'ERROR :.*" ).HandleCallback += self.HandleStdoutError

and add a new line below it, so it looks like this (and save the file):

self.AddStdoutHandlerCallback( "'ERROR :.*" ).HandleCallback += self.HandleStdoutError self.AddStdoutHandlerCallback( ".*Error rendering.*" ).HandleCallback += self.HandleStdoutError

Re-queue your troublesome job or submit a fresh test and see how that works for you.

If I had to guess, I would say there is an illegal character at the end of your camera name, like a TAB or WHITESPACE characters OR you have one or more objects in your scene which have exactly the same name as the camera and 3ds Max is getting confused as to which object is actually the camera! If you like, you can delete all geo out of your scene and just send us the empty 3dsMax scene file with the camera in it (+ any helpers/refs that are chain linked to the camera) and we can check it out for you.

Finally, try opening up the SMTD submission interface instead and running the “Sanity Check” system. Does it flag anything wrong with the camera there?

Thanks Mike for your quick answer!

I added the line of code to our 3dsCmd.py file and it still doesn’t seem to catch the error.

The error as you were saying is happening because the camera has a space at the end of its name.

I’m not too too worried about the camera naming problem as I can fix it in my sanity check.
My main problem is that it seems like errors during the render are not being seen by Deadline as errors and it just completes the job.

We are not using the regular Deadline script to send our job but a custom version of it.
The weird thing is that when I submit exactly the same job with your script, the sanity check doesn’t detect an error with the camera and it just renders properly without any errors.
It could be something that is related to our script but we didn’t have any problem before updating to Deadline 9. Can you think about something that was updated in Deadline 9 that could create such a behavior?
If I am to look into updating our script, which direction should I look into?

Please let me know if you have any questions or need clarifications.
Thanks!
Chris

Update:
When I sent the job with the regular submitter, I didn’t use the V-ray DBR off load. This is when it worked.
If I send the same job with the regular submitter and the DBR (Off load) on, I’m getting the same exact error “Error rendering frame 0: Camera Cam_Entrance_ defined but not found”. In this case Deadline doesn’t detect the error and just completes the task.

So it might be related to the V-ray DBR in some ways.

I just attached the empty scene with just the camera if you’d like to test it.

Let me know what you think.

Thanks!

Chris
Test_Scewne_.zip (1.51 MB)

The root issue here is I forgot earlier an old issue which was introduced in Max 2016 when “3dsmaxcmd.exe” generates its StdOut in utf-16 instead of utf-8, which results in our StdOut handlers not ‘catching’ on whatever our RegEx’s are looking for. So, in your example, we are currently unable to catch and raise an exception when we hit the “…Error…” StdOut line in the log.

For the time being, please sanity check your camera names before submission. Someone in our core engineering team is looking into this above root issue as I write this. No ETA as yet. Additionally, I’ll take a look at whether we can add a sanity checker ourselves. (I thought previously we did already have one; but in hindsight, I think I was getting confused between code I wrote in-house when I use to work in post-production compared to Thinkbox code). Regardless, adding a few simple trim commands to the end of camera names should be trivial to add.

Thanks Mike!

Unfortunately the camera error is just one of the error that Vray generates when there is a problem…
I just had the problem happen again today where Vray just gives me a generic error:
2017-07-19 14:57:46: 0: STDOUT: 7/19/2017 14:57:46 PM; An unexpected exception has occurred in the network renderer and it is terminating.
2017-07-19 14:57:46: 0: STDOUT: 7/19/2017 14:57:46 PM; Error rendering frame 0: An unexpected exception has occurred in the network renderer and it is terminating.
2017-07-19 14:57:47: 0: STDOUT: 7/19/2017 14:57:47 PM; Job Completed with Error(s) - see above

I attached the error report to this post.

So even with a sanity check we can’t really check that.
Also this problem only started as soon as we installed Deadline 9. Everything was working perfectly before that.
Do you think we should revert back to Deadline 8 for the time being?

Thanks,

Chris
Job_2017-07-19_14-57-49_596fabacc8a2cb0a20e90be5.txt (16 KB)

Is this other error during a non-DBR-offload job? If so, why not use the better, more advanced “3dsmax” plugin which is the standard plugin the majority of our 3ds Max customers use for all types of rendering. I doubt Deadline 8 will provide any difference here, as its the version of 3ds Max (2016 or newer) that is causing the issue here. If your saying that this only started to happen in Deadline 9, did you also upgrade your version of 3ds Max at the same time or relatively same time, as updating Deadline from 8 to 9?

The overall annoying thing with using the basic 3dsCmd (3dsmaxcmd.exe) plugin, is the total lack of half decent error messages when things go wrong in the renderer or 3dsMax. Its essentially one of the major reasons, why our users prefer our more advanced “3dsmax” (Lightning c++ plugin) solution: deadline.thinkboxsoftware.com/fe … esk-3dsmax

Hi Mike,

Once again thanks for your quick reply.

Im not sure if I understand clearly the difference between lightning and 3dsmxcmd so please correct me if I’m wrong but I created our internal script based on this tutorial deadline.thinkboxsoftware.com/cu … ubmission/ and when not using our internal script to submit our jobs we use SMTD so as far as I understand we are using lightning for submitting all of our jobs.

The only thing that seems to be related to 3dsmaxcmd when sending our jobs is when we use VRay DBR off load we check the 3dsmaxcmd option as we don’t use 3dsmax licenses for our renderfarm. Is that part of the problem?

We have been using 3dsmax 2016 for almost 2 years and started using deadline just more than a year ago so we’ve always been using max 2016 with deadline. The problem started as soon as we installed deadline 9 and I can’t think about any major software upgrade that has happened at the same time.

I just sent that test again (the one with the corrupted camera name) through SMTD with Vray DBR off-load (with 3dsmsxcmd checked) and the result is the same: the job creates an error because of the camera but deadline still completes the job.

We haven’t changed our workflow or software when going from deadline 8 to 9 but I’m sure that deadline 8 was detecting these kinds of problems. I understand that the camera problem could be tested before submission but there are a lot more errors that our deadline manager doesn’t detect as being problems.
Do you think that it could just come from an option to check somewhere in the repository settings?

I attached the file that I used for testing to this post.

Let me know if you need more information or if you have questions.

Thanks.

Chris

Deadline has 2 x plugins for 3ds Max:

  1. “3dsmax” (uses our custom ‘Lightning’ plugin under-the-hood, which you don’t need to worry about)
  2. “3dsCmd” - basic plugin, essentially functionally identical to built-in Backburner plugin.

So, essentially, you are using the more advanced “3dsmax” plugin already, except when, as you said yourself, you push to “3dsCmd” to save a license for offload-DBR. So, that clears that up.

If your getting errors in the normal 3dsmax plugin, then that would be a separate issue, as we don’t have an encoding issue there.

See attached for updated SMTD(3dsmax) and 3dsCmd in-app sanity checkers, both of which now have a “CheckForBadCameraNames” function in them. Un-zip to the same named directories in your current repo, overwriting the same named files already in those locations. Restart SMTD or 3dsCmd in-app submitter and ensure “Run Sanity Check Automatically Before Submission” checkbox is enabled (this setting can be configured to be on by default and sticky if required. See here for more info on that:
deadline.thinkboxsoftware.com/ma … r-defaults
docs.thinkboxsoftware.com/produ … y-defaults

submission.zip (22.5 KB)

In terms of why there seems to be a difference for you between DL 9 and DL 8. I’m not sure. I think between these versions we did migrate just the “3dsCmd” plugin from being a “SimplePlugin” to an “AdvancedPlugin”, but you shouldn’t have noticed any difference there. One way to confirm this, is to simply temporarily replace the “3dsCmd” plugin in your repo in DL9 with the old one from your DL8 repo. (Do a backup before you do this). I’ll run a test here later today to see if I can see any difference here as well.

I also note that one of our core developers quickly overhauled a load of code overnight to add a special new property, which will allow us to support different encodings in a future release, so we are working on fixing this issue, but it will be a while before you get your hands on something here.

Ah…I think I might be onto something. Perhaps there is a difference between MonitoredManagedProcess and ManagedProcess, which essentially would be the difference between a Deadline Simple and Advanced plugin. So, yeah, try a swap out of DL8 “3dsCmd” plugin into your DL9 repo and see if that helps?

It worked!

It’s one of the first time that I’m happy to see an error in Deadline!

So Deadline caught the error as being an error, created an error report and another blade picked up which is exactly what I wanted :wink:
I attached the crash report to this post.

So do you know exactly why this is happening?
Is that an easy and quick fix or would fixing it involve reverting back your code in a big way?
Are we just good running deadline with version 8 of the 3dsmaxcmd? Will that create other unexpected problems for us?

Thanks so much for your help this was really driving us crazy!

Chris
Job_2017-07-20_12-45-56_5970de4326322103ec2c6026 Deadline 8 3dsmaxCmd Plugin.txt (17.6 KB)

OK, we have figured out what is going on here. You can just keep using the DL8 version of the plugin for now and we will get this fixed up for the next service pack of DL9, which when you download and install it, it will by default, overwrite the 3dsCmd plugin with the newer files. Thanks for helping to track down this issue!

Thanks for being so reactive!

Privacy | Site terms | Cookie preferences