AWS Thinkbox Discussion Forums

task done, frames are missing

Deadline 10.0.12.1, redshift and maya 2018
When job is submitted to deadline everything renders fine but there are tasks with missing frames, task is done, time spent on it but frame is missing. I see that output size in task monitor is also missing as well.
Looking into log and frames that are fine I can see redshift bucket progress, in bad frames that part is missing.
Rendering form live maya GUI works fine.
Basically after each render I have to go to live maya and render missing to fill in holes.
Submitting only missing ones in deadline will also produce same result some frames are added some are missing.

In different projects and different slaves so probably not project and slave related.
Any ideas?
Attaching task logs for good and bad task.


missingframes.txt (76.6 KB)
goodframes.txt (444 KB)

And based on log and replyu on redshift ti seems that in case of missing frames redshift never started rendering so maybe something with command line?

For now solution is to simple render 1 frame per task and then find out those without output file size and then requeue and requeue till they are added. From what I can se so far this happens in maya 2017 and 2018.

One handy feature for missing frames is that Deadline has a feature for that:

2018-03-20 09_10_39-Deadline Monitor  -  C__DeadlineRepository10.png

Things to note here:

  1. Because local rendering is on, we can see that there were no frames written to disk on the “missingframes” task.
  2. The tasks are a bit different in that one is the beginning of the job starting up and one is a continuation

The big issue here is that Deadline isn’t catching that the output wasn’t written. There was a feature some time ago that would throw an error if output wasn’t written, but there were some workflows where that was oddly intentional (I think it was render element output overrides).

Were these from the same job?

Tested on two different projects, and both times using render local option enable and disabled final effect was the same.
But what is noticeable is that in those cases where there is no frames Redshift never actually picked up rendering, there is no buckets done, no redshift progress at all. Is it possible that something just messes up and redshift don’t do any rendering at all but it is still marked done and time is spent?

It could be. Actually, are you using concurrent tasks at all? I think we discovered recently that access to the same GPU in multiple tasks / Slaves is not support and while it usually locks up, I wonder if this may be an issue.

I suppose the easy workaround for now would be to find a way to fail if we don’t see Redshift has rendered. I’d rather see us implement something more universal though. I’ll chat with the integration team about this one…

Not using concurent tasks. Have render nodes with 4 GPUs each, and running 2 slaves per node, each slave have it’s GPU affinity using 2 GPUs.

Short version
slave_A-1 GPU 0, 2
slave_A-2 GPU 1, 3
and so on for each slave

I discussed the idea of renderer short-circuit detection, but we’re not sure if that would be a good test here yet. One thing we know is that Redshift doesn’t follow the same methods when we ask it to render so this may have something to do with that implementation.

Would you be willing to run a few more test? Mainly, what we’re looking for:

  1. Can you re-render with MayaBatch and enable “log script contents to render log” in the Plugin configuration. We’d want a success and fail again
  2. Can you try rendering without MayaBatch? This will disable some of the more advanced settings, but we’re trying to rule out melscript

Here’s some info on the MayaBatch config options for request #1:
docs.thinkboxsoftware.com/produ … figuration

Sorry bin a bit bussy on this side.
But just today seen this on forum as well:

[i]Hey everyone,

We’ve been using Exocortex for some time now and recently updated to Maya 2018.2.

Problem is when we send renders to our farm some nodes and workstations appear to skip-not-read the Alembic files on our servers and the renders come out with the meshes in default position.

It started happening less frequently when we set all our computers to DG evaluation, which we ended up doing cus we were also having problems with Viewport display on Maya itself.

Our workflow is moving from having the Exocortex nodes on a referenced file and then repathing, to actually creating the nodes on the file itself. In both cases, we’ve got this error.

Any ideas on what could be happening?

Thanks![/i]

Missing frames along with couple more weird issues I run into like frames are rendered but objects inside scene are on and off from frame to frame and couple similar creepey stuf… it seems that all could be due to maya 2018.2 update!

I’ve uninstalled maya and back to pure 2018 version now and testing… first couple jobs seems perfectly fine… will do couple more tests to confirm…

Also beside these issues tthere is other one, if you submit maya 2018.2 job to deadline it won’t render. error reading file, even tho it loads jsut fine in gui maya 2018. So there is that as well…

The Maya 2018 failing to open the file is usually because it’s not PU2. Maya will refuse to open a file whose version number comes from a newer version. Claims the scene is corrupt. An old hack that still works today is to save as ASCII and modify the version in the file header yourself.

For this though, mepp was discussing a discovery he made to day having to do with path mapping and file nodes which did not contain file paths. The bug seems to be that the first tasks are failing but subsequent ones for the already loaded Maya instances seem to work just fine. He has an idea of what he can do to fix that problem, so I’d like to run the script by you once we have something worth testing.

Speaking of mepp… Here’s an updated plugin file if you’re interested, works on top of the latest Deadline10!

It’s pretty much like Edwin described. We noticed that maya’s dirmap was failing on file nodes that didn’t actually have a filepath in them. Path mapping only happens on the first task for a Slave in mayabatch, so you’d end up seeing the first task for a Slave fail with no render output and all subsequent tasks would render fine. It was pretty frustrating since there was nothing indicating what was going wrong when the dirmap failed.

If you’re curious to see if you’re affected by this bug I’ve fixed. You’ll have “DeadlineRepository10/plugins/MayaBatch/MayaBatchUtils.mel” in your repository (one of the later versions of Deadline 10, (ie. 10.0.12) and the following script will tell you if you’re missing filepaths in nodes. Run this in the maya script editor with the scene in question open.

string $fileNodes[] = `ls -type file`;

for( $fileNode in $fileNodes )
{
    string $fileAttr = $fileNode + ".fileTextureName";
    string $curFile = `getAttr $fileAttr`;
    if($curFile == "")
    {
        print( "'" + $fileNode + "' doesn't have a filepath in it\n");
    }
}

If any node is printed, then we assume it has a filepath to be path mapped, but it doesn’t actually have a path. If you have any output from this script, update the attached file in your repository.

  • back-up the original file at: DeadlineRepository10/plugins/MayaBatch/MayaBatchUtils.mel:
  • unzip and copy the attached file to the location above

Let us know if this resolves the first task failing issue. There always the possibility that more might be going on here that’ll need to be addressed.

Cheers

1 Like

hey. sry a bit busy here but still struggling with missing frames. render goes actually seems o be going but frames are missing.
still same thing weird most of them are there but some missed and skipped, so if you have any progress on this or something to test with I’m all up for it. I’m wasting TON of time chasing these frames :frowning:
also since last time I’ve reinstalled back to pure maya 2018 no updates or anything to remove any chance of bugs there and updated repository to latest version as well. stills same thing so whatever you have to try I’m all up for it!
Also did some Houdini renders and cinema as well… all fine no missing frames so seems to be maya thing…not 100% sure but…
added another screen grab to see how it looks in monitor

Hey mirkoj,

The fix I posted above has not yet been released, it’s in an upcoming build. Can you run that piece of code in my above post in the Maya script editor? It should hopefully reveal if my suspicions on empty filepaths being correct.

Otherwise, could you apply the fix to your repository and let us know how that works for you?

Cheers

ah sorry missed that one,
indeed there were nodes after that mel script.
updating MayaBatchUtils.mel in repository now will do a test right away.

4 scenes that I tried that had issues so far, all 4 working without problem. All frames here, seems to be working! :slight_smile:
Thanks!!

Glad to hear it, thanks for testing it! :smiley:

Cheers

Just wondering, did this fix made it;s way into deadline version 10.0.15.5?

Yes, this fix is in Deadline 10.0.15.5.

Privacy | Site terms | Cookie preferences