Our graphics package utilizes the Video Copilot effect “Saber”. Because of the hundreds of files we generate I use Deadline to slave them out. One of our nodes refuses to finish a render. It will start and render the quicktime all the way up to 100% but never report that the task is done. After doing some digging I discovered when Saber is not present in the comp the issue goes away.

Here is the end snip of the output from the troublesome node:

and here is a snip from a node rendering the same job but can complete it:

There is a incongruity in the output lines and I wonder if that is the cause. The only difference between the 2 machines that I can note is that the troubled node is running windows 7 and the node that renders it fine is a windows 10. All the other versions match up.

I am at a loss at where else to check. Has anyone seen anything similar to this ?

thanks in advance : )

Hey polymangler,

What version of Deadline of are you running (ie. 10.0.13.6)? Have you tried running the command-line job on the errant Slave outside of Deadline? If you look in the log for the job, you’ll see the full command that we’re running which you can then run. I’d be interested in the results of that test.

I’m not familiar with the Saber plugin, but I wonder if there’s an issue with how Saber is notifying AE that it’s done. If it’s reporting that it’s 100% done and there’s an output file created by that task I could see that being a potential cause if the process is just hanging there…

As a temporary solution, if it’s just the one machine that’s causing issues, you can always blacklist that Slave from rendering jobs containing Saber.

Cheers

Saber often locks up AERender and we’re not sure why. Does anyone have access to the Video Copilot guys?

Many of the Video Co-Pilot plugins make an unfortunate reference to the AE UI, which when executed via the headless aerender executable, causes it to crash/hang at entry point or at exit. This seems to be the latter issue. I would report the issue to Video Co-Pilot directly as they need to fix all their plugins to be compliant, regardless of renderfarm manager used: videocopilot.net/support/

There is also a known AE CC 2018 issue combined with the “Fast Blur (Legacy)” AE effect in a comp. If you using 2018 version and you use this specific AE effect, they all bets are off here as its highly unstable currently and so, the Saber reported issue may be a false-positive here. (I doubt that is the case here, but I mention it so you can eliminate it if applicable).

Hey guys, just back to the office after some well deserved days off so let me run through the responses

Mepp: I’m running version 10.0.10.4, I’ve run the same render through the command line and the results were the same. Sadly blacklisting that one machine would not be a good solution since it is our only slave with all plug-ins loaded. Most of our nodes are off-site now, we just keep this guy kicking around for AE render purposes

eamsler: That is a real bummer to hear, it a big part of this shows package

MikeOwen: I will let my voice join the chorus over at Video Copilot and see if they have any solutions.

Thank you everyone for taking the time to help me with my issue.

Cheers,

No problem. You’re not the first to hit this one, and if they need anything from our side I’ll make the time.

Ah, so good news! I was reminded of a hack I implemented for a company. It’s buggy, but it sort of works if you’re not rendering QuickTimes?

The gist is that we can hack the Deadline script to forcibly close AE after a timeout, but it was often not that reliable. You can use it at your own risk sort of thing.

What I did was hijack the “HandleRenderSuccess” call which is run when we see “Finished Composition” in the render log. We only seem to see that when AE is rendering a frame sequence and not when rendering a QuickTime if I recall correctly. In that function, I let AE sit for a little while and then close it. Without that pause, final output frames and video wouldn’t be written out correctly.

You can test this code by making a copy of the “[repo]/plugins/AfterEffects” folder and putting it into “[repo]/custom/plugins”. Then edit “[repo]/custom/plugins/AfterEffects/AfterEffects.py” and replace the “HandleRenderSuccess” with what’s below.

    def HandleRenderSuccess( self ):
        if not self.ExitInProgress:
            self.RenderSuccess = True

            # Edwin: Because Saber locks up AfterEffects, we need to force it to close
            self.LogInfo("Found 'Finished Composition', waiting 30 seconds then exiting")
            sleep(30)
            self.AbortRender("Exiting due to success", AbortRender.Success)

YMMV on this one pretty wildly, but at least AE will eventually close when using Sabre.

Hello,

I have been running into a similar issue as polymangler, where slaves will get stuck “rendering” on comps that contain Saber. The difference is that I am rendering out a .TGA sequence and not a quicktime movie.

I tried hacking the “HandleRenderSuccess” call as eamsler suggested, but that threw errors saying the “ExitInProgress” attribute was not defined.

Currently using Deadline 10.0.20.2 Release (12de851eb) and After Effects 15.1.2.

Any suggestions would be greatly appreciated. Let me know if there is any more information I can provide.

Thanks!

Ah! It looks like " ExitInProgress" is completely unneeded. I’ve updated the code above. Try that again.

Thanks for the quick reply @eamsler!

I did a quick test render with the update you provided, and I am still getting the same error.

=======================================================
Error
=======================================================
AttributeError : 'AfterEffectsPlugin' object has no attribute 'ExitInProgress' (Python.Runtime.PythonException)
File "C:\Users\render\AppData\Local\Thinkbox\Deadline10\slave\T7910\plugins\5bec8b98f0d4fd2678963941\AfterEffects.py", line 625, in HandleRenderSuccess
if not self.ExitInProgress:
at Deadline.Plugins.PluginWrapper.RenderTasks(String taskId, Int32 startFrame, Int32 endFrame, String& outMessage, AbortLevel& abortLevel)
=======================================================
Type
=======================================================
RenderPluginException
=======================================================
Stack Trace
=======================================================
at Deadline.Plugins.Plugin.RenderTask(String taskId, Int32 startFrame, Int32 endFrame)
at Deadline.Slaves.SlaveRenderThread.c(TaskLogWriter aem)

Thanks!

Mmm… It definitely shouldn’t be there. Can you confirm that if you have a file in “[repo]/custom/AfterEffects” that it’s the file you’re editing? If not, just go for “[repo]/plugins/AfterEffects”. The smoking gun here is the line number of “line 625” so I’d double check your edits on that line.

If this is a bit overwhelming, feel free to call us here at +1-866-419-0283 ext 2. Should be a quick screenshare, so just download Chime before calling so we can look over your shoulder while edits are happening.

I would like to to try this fix.

Currently, I edited the AE submitToDeadline script to search effects used in the Comp. If it finds Saber, it enables the auto-timeout. This was not ideal as a new timeout is always recalculated using the latest recalculated timeout.

So, to confirm the code above, we do not use self.ExitInProgress and the code is

def HandleRenderSuccess( self ):
        self.RenderSuccess = True

        # Edwin: Because Saber locks up AfterEffects, we need to force it to close
        self.LogInfo("Found 'Finished Composition', waiting 30 seconds then exiting")
        sleep(30)
        self.AbortRender("Exiting due to success", AbortRender.Success)

thanks!

Jason

Jason,

Would you mind sharing the edits you made to the AEsubmitToDeadline script?

Similarly, is it possible to edit the AEsubmitToDeadline script to have the Auto Task Timeout automatically enabled for all AE jobs submitted to Deadline?

Thanks!
-Andrew

Hi Andrew,

in the default script, starting at line 1582 is where the job submission info file is created.

you just need to add the following.

submitInfoFile.writeln( “EnableAutoTimeout=true” );

But as I mentioned, this timeout is recalculated based on tasks that timeout. So if timeout is 2x, if task time is 10mins, timeout is set to 20mins. And when a task timeouts at 20mins, the timeout then gets recalculated to 40mins. For this reason, I added -

submitInfoFile.writeln( “OnTaskTimeout=Fail” );

This means the task timeouts, but says fails. Not very pretty, which is why I want to use the method mentioned earlier. Were you able to get that working?

In my case, I didn’t want to the auto timeout for all jobs… only ones with saber.

         if( saber_found == true){
          submitInfoFile.writeln( "OnTaskTimeout=Fail" );
          submitInfoFile.writeln( "EnableAutoTimeout=true" );
     }

saber_found is set using the following

	var saber_found = false;
function retrieveProjectItems (itemType){
	var typeOptions = new Array("Composition","Folder","Footage");
	for(var t = 0; t < 3; t++){
		if(itemType == typeOptions[t]){
			var proj, itemTotal, curItem, itemAry;
			itemAry = new Array();
			proj = app.project;
			itemTotal = proj.numItems;
			for(var i = 1; i <= itemTotal; i++){
				curItem = proj.item(i);
				if(curItem.typeName == itemType){
					itemAry[itemAry.length] = curItem;
				}
			}
			return itemAry;
		}
	}
}

function grabAllEffects(myComps){
	var totalLayerCount, curComp, myComps, myCompsLength, myEffectsLayerAry, np, curProp, propDepth;
	myCompsLength = myComps.length;
	myEffectsLayerAry = new Array();
	for(c = 0; c < myComps.length; c++){
		curComp = myComps[c];
		totalLayerCount = curComp.numLayers;
		for(l =1; l <= totalLayerCount; l++){
			curLayer = curComp.layer(l);
			np = curLayer.numProperties;
			for (lp = 1; lp <= np; lp++){
				curProp = curLayer.property(lp);
				if(curProp.matchName == "ADBE Effect Parade"){
					propDepth = curProp.numProperties;
					if(propDepth > 0){
						for(ep = 1; ep <= propDepth; ep++){
							 myEffectsLayerAry[myEffectsLayerAry.length] = {'fxN': curProp.property(ep).name, 'CompN': curComp.name};
						}
					}
				}
			}
		}
	}
	return myEffectsLayerAry;
}    


var myComps = retrieveProjectItems ("Composition");
var totalEffectCount, myEffectsArray

myEffectsArray = new Array();
myEffectsArray = grabAllEffects(myComps);

totalEffectCount = grabAllEffects(myComps).length;


for (var e = 0; e < totalEffectCount; e++){
	if (myEffectsArray[e].fxN == "Saber"){
        alert("Comp: " + myEffectsArray[e].CompN + " contains " + myEffectsArray[e].fxN + " plugin.\n\n" + "This causes some render tasks not to exit on certain nodes.\nAn auto timeout hascont been set.\n\n" + "NOTE: Ignore FAILED tasks as all frames should be rendered.\n\nContact support if all frames have not been rendered.");
        saber_found = true;
        break;
    }
}