AWS Thinkbox Discussion Forums

Deadline failing prematurely HtoA jobs

Hello,

Recently we’ve had a problem with our HtoA jobs sent to Deadline, where Deadline itself would fail the job even if the job does actually render properly. I’ve been able to track the source of the issue to Deadline’s Houdini plugin STDOUT watcher.

In the scenes that fail, we have several Arnold volume procedurals which are generated automatically pointing to VDB sequences that depending on the frame it’s rendering they may exist or not. This should not be a problem since Arnold has the option to disable “Abort on error”, which would make this setup render the VDB files that exist in the rendered frame and ignore the ones that don’t.

The issue here is that when Arnold gets to the invalid VDB, it (as it should) writes in it’s log a WARNING which contains the word “Error”:

2017-02-18 12:09:01:  0: STDOUT: 00:00:18   236MB WARNING |   [volume_openvdb] IoError: could not get size of file <path_to_vdb>/Smoke_E_4_V08.0.vdb

But the Houdini plugin has the following line, which forces the job to fail:

self.AddStdoutHandlerCallback( "(Error: .*)" ).HandleCallback += self.HandleStdoutError

In Maya this issue could be fixed by disabling “Strict error checking”, but Houdini lacks of that feature. We are currently using:

Deadline Version: 7.2.1.10 R (b8ff445b3) FranticX Version: 2.3.0.0 R (be7e2c3ac)

We have edited the Houdini plugin temporarily, but this is not a final solution to the issue. Is there any workaround for this issue? Is it fixed in newer versions?

Thanks,
Salvador.

It looks like we haven’t changed it.

I’ll see about adding this in. What do you think about catching “WARNINGS” and considering them non-fatal? The way the handlers work, the first one that catches ends the matching so that would protect any other warning lines.

Would you be willing to put in a line matching “WARNING” to see if it would work on a more wide-spread basis? I’ve opened a discussion issue about whether we can create a way to ignore errors, or if we should just suppress anything with “WARNING”

I’ve been running some tests and it seems that the “WARNING” solution will not be enough. I’ve attached a file with an Arnold log of a successfully rendered image without “Abort on error” and it says:

00:00:00    61MB ERROR   |   [proc] /obj/no_proc:procedural: error reading dynamic library from WRONG_PROC_PATH: No se puede encontrar el módulo especificado.

But still renders, as it should. I’m not sure how Mantra does this, but Arnold exits with code 1 when encountered with an error unless “Abort on error” is off, which would already report on Deadline as a failure without having to watch the logs. In any case, I think it’s a safer idea to implement “Strict error checking” since arbitrary procedurals can output log as well, and you can never know what will they write.

Thanks,
Salvador.
ai.log (14.9 KB)

I definitely think you’re right here. I’ll bring it up in the discussion and we’ll promote it to a feature request.

I’m having this same problem - could not get size of file for a vdb

Is there a workaround for this? I tried disabling Abort on Error on the Arnold Rop but that didn’t help…

Hey Michael,

Have you tried, in addition to disabling Abort on Error, editing <DeadlineRepository>\plugins\Houdini\Houdini.py and replacing the following line:

self.AddStdoutHandlerCallback( "(Error: .*)" ).HandleCallback += self.HandleStdoutError

With something like this:

self.AddStdoutHandlerCallback( "\s+(Error: .*)" ).HandleCallback += self.HandleStdoutError

This will only bypass the IoError and not generic Arnold errors like error reading dynamic library from WRONG_PROC_PATH.

If you want to bypass all Arnold errors you’ll have to define a new handler like @eamsler suggested, but in this case I’m not sure what the body of the function would be. Maybe just pass?

Privacy | Site terms | Cookie preferences