Houdini Mantra IFD rendering

eamsler · January 3, 2017, 7:13pm

Okay, no luck over here for a few reasons. It’s complaining about “/out/logo/merge1” over here, and there were a very large number of unknown attributes when the hip file loaded.

Maybe it’d be better if we set up a call since this is so hard to reproduce? It’s been fairly quiet over here while people are ramping back up after the break.

Nick_Weeden · January 3, 2017, 10:16pm

Hi Edwin,

Is it because the paths are all missing? To make our files work with deadline we have to hard code all the paths so they need to be re-pathed once you’ve opened it. It doesn’t work if we leave it with the relative $HIP variable.

Nick

eamsler · January 4, 2017, 6:06pm

It might be. I’ll take another crack at it this week. My Houdini abilities aren’t great, so it’s an excuse to (re)learn things.

antoinedurr · January 4, 2017, 7:41pm

You need to globally set and use something like $JOB, i.e. something which is /not/ reset on every invocation of Houdini. In other words, set $JOB to the root of your project (which has hip/geo/tx/etc. subdirs under it), and make all file paths relative to $JOB.

Nick_Weeden · January 5, 2017, 8:31am

Hi Edwin,

I’ll have a go at setting the $JOB variable and seeing if that makes it easier as Antoine suggests. If so, I’ll send you a new file.

Nick

Nick_Weeden · January 5, 2017, 3:37pm

OK! So I’ve uploaded a new file, I’ve included all the dependent files this time (sorry, I left some out before) and removed most of the things you don’t need. I’ve also set it up with the $JOB variable (thanks Antoine that seems to work ). So you should be able to open the file and then set the project (File->Set Project) to the folder you’ve saved it into. The paths should all work relative to that path then, including the outputs. The output node you need to select when submitting is “particles”.

Hopefully this should now work for you.

Nick

project files: https://we.tl/FHrfj6LU67

eamsler · January 6, 2017, 2:51pm

So… Here’s a fun turn. It seems to be working fine?

Screen Shot 2017-01-06 at 8.51.08 AM.png

Nick_Weeden · January 6, 2017, 3:05pm

Hi Edwin,

The job hasn’t completed yet though has it…? My first frame doesn’t fail until the 70th task.

Nick

Nick_Weeden · January 6, 2017, 3:10pm

It completes most of the frames fine.

[attachment=0]deadlinemonitor_2017-01-06_15-08-32.png[/attachment]

eamsler · January 6, 2017, 3:43pm

It looks like it’s getting stuck on frames 32 and 33 (I’m using concurrent tasks). I’ll reboot in case something weird happened when my workstation went to sleep.

Nick_Weeden · January 9, 2017, 11:18am

Hi Edwin,

Any update? I’m kind of hoping it’s erroring for you now…! The job does seem to change up the frames it errors on but generally it’s been from frame 70 onward. We run slaves with concurrent tasks and without and both failed in the same way. Are you running it on a Mac as well? We’re on Windows 7 but I’m not sure if this would make a difference…

Cheers

Nick

eamsler · January 9, 2017, 4:24pm

Nothing. Restarted the machine and the Slaves a few times. I’ve now deleted all the IFDs to see if it gets stuck on a different frame.

Update:

Still locking up on Frame 32 after deleting the IFDs and EXRs. I’ll move this test over to a Windows box. I’m testing with 15.0.179.25.

eamsler · January 10, 2017, 3:31pm

Moved to Windows and Houdini 15.0.459 and I’m hitting some pretty excessive crashing here, but not the same as you were seeing:

[attachment=0]Screen Shot 2017-01-09 at 5.17.38 PM.png[/attachment]

2017-01-09 17:15:36:  0: INFO: Argument:  -V 4a -j 0 -f "C:/Users/Edwin/AppData/Local/Thinkbox/Deadline8/slave/support-01/jobsData/5874112d52246c2e503b9024/thread0_temps1PhA0/Particle_Logo.0010.ifd.sc"
2017-01-09 17:15:36:  0: INFO: Full Command: "C:\Program Files\Side Effects Software\Houdini 15.0.459\bin\mantra.exe"  -V 4a -j 0 -f "C:/Users/Edwin/AppData/Local/Thinkbox/Deadline8/slave/support-01/jobsData/5874112d52246c2e503b9024/thread0_temps1PhA0/Particle_Logo.0010.ifd.sc"
2017-01-09 17:15:36:  0: INFO: Startup Directory: "C:\Program Files\Side Effects Software\Houdini 15.0.459\bin"
2017-01-09 17:15:36:  0: INFO: Process Priority: BelowNormal
2017-01-09 17:15:36:  0: INFO: Process Affinity: default
2017-01-09 17:15:36:  0: INFO: Process is now running
2017-01-09 17:15:36:  0: STDOUT: [17:15:36] Loading IFD from C:/Users/Edwin/AppData/Local/Thinkbox/Deadline8/slave/support-01/jobsData/5874112d52246c2e503b9024/thread0_temps1PhA0/Particle_Logo.0010.ifd.sc
2017-01-09 17:15:36:  0: STDOUT: [17:15:36] Registering procedural 'engineinstance': mantra/VRAY_ProcEngine.dll
2017-01-09 17:15:36:  0: STDOUT: [17:15:36] Registering procedural 'alembic': <built-in>
2017-01-09 17:15:36:  0: STDOUT: [17:15:36] Registering procedural 'packed': <built-in>
2017-01-09 17:15:36:  0: STDOUT: [17:15:36] Registering procedural 'sprite': <built-in>
2017-01-09 17:15:36:  0: STDOUT: [17:15:36] Registering procedural 'image3d': <built-in>
2017-01-09 17:15:36:  0: STDOUT: [17:15:36] Registering procedural 'file': <built-in>
2017-01-09 17:15:36:  0: STDOUT: [17:15:36] Registering procedural 'fur': <built-in>
2017-01-09 17:15:36:  0: STDOUT: [17:15:36] Registering procedural 'program': <built-in>
2017-01-09 17:15:36:  0: STDOUT: [17:15:36] Registering procedural 'image3dvolume': <built-in>
2017-01-09 17:15:36:  0: STDOUT: [17:15:36] Registering procedural 'vexvolume': <built-in>
2017-01-09 17:15:36:  0: STDOUT: [17:15:36] Registering procedural 'ptinstance': <built-in>
2017-01-09 17:15:36:  0: STDOUT: [17:15:36] Registering procedural 'ptreplicate': <built-in>
2017-01-09 17:15:36:  0: STDOUT: [17:15:36] Registering procedural 'agent': <built-in>
2017-01-09 17:15:38:  0: STDOUT: [17:15:38] mantra Version 15.0.459 (Compiled on Apr 27 2016)
2017-01-09 17:15:39:  0: STDOUT: 6244: Fatal error: Segmentation fault
2017-01-09 17:15:39:  0: STDOUT: -- TRACEBACK BEGIN --
2017-01-09 17:15:39:  0: STDOUT: Traceback from mantra 15.0.459 (Compiled on windows-x86_64-cl17):
2017-01-09 17:15:39:  0: STDOUT: CURRENT THREAD 12280
2017-01-09 17:15:39:  0: STDOUT: +0x5236a2c1 [UT_StackTrace::doTracebackIntoBuffer]
2017-01-09 17:15:39:  0: STDOUT: +0x5208c9ad [UT_CrashHandler::signals]
2017-01-09 17:15:39:  0: STDOUT: +0x5208cb88 [UT_CrashHandler::signals]
2017-01-09 17:15:39:  0: STDOUT: +0x5231dcf6 [UT_Signal::processSignal]
2017-01-09 17:15:39:  0: STDOUT: +0x24e7f5c5 [XcptFilter]
2017-01-09 17:15:39:  0: STDOUT: +0x768b4dee
2017-01-09 17:15:39:  0: STDOUT: +0x24e7e98b [_C_specific_handler]
2017-01-09 17:15:39:  0: STDOUT: +0x3fa533fd [_chkstk]
2017-01-09 17:15:39:  0: STDOUT: +0x3fa14847 [RtlRaiseException]
2017-01-09 17:15:39:  0: STDOUT: +0x3fa5258a [KiUserExceptionDispatcher]
2017-01-09 17:15:39:  0: STDOUT: +0x5000592b [blosc_set_nthreads]
2017-01-09 17:15:39:  0: STDOUT: +0x500018d4 [blosc_compress_ctx]
2017-01-09 17:15:39:  0: STDOUT: +0x50002e61 [blosc_set_nthreads]
2017-01-09 17:15:39:  0: STDOUT: +0x50002325 [blosc_list_compressors]
2017-01-09 17:15:39:  0: STDOUT: +0x50001aaf [blosc_decompress_ctx]
2017-01-09 17:15:39:  0: STDOUT: +0x520fb5b2 [UT_IStreamBuf::getErrorStr]
2017-01-09 17:15:39:  0: STDOUT: +0x520fbaf6 [UT_IStreamBuf::getErrorStr]
2017-01-09 17:15:39:  0: STDOUT: +0x520fc8ae [UT_IStreamBuf::getLine]
2017-01-09 17:15:39:  0: STDOUT: +0x520f4e48 [UT_IStream::getLine]
2017-01-09 17:15:39:  0: STDOUT: +0x6903d957 [VRAY_Scene::load]
2017-01-09 17:15:39:  0: STDOUT: +0x768b1929
2017-01-09 17:15:39:  0: STDOUT: +0x768b450b
2017-01-09 17:15:39:  0: STDOUT: +0x3d8413d2 [BaseThreadInitThunk]
2017-01-09 17:15:39:  0: STDOUT: +0x3f9d54e4 [RtlUserThreadStart]
2017-01-09 17:15:39:  0: STDOUT: OTHER THREAD 3404
2017-01-09 17:15:39:  0: STDOUT: +0x3fa509fa [NtDelayExecution]
2017-01-09 17:15:39:  0: STDOUT: +0x3ce1121a [SleepEx]
2017-01-09 17:15:39:  0: STDOUT: +0x5210fdf1 [UTplaybackObjectStaticInit::operator=]
2017-01-09 17:15:39:  0: STDOUT: +0x5239ae5d [UT_Thread::threadWrapper]
2017-01-09 17:15:39:  0: STDOUT: +0x5239b339 [UT_Thread::getMyThreadId]
2017-01-09 17:15:39:  0: STDOUT: +0x24e33fef [beginthreadex]
2017-01-09 17:15:39:  0: STDOUT: +0x24e34196 [endthreadex]
2017-01-09 17:15:39:  0: STDOUT: +0x3d8413d2 [BaseThreadInitThunk]
2017-01-09 17:15:39:  0: STDOUT: +0x3f9d54e4 [RtlUserThreadStart]
2017-01-09 17:15:39:  0: STDOUT: OTHER THREAD 10044
2017-01-09 17:15:39:  0: STDOUT: +0x3fa5074a [ZwRemoveIoCompletion]
2017-01-09 17:15:39:  0: STDOUT: +0x3c32c949 [Tcpip4_WSHGetWildcardSockaddr]
2017-01-09 17:15:39:  0: STDOUT: +0x3d8413d2 [BaseThreadInitThunk]
2017-01-09 17:15:39:  0: STDOUT: +0x3f9d54e4 [RtlUserThreadStart]
2017-01-09 17:15:39:  0: STDOUT: -- TRACEBACK END --
2017-01-09 17:15:39:  0: STDOUT: Error allocating memory!Error allocating memory!
2017-01-09 17:15:39:  0: INFO: Process exit code: 139
2017-01-09 17:15:39:  0: Done executing plugin command of type 'Render Task'
2017-01-09 17:15:39:  0: An exception occurred: Error: Renderer returned non-zero error code, 139. Check the log for more information.
2017-01-09 17:15:39:     at Deadline.Plugins.PluginWrapper.RenderTasks(String taskId, Int32 startFrame, Int32 endFrame, String& outMessage, AbortLevel& abortLevel) (Deadline.Plugins.RenderPluginException)

Nick_Weeden · January 10, 2017, 3:41pm

Hi Edwin,

The error doesn’t always appear to be the same, you’re getting the same 139 exit code though. Does that ifd render through the command line?

Here’s the two main errors on my latest job:

2016-12-23 14:17:05: 0: STDOUT: [14:17:05] Registering procedural ‘ptreplicate’:
2016-12-23 14:17:05: 0: STDOUT: [14:17:05] Registering procedural ‘agent’:
2016-12-23 14:17:05: 0: STDOUT: [14:17:05] mantra Version 15.0.313 (Compiled on Dec 3 2015)
2016-12-23 14:17:05: 0: STDOUT: [14:17:05] User defined global attribute ‘global:fps’ ‘real’[1]
2016-12-23 14:17:05: 0: STDOUT: [14:17:05] reading geometry from stdin
2016-12-23 14:17:05: 0: STDOUT: [14:17:05] mantra: Error loading geometry /obj/logo/merge2 from stdin
2016-12-23 14:17:06: 0: INFO: Process exit code: 1
2016-12-23 14:17:06: 0: Done executing plugin command of type ‘Render Task’
2016-12-23 14:17:06: 0: An exception occurred: Error: Renderer returned non-zero error code, 1. Check the log for more information.
2016-12-23 14:17:06: at Deadline.Plugins.PluginWrapper.RenderTasks(String taskId, Int32 startFrame, Int32 endFrame, String& outMessage, AbortLevel& abortLevel) (Deadline.Plugins.RenderPluginException)

Nick

eamsler · January 10, 2017, 3:42pm

Welp, Windows got further, but it’s crashed some 4,300 times to get to frame 250ish. All of my errors are the memory allocation problem, so I’ll move it to the beefiest machine I have access to and see where it goes from there

I get the exit 139 lots, but not one instance of exit code 1 yet.

Nick_Weeden · January 10, 2017, 3:48pm

That’s interesting. I didn’t think there was a logic to the erroring regarding memory but I did notice some weirdness. When the tasks work at max they peaked at 600MB of memory. The failed jobs peaked at anywhere from 1-35GB.

Nick

Nick_Weeden · January 19, 2017, 12:24pm

Hi Edwin,

Just wondering how you were getting on with this and if the file is failing in a way that you can use to resolve the problem…?

Thanks

Nick

eamsler · January 20, 2017, 3:28pm

Sorry, didn’t get to this yesterday. Hit some fires this week, so thanks again for poking me.

At this point, I have no idea why Houdini is throwing these memory errors. I think given the post over at forums.thinkboxsoftware.com/vie … 11&t=15076, I might need to downgrade my machine to something much earlier and see if it makes a difference. The test should only be 30 minutes, I just need to get a block of time. My day’s compressed today, but Monday should be business as usual!

antoinedurr · January 22, 2017, 10:25am

Have you tried turning off Deadline’s drive letter mapping? Maybe it’s seeing Z: or something like that in the binary portion of the data and (unwisely) modifying it.

eamsler · January 25, 2017, 3:57pm

So, update from the other thread, it seems that we might be reading the binary IFDs in as ASCII and writing them out as UTF8 after an update someone did. That’d explain weird behavour since any byte bigger than 127 would take up two bytes and be completely unreadable for whatever was parsing it in. That doesn’t really explain why it worked at all though.

Are you using path mapping, and if so, can you try turning it off? Might break in new ways then (hopefully with a path not found and not this stdin business).

To talk about the problem a little more, the plan is to change the core code to try and match the output encoding to the import encoding.