AWS Thinkbox Discussion Forums

No feedback for Fusion jobs

Hello guys

Since upgrading to deadline 8 it seems we lost feedback from Fusion Jobs

We should get the progress report within the last ‘STDOUT:’ and ‘INFO: Process exit code: 0’
but nothing shows up

I parsed the plugin code a bit but it does not seem to have changed that much from deadline 7
Any idea ?

Same for any Fusion version from 6.4 to 8.1 btw

[code]=======================================================
Log

2016-09-05 10:35:35: BEGIN - WS341\render
2016-09-05 10:35:35: 0: Start Job timeout is disabled.
2016-09-05 10:35:35: 0: Task timeout is 1800 seconds (Regular Task Timeout)
2016-09-05 10:35:35: 0: Plugin rendering frame(s): 1047
2016-09-05 10:35:36: 0: Executing plugin command of type ‘Render Task’
2016-09-05 10:35:36: 0: INFO: Stdout Redirection Enabled: True
2016-09-05 10:35:36: 0: INFO: Stdout Handling Enabled: True
2016-09-05 10:35:36: 0: INFO: Popup Handling Enabled: False
2016-09-05 10:35:36: 0: INFO: Using Process Tree: True
2016-09-05 10:35:36: 0: INFO: Hiding DOS Window: False
2016-09-05 10:35:36: 0: INFO: Creating New Console: False
2016-09-05 10:35:36: 0: INFO: Running as user: render
2016-09-05 10:35:36: 0: INFO: Executable: “C:\Program Files\eyeon\Render Slave 6.4\ConsoleSlave.exe”
2016-09-05 10:35:36: 0: INFO: Argument: “P:\projects\deep\work\shots\ACT02\SQ0310\SH0070\comp\comp\deep_ACT02_SQ0310_SH0070_comp-comp_l_v036_EM.comp” /render /start 1047 /end 1047 /step 1 /verbose /quiet /quietlicense /clean /quit
2016-09-05 10:35:36: 0: INFO: Full Command: “C:\Program Files\eyeon\Render Slave 6.4\ConsoleSlave.exe” “P:\projects\deep\work\shots\ACT02\SQ0310\SH0070\comp\comp\deep_ACT02_SQ0310_SH0070_comp-comp_l_v036_EM.comp” /render /start 1047 /end 1047 /step 1 /verbose /quiet /quietlicense /clean /quit
2016-09-05 10:35:36: 0: INFO: Startup Directory: “C:\Program Files\eyeon\Render Slave 6.4”
2016-09-05 10:35:36: 0: INFO: Process Priority: BelowNormal
2016-09-05 10:35:36: 0: INFO: Process Affinity: default
2016-09-05 10:35:36: 0: INFO: Process is now running
2016-09-05 10:35:36: 0: STDOUT: WARNING! XML Element effect has unknown attributes 'Unknown attributes ‘cstring’
2016-09-05 10:35:36: 0: STDOUT: WARNING! XML Element param has unknown attributes 'Unknown attributes ‘position’
2016-09-05 10:35:36: 0: STDOUT: WARNING! XML Element effect has unknown attributes 'Unknown attributes ‘cstring’
2016-09-05 10:37:02: 0: INFO: Process exit code: 0
2016-09-05 10:37:02: 0: Done executing plugin command of type ‘Render Task’
2016-09-05 10:37:02: 0: Minimum required render time is 30 seconds
2016-09-05 10:37:02: 0: Actual render time was 87 seconds

=======================================================
Details

Date: 09/05/2016 10:37:03
Frames: 1047
Job Submit Date: 09/05/2016 10:30:31
Job User: eren.ma
Average RAM Usage: 6906980864 (11%)
Peak RAM Usage: 19063111680 (28%)
Average CPU Usage: 19%
Peak CPU Usage: 81%
Used CPU Clocks (x10^6 cycles): 1488580
Total CPU Clocks (x10^6 cycles): 7834628

=======================================================
Slave Information

Slave Name: ws341
Version: v8.0.7.3 Release (f33fcb7d3)
Operating System: Windows 7 Professional (SP1)
Running As Service: No
Machine User: render
IP Address: 10.200.34.81
MAC Address: DC:4A:3E:70:37:30
CPU Architecture: x64
CPUs: 40
CPU Usage: 30%
Memory Usage: 39.9 GB / 63.9 GB (62%)
Free Disk Space: 94.061 GB (94.061 GB on C:)
Video Card: NVIDIA Quadro M4000
[/code]

Hmm. That’s interesting… We might suppressing output lines but somehow miscalculating things. Standard output would be great here.

Can you run it from the command line and throw the whole thing back to us? You can do that like so:

"C:\Program Files\eyeon\Render Slave 6.4\ConsoleSlave.exe" "P:\projects\deep\work\shots\ACT02\SQ0310\SH0070\comp\comp\deep_ACT02_SQ0310_SH0070_comp-comp_l_v036_EM.comp" /render /start 1047 /end 1047 /step 1 /verbose /quiet /quietlicense /clean /quit > output.txt
explorer .

That should write a file called “output.txt” and open an explorer window showing it. Just attach it here and we’ll take a look see.

I did get the Warning XML (…) in the command line itself

---------------------------------------------------
Starting Fusion Render Slave x64 6.4 build 1118 at 06/Sep/16 13:54:25
C:\Program Files\eyeon\Render Slave 6.4\ConsoleSlave.exe
---------------------------------------------------
Checking CPU capabilities... OK
Checking OS Version... OK
Initialising QuickTime
Checking for licenses...
Obtain License
Updating Registry
Obtaining System Information
Loading support DLLs
Loading eyeonOperators
Loading eyeonFormats
Loading RichEdit
Loading Plugins
Loading plugin C:\Program Files\eyeon\Render Slave 6.4\Plugins\AT\BilateralFilter.plugin
Loading plugin C:\Program Files\eyeon\Render Slave 6.4\Plugins\eyeon\3D.plugin
Loading plugin C:\Program Files\eyeon\Render Slave 6.4\Plugins\eyeon\Bins.plugin
The Bins.plugin plugin requires version 6.4.0.1118 of Fusion x64.
(This is version 6.4.0.1118 of Fusion Render Slave x64)Loading plugin C:\Program Files\eyeon\Render Slave 6.4\Plugins\eyeon\CinemaRaw\CinemaRaw.plugin
Loading plugin C:\Program Files\eyeon\Render Slave 6.4\Plugins\eyeon\ContextMenu.plugin
The ContextMenu.plugin plugin requires version 6.4.0.1118 of Fusion x64.
(This is version 6.4.0.1118 of Fusion Render Slave x64)Loading plugin C:\Program Files\eyeon\Render Slave 6.4\Plugins\eyeon\Dimension.plugin
Loading plugin C:\Program Files\eyeon\Render Slave 6.4\Plugins\eyeon\DirectShow.plugin
Loading plugin C:\Program Files\eyeon\Render Slave 6.4\Plugins\eyeon\FBX.plugin
Loading plugin C:\Program Files\eyeon\Render Slave 6.4\Plugins\eyeon\Fuses.plugin
Loading plugin C:\Program Files\eyeon\Render Slave 6.4\Plugins\eyeon\OMF.plugin
Loading plugin C:\Program Files\eyeon\Render Slave 6.4\Plugins\eyeon\OpenEXR.plugin
Loading plugin C:\Program Files\eyeon\Render Slave 6.4\Plugins\eyeon\OpenFX.plugin
Loading plugin C:\Program Files\eyeon\Render Slave 6.4\Plugins\eyeon\Paint.plugin
Loading plugin C:\Program Files\eyeon\Render Slave 6.4\Plugins\eyeon\Particles.plugin
Loading plugin C:\Program Files\eyeon\Render Slave 6.4\Plugins\eyeon\Primatte4.plugin
Loading plugin C:\Program Files\eyeon\Render Slave 6.4\Plugins\eyeon\QuickTime.plugin
Loading plugin C:\Program Files\eyeon\Render Slave 6.4\Plugins\eyeon\Text.plugin
Loading plugin C:\Program Files\eyeon\Render Slave 6.4\Plugins\eyeon\Utilities.plugin
Loading plugin C:\Program Files\eyeon\Render Slave 6.4\Plugins\Krokodove\Krokodove.plugin
Loading plugin C:\Program Files\eyeon\Render Slave 6.4\Plugins\LB_Tools\LB_Toolsx64.plugin
Loading plugin C:\Program Files\eyeon\Render Slave 6.4\Plugins\PointCacheLoader\PointCacheLoader.plugin
Loading plugin C:\Program Files\eyeon\Render Slave 6.4\Plugins\RSMB5ForFusion\RSMB_64.plugin
Loading plugin C:\Program Files\eyeon\Render Slave 6.4\Plugins\Twixtor6ForFusion\Twixtor_64.plugin
Loading plugin C:\Program Files\eyeon\Render Slave 6.4\Plugins\Twixtor6ForFusion\VectorCreate_64.plugin
Initialising Class Registry
Building Format List
Creating MainFrame
Initialising GLTools
Initialising Texture Manager
Adding Global App Events
Registering System Sounds
Initialising Scripting Subsystem
Loading Comp at P:\projects\deep\work\shots\ACT02\SQ0310\SH0070\comp\comp\deep_ACT02_SQ0310_SH0070_comp-comp_l_v036_EM.comp
Rendering Comp, frames 1047, step 1

Listening for commands...

Render started at Tue 1:54PM  (Range: 1047 to 1047)
Rendered frame 1047 (1 of 1), took 101.964 secs
Rendered frame 1047 (1 of 1), took 0.343 secs
Render completed successfully at Tue 1:56PM - Total Time: 0h 0m 0.34s, Average: 2.92 frames/second
Auto-exiting with errcode 0
Closing outstanding documents
Cleanup licenses

Hmm… We’re definitely eating that progress data too. Usually that’s a good thing as it’ll cut down on log size, but we must not be doing the calculations properly.

Update: I did some quick digging and the only lines we should be supressing are those which contain this:

"Progress: ([0-9]+) %"

I don’t know how or why that’s being suppressed here.

Just so I have something more to go on, the main complaint is that you’re not getting progress information on the tasks when doing multiple frames per task, or would you want to see the output in the log?

If you look for “self.SuppressThisLine()”, that is where we’re excluding information from the Slave log. Feel free to try commenting them out to see if it makes a difference.

Also, what build of 8 are you on?

Well

There’s a lack of progress report which is annoying but can be dealt with
The main problem is that there’s no catch on errors,
e.g. if a frame fails to write, it’s not failing the job

and actually getting full report is also good when trying to debug jobs acting weirdly

Also on top of that I had to add the following code at line 157 of FusionCmd.py plugin file
It so happens (at least in our setup) that Fusion sometimes exits with this error code while it succeeded renders, no clue why

elif exitCode == -1073741819: return

We’re using Deadline 8.0.7.3
Edit ; Just saw 8.0.8.1 is out
Will try to upgrade today
/
8.0.8.1 rolled out on Repository/Pulse/Windows rendering
Windows/Linux Clients/Linux Rendering on their way
Issue still there for Fusion Winx64

Hmm. The strange thing here is that Deadline is catching some of the log, but not all of it.

The only guess I have for this is that maybe Fusion writes that output to STDERR and not STDOUT and somehow we’re not catching that like we always have. When you pasted the command line output, did you use the pipe option I sent along, or did you copy from the command prompt?

Also, this seems like it’s turning into one of those “call to troubleshoot” problems. Here’s what I want to do: (in case you’d rather run with it)

  1. Take that command line run that gave all the output and feed it through the Command Line plugin in the Monitor

This would make sure it’s not a Fusion plugin problem. Depending on that, we’d investigate a different angle.

  1. Try piping that render text to two different files:
"C:\Program Files\eyeon\Render Slave 6.4\ConsoleSlave.exe" "P:\projects\deep\work\shots\ACT02\SQ0310\SH0070\comp\comp\deep_ACT02_SQ0310_SH0070_comp-comp_l_v036_EM.comp" /render /start 1047 /end 1047 /step 1 /verbose /quiet /quietlicense /clean /quit 1> stdout.txt 2> stderr.txt

That’ll make two files and filter out the different streams to stdout.txt and stderr.txt.

This all feels like a stream redirect issue…

Hi,

We are experiencing the same kind of problem.

We are using the comand line Fusion plugin. Normal plugin is not working at all.

Deadline receives a -1073741819 returncode which should refer to a memory problem but I think that is very unlikely. By adding this code in the checkexitcode() part the job continues.

The main problem is that Deadline is not receiving the standard exitcode but always receives a -1073741819 code

In the attached log you can see that I have some bad frames (I have to look into that later) which are caused by a input mov file.

The exitcode fusion throws is 10 which s great because it shows things went wrong. Deadline receives the -1073741819 code always. So my guess is that deadline is somehow not receiving a proper errorcode anymore.

If all renders are ok there is no problem but I want to detect the wrongly rendered frames.

Thanks for your attention!

Patrick

Deadline Version: 8.0.9.1
Fusion Version: 8.2 Build 2 (2016-09-13 no BETA)
FusionLog.txt (76.9 KB)

The exit code hasn’t really gone wrong before this. I can only see possible problems if you have ‘render as user’ enabled in the user security settings.

I think we’ll need to include an option to ignore that exit code… Any issues with that idea from anyone?

Thanks for the reply.

I would have no problem with ignoring that exit code. But I would like to see the exitcode that fusion returned. There are already some exitcodes within the scripts that seems to be right (errorcode 10 for instance)

It would be helpfull if the ‘right’ exitcode will be used so we can see if a comp goes wrong.

Best,

Patrick

Deadline has never used the wrong exit code before. At least not that I can remember. It’s built into the operating system, so not much can go wrong without ‘run as user’.

Can you try running it from the command line and then running “echo %ERRORLEVEL%” immediately after? That should show whatever the true exit code was from the application. The error code from the program output though might be entirely different. We could just look for the text “Auto-exiting with errcode (\d+)” and that would return the number from that line.

Info on running from the command line is here:
docs.thinkboxsoftware.com/produc … oting.html

Maybe we can check for both? What errcodes should we check for?

Hi Edwin,

Sorry for the silence. The %errorlevel% is -1073741819. Always. I checked with a batch job. So the command line version of fusion probably only returns that code. Which is a shame. I have attached a somewhat edited output.txt of the command line output.

As you can see Fusion reports an exiting with code 0 but somehow it doesn’t. I also read that the errorcode -1073741819 might have something to do with the Open CL drivers but I installed the latest version of them but it made no difference.

I am not sure if you can get the verbose output and check for this part.

Render completed successfully at Thu 4:16PM - Total Time: 0h 0m 5.63s, Average: 1.41 seconds/frame
Auto-exiting with errcode 0
Cleanup licenses

I hope this can be solved because if something goes wrong now the tasks will still be flagged as complete.

Best regards and thanks for your help!

Patrick
Output.txt (3.01 KB)

Okay! I’ll make a Git issue for that and we’ll hopefully be able to get something out for you either in the 8.0 service pack or 8.1 (not sure which will be out next). As usual, I’ll try to report back once it’s been implemented.

Hi Edwin,

Thanks! Looking forward to the update.

It would be great if the team can also fix the normal (non commandline) plugin of Fusion. This has been mentioned before. The process is unable to kill the fuscript.exe becasue of a rights issue.

Best regards,

Patrick

Well, I just got feedback that the exit code problem is solved in SP12. Not sure on the standard Fusion plugin. Can you upgrade and try both again?

Digging back an old thread, I’ve been busy around and did not have time to provide further information

It seems both Fusion and FusionCmd plugins are inverting Stderr and Stdout

Provided are the logs and command used to obtain them

I confirm the bug is still present in 8.0.13
STDERR.txt (249 Bytes)
STDOUT.TXT (3.54 KB)
COMMAND.txt (300 Bytes)

The output seems alright. I’d expect non-normal operations like warnings and errors to come through STDERR and the rest to go to STDOUT. The good news is Deadline mushes them both into the same stream handling functions, so for us it doesn’t matter much. The good news is that SP12 added the “Exit code -1073741819 is no longer treated as an error during rendering.”, so you should be in good shape with Fusion now in SP13. Are you getting new errors?

Actually yes

To be clear I only see the STDERR catched by deadline

The STDOUT is just nowhere to be found
this poses problem as there is no progress report, but the most problematic is that it’s not catching on errors because of that

So, after digging into our issue tracker this may have been caused by a bad auto-upgrade. Would you be able to uninstall and re-install Deadline? We’d want to clear out the ‘bin’ directory before re-installing. The guess is that there may be a bad DLL.

If that makes no difference, it might be related to a different problem that we’re working on this week.

1649 jobs queued, I guess the timing’s not right :slight_smile:

and if we’re talking about reinstalling the client, I actually do not go for autoupgrade

I got a script manually installing deadline each time a new release is pushed

Also the problem is present on all 330 nodes …

Privacy | Site terms | Cookie preferences