Maxwell Render delete temp files error

[Tim_Ellis]

If a render has specific passes selected within the scene, Deadline will not delete the slave versions on completion or extract those passes from the final render.

For example, I have these render channels selected in my Maxwell scene:-

Diffuse and reflections.
Alpha
Material
Object
Normal

On job completion, the mxi files created by each slave are copied back to the specified render path and merged. When merging is complete, the intermediate files are deleted as expected, but this is not true for render channels.

eg for two co-operative nodes after job has completed and merged:-

teapot_flush.mxi
teapot_flush.png
teapot_flush_s_material_0001.png
teapot_flush_s_material_0002.png
teapot_flush_s_object_0001.png
teapot_flush_s_object_0002.png
teapot_flush_s_alpha_0001.png
teapot_flush_s_alpha_0002.png
etc.

Ideally these files should be removed when the ‘delete intermediate files’ flag is checked, when the temp mxi’s and diffuse channels are removed by Deadline.
Then ideally any additional passes should be extracted from the mxi when the final colour output is produced.

This is from the Maxwell knowledge base about extracting all channels from an mxi:- http://support.nextlimit.com/display/plugdevspace/CmaxwellMxi#CmaxwellMxi-extractchannels

extractChannels                    Extracts all the channels embeded in the MXI to disk.

Language                    Syntax

C++                           bool extractChannels( const char* fileName, const char* destinationFolder, const char* extension, const byte& depth ) 
Python                      bool extractChannels( str fileName, str destinationFolder, str extension, int depth ) 

Parameters

Type                       Name               In/Out               Description
const char*          filename               in                     File name (not full path) of the destination files 
const char*         destinationFolder       in                     Folder where the channels will be saved. 
const char*         extension               in                     File extension of the saved channels. 
const byte&        depth                    in                     Channels bitDepth. 

Return value

false = Error saving channels.
true = Success.

I’m about to start testing a Python application of this function, added to the standard Maxwell.py submission script to see if I can get any results.

Build 18 Deadline Version: 6.0.0.51030

Tim.

[Tim_Ellis]

I’ve solved the extracting render channels problem:-

Add the following two lines to Maxwell.py

arguments += " -extractchannels:"" + OutputFile + “”"
here:-

Line #
360	if len( outputFile ) > 0:
361	arguments += " -image:\"" + outputFile + "\""
362	arguments += " -extractchannels:\"" + outputFile + "\""

and
arguments += " -extractchannels:"" + currOutputFile + “”"
here:-

Line #
398	currOutputFile = Path.Combine( directory, filename + "_" + filenamePadding + extension )
399	arguments += " -image:\"" + currOutputFile + "\""
400	arguments += " -extractchannels:\"" + currOutputFile + "\""

This extracts all render channels to the output folder, when the merge process has completed.

Not sure about deleting the remaining files in my first post yet.

Tim.

[rrussell]

Hi Tim,

Thanks for the code! Do you think the -extractchannels option should always be used, or should we maybe expose a checkbox to the submitter to control if it’s used or not?

Cheers,

• Ryan

[Tim_Ellis]

Hi Tim,

Thanks for the code! Do you think the -extractchannels option should always be used, or should we maybe expose a checkbox to the submitter to control if it’s used or not?

Cheers,

• Ryan

Maxwell default network render system will always extract the channels on completion and single renders always extract the files during render. So I would say yes, they should always be extracted.

Otherwise it’s an extra step to open the MXI once rendered and then save out each channel. If rendering an animation, then I’d expect the channels to be extracted when rendered, otherwise each mxi has to be loaded and channels saved out. As you can imagine, this would be a lengthy process on large frame ranges.

I don’t think the end user would need to specify it in the submission script/dialogue. They would have already specified which channels should be rendered when creating the scene. I vote for always on.

Tim.

[rrussell]

Thanks! We’ll add the -extractchannels argument for beta 19.

Cheers,

• Ryan

[Tim_Ellis]

Hi Ryan,

I’ve been using the new -extractchannels argument with consistent success, since implementing it into the code. Very happy to hear it will be included in the next beta release.

However, I still have many image files for the channels, which are returned to the output folder by each slave on job completion and these are not being removed automatically.

Can you suggest anything to remove these extra images? I’ve tried changing the ‘_s’ naming variables in the hope that these files would then be picked up and deleted, but it did not make any difference.

The individual slave mxi and full colour passes are deleted correctly once the files are merged, just not the render channels.

I’ll look into it a bit more in the meantime, to see if I can isolate or add anything that works.

Cheers,

Tim.

[rrussell]

The issue with deleting those extra files is that currently the Maxwell plugin isn’t aware of them. It’s just aware of the original output file. You can take a look at the PostRenderTasks function in Maxwell.py to see what we’re currently doing.

Maybe we could do some pattern matching to look for files ending with _s#### in the output folder and deleting them. What do you think?

[Tim_Ellis]

HI Ryan,

I’ve implemented additional arguments in the postrendertask section. I’m testing this now and will post results soon.

Cheers,

Tim.

[Tim_Ellis]

Ok Ryan, here you are. Working, tested and confirmed over four separate jobs.

This replaces the full postrendertasks section

	def PostRenderTasks( self ):
		if self.MergeJob and self.GetBooleanPluginInfoEntryWithDefault( "DeleteFiles", False ):
			outputFile = self.GetPluginInfoEntryWithDefault( "OutputFile", "" ).strip()
			outputFile = RepositoryUtils.CheckPathMapping( outputFile )
			outputFile = self.FixPathSeparators( outputFile )
			
			coopJobs = self.GetIntegerPluginInfoEntryWithDefault( "CoopJobs", 0 )
			if self.GetBooleanPluginInfoEntry( "SingleFile" ):
				self.LogInfo( "Deleting intermediate files" )
				for seedFile in self.MxiSeedFiles:
					self.LogInfo( "  Deleting " + seedFile )
					File.Delete( seedFile )
				
				if len( outputFile ) > 0:
					for seed in range( 1, coopJobs + 1 ):
						padding = StringUtils.ToZeroPaddedString( seed, 4, False )
						directory = Path.GetDirectoryName( outputFile )
						filename = Path.GetFileNameWithoutExtension( outputFile )
						extension = Path.GetExtension( outputFile )
						
						seedOutputFile = Path.Combine( directory, filename + "_s" + padding + extension )
						self.LogInfo( "  Deleting " + seedOutputFile )
						File.Delete( seedOutputFile ) 

				if len( outputFile ) > 0:
					for seed in range( 1, coopJobs + 1 ):
						padding = StringUtils.ToZeroPaddedString( seed, 4, False )
						directory = Path.GetDirectoryName( outputFile )
						filename = Path.GetFileNameWithoutExtension( outputFile )
						extension = Path.GetExtension( outputFile )
						
						seedOutputFile = Path.Combine( directory, filename + "_s_material_" + padding + extension )
						self.LogInfo( "  Deleting " + seedOutputFile )
						File.Delete( seedOutputFile ) 

				if len( outputFile ) > 0:
					for seed in range( 1, coopJobs + 1 ):
						padding = StringUtils.ToZeroPaddedString( seed, 4, False )
						directory = Path.GetDirectoryName( outputFile )
						filename = Path.GetFileNameWithoutExtension( outputFile )
						extension = Path.GetExtension( outputFile )
						
						seedOutputFile = Path.Combine( directory, filename + "_s_object_" + padding + extension )
						self.LogInfo( "  Deleting " + seedOutputFile )
						File.Delete( seedOutputFile ) 

				if len( outputFile ) > 0:
					for seed in range( 1, coopJobs + 1 ):
						padding = StringUtils.ToZeroPaddedString( seed, 4, False )
						directory = Path.GetDirectoryName( outputFile )
						filename = Path.GetFileNameWithoutExtension( outputFile )
						extension = Path.GetExtension( outputFile )
						
						seedOutputFile = Path.Combine( directory, filename + "_s_alpha_" + padding + extension )
						self.LogInfo( "  Deleting " + seedOutputFile )
						File.Delete( seedOutputFile ) 

				if len( outputFile ) > 0:
					for seed in range( 1, coopJobs + 1 ):
						padding = StringUtils.ToZeroPaddedString( seed, 4, False )
						directory = Path.GetDirectoryName( outputFile )
						filename = Path.GetFileNameWithoutExtension( outputFile )
						extension = Path.GetExtension( outputFile )
						
						seedOutputFile = Path.Combine( directory, filename + "_s_fresnel_" + padding + extension )
						self.LogInfo( "  Deleting " + seedOutputFile )
						File.Delete( seedOutputFile ) 

				if len( outputFile ) > 0:
					for seed in range( 1, coopJobs + 1 ):
						padding = StringUtils.ToZeroPaddedString( seed, 4, False )
						directory = Path.GetDirectoryName( outputFile )
						filename = Path.GetFileNameWithoutExtension( outputFile )
						extension = Path.GetExtension( outputFile )
						
						seedOutputFile = Path.Combine( directory, filename + "_s_normals_" + padding + extension )
						self.LogInfo( "  Deleting " + seedOutputFile )
						File.Delete( seedOutputFile ) 

				if len( outputFile ) > 0:
					for seed in range( 1, coopJobs + 1 ):
						padding = StringUtils.ToZeroPaddedString( seed, 4, False )
						directory = Path.GetDirectoryName( outputFile )
						filename = Path.GetFileNameWithoutExtension( outputFile )
						extension = Path.GetExtension( outputFile )
						
						seedOutputFile = Path.Combine( directory, filename + "_s_position_" + padding + extension )
						self.LogInfo( "  Deleting " + seedOutputFile )
						File.Delete( seedOutputFile ) 

				if len( outputFile ) > 0:
					for seed in range( 1, coopJobs + 1 ):
						padding = StringUtils.ToZeroPaddedString( seed, 4, False )
						directory = Path.GetDirectoryName( outputFile )
						filename = Path.GetFileNameWithoutExtension( outputFile )
						extension = Path.GetExtension( outputFile )
						
						seedOutputFile = Path.Combine( directory, filename + "_s_roughness_" + padding + extension )
						self.LogInfo( "  Deleting " + seedOutputFile )
						File.Delete( seedOutputFile ) 

				if len( outputFile ) > 0:
					for seed in range( 1, coopJobs + 1 ):
						padding = StringUtils.ToZeroPaddedString( seed, 4, False )
						directory = Path.GetDirectoryName( outputFile )
						filename = Path.GetFileNameWithoutExtension( outputFile )
						extension = Path.GetExtension( outputFile )
						
						seedOutputFile = Path.Combine( directory, filename + "_s_zbuffer_" + padding + extension )
						self.LogInfo( "  Deleting " + seedOutputFile )
						File.Delete( seedOutputFile ) 

				if len( outputFile ) > 0:
					for seed in range( 1, coopJobs + 1 ):
						padding = StringUtils.ToZeroPaddedString( seed, 4, False )
						directory = Path.GetDirectoryName( outputFile )
						filename = Path.GetFileNameWithoutExtension( outputFile )
						extension = Path.GetExtension( outputFile )
						
						seedOutputFile = Path.Combine( directory, filename + "_s_motion_" + padding + extension )
						self.LogInfo( "  Deleting " + seedOutputFile )
						File.Delete( seedOutputFile ) 

				if len( outputFile ) > 0:
					for seed in range( 1, coopJobs + 1 ):
						padding = StringUtils.ToZeroPaddedString( seed, 4, False )
						directory = Path.GetDirectoryName( outputFile )
						filename = Path.GetFileNameWithoutExtension( outputFile )
						extension = Path.GetExtension( outputFile )
						
						seedOutputFile = Path.Combine( directory, filename + "_s_shadow_" + padding + extension )
						self.LogInfo( "  Deleting " + seedOutputFile )
						File.Delete( seedOutputFile ) 

			else:
				frame = self.GetStartFrame()
				filenamePadding = StringUtils.ToZeroPaddedString( frame, 4, False )
				
				self.LogInfo( "Deleting intermediate files for frame " + str(frame) )
				for seedFile in self.MxiSeedFiles:
					self.LogInfo( "  Deleting " + seedFile )
					File.Delete( seedFile )
				
				if len( outputFile ) > 0:
					for seed in range( 1, coopJobs + 1 ):
						padding = StringUtils.ToZeroPaddedString( seed, 4, False )
						directory = Path.GetDirectoryName( outputFile )
						filename = Path.GetFileNameWithoutExtension( outputFile )
						extension = Path.GetExtension( outputFile )
						
						seedOutputFile = Path.Combine( directory, filename + "_s" + padding + "_." + filenamePadding + extension )
						self.LogInfo( "  Deleting " + seedOutputFile )
						File.Delete( seedOutputFile )

				if len( outputFile ) > 0:
					for seed in range( 1, coopJobs + 1 ):
						padding = StringUtils.ToZeroPaddedString( seed, 4, False )
						directory = Path.GetDirectoryName( outputFile )
						filename = Path.GetFileNameWithoutExtension( outputFile )
						extension = Path.GetExtension( outputFile )
						
						seedOutputFile = Path.Combine( directory, filename + "_s_material_" + padding + extension )
						self.LogInfo( "  Deleting " + seedOutputFile )
						File.Delete( seedOutputFile ) 

				if len( outputFile ) > 0:
					for seed in range( 1, coopJobs + 1 ):
						padding = StringUtils.ToZeroPaddedString( seed, 4, False )
						directory = Path.GetDirectoryName( outputFile )
						filename = Path.GetFileNameWithoutExtension( outputFile )
						extension = Path.GetExtension( outputFile )
						
						seedOutputFile = Path.Combine( directory, filename + "_s_object_" + padding + extension )
						self.LogInfo( "  Deleting " + seedOutputFile )
						File.Delete( seedOutputFile ) 

				if len( outputFile ) > 0:
					for seed in range( 1, coopJobs + 1 ):
						padding = StringUtils.ToZeroPaddedString( seed, 4, False )
						directory = Path.GetDirectoryName( outputFile )
						filename = Path.GetFileNameWithoutExtension( outputFile )
						extension = Path.GetExtension( outputFile )
						
						seedOutputFile = Path.Combine( directory, filename + "_s_alpha_" + padding + extension )
						self.LogInfo( "  Deleting " + seedOutputFile )
						File.Delete( seedOutputFile ) 

				if len( outputFile ) > 0:
					for seed in range( 1, coopJobs + 1 ):
						padding = StringUtils.ToZeroPaddedString( seed, 4, False )
						directory = Path.GetDirectoryName( outputFile )
						filename = Path.GetFileNameWithoutExtension( outputFile )
						extension = Path.GetExtension( outputFile )
						
						seedOutputFile = Path.Combine( directory, filename + "_s_fresnel_" + padding + extension )
						self.LogInfo( "  Deleting " + seedOutputFile )
						File.Delete( seedOutputFile ) 

				if len( outputFile ) > 0:
					for seed in range( 1, coopJobs + 1 ):
						padding = StringUtils.ToZeroPaddedString( seed, 4, False )
						directory = Path.GetDirectoryName( outputFile )
						filename = Path.GetFileNameWithoutExtension( outputFile )
						extension = Path.GetExtension( outputFile )
						
						seedOutputFile = Path.Combine( directory, filename + "_s_normals_" + padding + extension )
						self.LogInfo( "  Deleting " + seedOutputFile )
						File.Delete( seedOutputFile ) 

				if len( outputFile ) > 0:
					for seed in range( 1, coopJobs + 1 ):
						padding = StringUtils.ToZeroPaddedString( seed, 4, False )
						directory = Path.GetDirectoryName( outputFile )
						filename = Path.GetFileNameWithoutExtension( outputFile )
						extension = Path.GetExtension( outputFile )
						
						seedOutputFile = Path.Combine( directory, filename + "_s_position_" + padding + extension )
						self.LogInfo( "  Deleting " + seedOutputFile )
						File.Delete( seedOutputFile ) 

				if len( outputFile ) > 0:
					for seed in range( 1, coopJobs + 1 ):
						padding = StringUtils.ToZeroPaddedString( seed, 4, False )
						directory = Path.GetDirectoryName( outputFile )
						filename = Path.GetFileNameWithoutExtension( outputFile )
						extension = Path.GetExtension( outputFile )
						
						seedOutputFile = Path.Combine( directory, filename + "_s_roughness_" + padding + extension )
						self.LogInfo( "  Deleting " + seedOutputFile )
						File.Delete( seedOutputFile ) 

				if len( outputFile ) > 0:
					for seed in range( 1, coopJobs + 1 ):
						padding = StringUtils.ToZeroPaddedString( seed, 4, False )
						directory = Path.GetDirectoryName( outputFile )
						filename = Path.GetFileNameWithoutExtension( outputFile )
						extension = Path.GetExtension( outputFile )
						
						seedOutputFile = Path.Combine( directory, filename + "_s_zbuffer_" + padding + extension )
						self.LogInfo( "  Deleting " + seedOutputFile )
						File.Delete( seedOutputFile ) 

				if len( outputFile ) > 0:
					for seed in range( 1, coopJobs + 1 ):
						padding = StringUtils.ToZeroPaddedString( seed, 4, False )
						directory = Path.GetDirectoryName( outputFile )
						filename = Path.GetFileNameWithoutExtension( outputFile )
						extension = Path.GetExtension( outputFile )
						
						seedOutputFile = Path.Combine( directory, filename + "_s_motion_" + padding + extension )
						self.LogInfo( "  Deleting " + seedOutputFile )
						File.Delete( seedOutputFile ) 

				if len( outputFile ) > 0:
					for seed in range( 1, coopJobs + 1 ):
						padding = StringUtils.ToZeroPaddedString( seed, 4, False )
						directory = Path.GetDirectoryName( outputFile )
						filename = Path.GetFileNameWithoutExtension( outputFile )
						extension = Path.GetExtension( outputFile )
						
						seedOutputFile = Path.Combine( directory, filename + "_s_shadow_" + padding + extension )
						self.LogInfo( "  Deleting " + seedOutputFile )
						File.Delete( seedOutputFile )

I’ve tried to keep it as clean as possible, but perhaps you know of a better way than this.

Tim.

[rrussell]

Awesome! Thanks so much for the code. We’ll include this cleanup stuff in the next beta.

I’ve cleaned up the code a bit to remove a bunch of duplicate code. I also created a DeleteSeedOutputFile function to help out. Here’s what we’ll be shipping in the next beta:

    def PostRenderTasks( self ):
        if self.MergeJob and self.GetBooleanPluginInfoEntryWithDefault( "DeleteFiles", False ):
            outputFile = self.GetPluginInfoEntryWithDefault( "OutputFile", "" ).strip()
            outputFile = RepositoryUtils.CheckPathMapping( outputFile )
            outputFile = self.FixPathSeparators( outputFile )
            
            coopJobs = self.GetIntegerPluginInfoEntryWithDefault( "CoopJobs", 0 )
            if self.GetBooleanPluginInfoEntry( "SingleFile" ):
                self.LogInfo( "Deleting intermediate files" )
                for seedFile in self.MxiSeedFiles:
                    self.DeleteSeedOutputFile( seedFile )
                
                if len( outputFile ) > 0:
                    for seed in range( 1, coopJobs + 1 ):
                        padding = StringUtils.ToZeroPaddedString( seed, 4, False )
                        directory = Path.GetDirectoryName( outputFile )
                        filename = Path.GetFileNameWithoutExtension( outputFile )
                        extension = Path.GetExtension( outputFile )
                        
                        self.DeleteSeedOutputFile( Path.Combine( directory, filename + "_s" + padding + extension ) )
                        self.DeleteSeedOutputFile( Path.Combine( directory, filename + "_s_material_" + padding + extension ) )
                        self.DeleteSeedOutputFile( Path.Combine( directory, filename + "_s_object_" + padding + extension ) )
                        self.DeleteSeedOutputFile( Path.Combine( directory, filename + "_s_alpha_" + padding + extension ) )
                        self.DeleteSeedOutputFile( Path.Combine( directory, filename + "_s_fresnel_" + padding + extension ) )
                        self.DeleteSeedOutputFile( Path.Combine( directory, filename + "_s_normals_" + padding + extension ) )
                        self.DeleteSeedOutputFile( Path.Combine( directory, filename + "_s_position_" + padding + extension ) )
                        self.DeleteSeedOutputFile( Path.Combine( directory, filename + "_s_roughness_" + padding + extension ) )
                        self.DeleteSeedOutputFile( Path.Combine( directory, filename + "_s_zbuffer_" + padding + extension ) )
                        self.DeleteSeedOutputFile( Path.Combine( directory, filename + "_s_motion_" + padding + extension ) )
                        self.DeleteSeedOutputFile( Path.Combine( directory, filename + "_s_shadow_" + padding + extension ) )                
            else:
                frame = self.GetStartFrame()
                filenamePadding = StringUtils.ToZeroPaddedString( frame, 4, False )
                
                self.LogInfo( "Deleting intermediate files for frame " + str(frame) )
                for seedFile in self.MxiSeedFiles:
                    self.DeleteSeedOutputFile( seedFile )
                
                if len( outputFile ) > 0:
                    for seed in range( 1, coopJobs + 1 ):
                        padding = StringUtils.ToZeroPaddedString( seed, 4, False )
                        directory = Path.GetDirectoryName( outputFile )
                        filename = Path.GetFileNameWithoutExtension( outputFile )
                        extension = Path.GetExtension( outputFile )
                        
                        self.DeleteSeedOutputFile( Path.Combine( directory, filename + "_s" + padding + "_." + filenamePadding + extension ) )
                        self.DeleteSeedOutputFile( Path.Combine( directory, filename + "_s_material_" + padding + "_." + filenamePadding + extension ) )
                        self.DeleteSeedOutputFile( Path.Combine( directory, filename + "_s_object_" + padding + "_." + filenamePadding + extension ) )
                        self.DeleteSeedOutputFile( Path.Combine( directory, filename + "_s_alpha_" + padding + "_." + filenamePadding + extension ) )
                        self.DeleteSeedOutputFile( Path.Combine( directory, filename + "_s_fresnel_" + padding + "_." + filenamePadding + extension ) )
                        self.DeleteSeedOutputFile( Path.Combine( directory, filename + "_s_normals_" + padding + "_." + filenamePadding + extension ) )
                        self.DeleteSeedOutputFile( Path.Combine( directory, filename + "_s_position_" + padding + "_." + filenamePadding + extension ) )
                        self.DeleteSeedOutputFile( Path.Combine( directory, filename + "_s_roughness_" + padding + "_." + filenamePadding + extension ) )
                        self.DeleteSeedOutputFile( Path.Combine( directory, filename + "_s_zbuffer_" + padding + "_." + filenamePadding + extension ) )
                        self.DeleteSeedOutputFile( Path.Combine( directory, filename + "_s_motion_" + padding + "_." + filenamePadding + extension ) )
                        self.DeleteSeedOutputFile( Path.Combine( directory, filename + "_s_shadow_" + padding + "_." + filenamePadding + extension ) )

    def DeleteSeedOutputFile( self, seedOutputFile ):
        if File.Exists( seedOutputFile ):
            self.LogInfo( "  Deleting " + seedOutputFile )
            File.Delete( seedOutputFile )

Cheers,

• Ryan