Affinity control per render job

Tim_Ellis · March 18, 2015, 9:47pm

I have a modular setup on my farm. I am testing affinity and wondered if I can assign affinity to a group or pool of slaves. Or can I launch a slave’ render process assigned per job via commandline?

Ideally I would like to run a single slave with affinity = all for gigapixel renders.
two slaves for high res rendering using affinity 0-7 for one slave, 8-15 for a second slave.
four slaves for low res rendering using affinity 0-3, 4-7, 8-11, 12-15 for each respective slave.

I know I can set each slaves affinity through the monitor, but can I do it through commandline, so I don’t need to manually change them for script based automatic job submisson?

Cheers,

Tim.

Tim_Ellis · March 19, 2015, 10:08am

After reading through the forums and also finding out the launcher is licensed and not the slave in V7, I think I can do this with groups and multiple slave instances pre-set to specific affinities.

Will post results.

Tim.

rrussell · March 19, 2015, 12:07pm

Actually, the slave is still the application that is licensed, it’s just that multiple slave instances on the same machine now share one license, instead of each requiring a license.

Cheers,
Ryan

Tim_Ellis · March 19, 2015, 12:47pm

Yes, sorry Ryan that is what understood, my brain interpreted it reversely.

Ok so I now have my slaves set up, 7 per node, each named and set to a specific affinity and also assigned to a matching group and pool.

node:1 slave:1 affinity:0-7 group:affinity_1 pool:affinity_0-7 concT:0 - --------------------------> 2 cpus/8cores 8 concTasks

node:1 slave:2 affinity:0-3 group:affinity_2 pool:affinity_0-3 concT:2 - --------------------------> 1 cpus/4cores 2 concTasks
node:1 slave:3 affinity:4-7 group:affinity_2 pool:affinity_4-7 concT:2 - --------------------------> 1 cpus/4cores 2 concTasks

node:1 slave:4 affinity:0-1 group:affinity_3 pool:affinity_0-1 concT:1 - --------------------------> 1 cpus/2cores 1 concTasks
node:1 slave:5 affinity:2-3 group:affinity_3 pool:affinity_2-3 concT:1 - --------------------------> 1 cpus/2cores 1 concTasks
node:1 slave:6 affinity:4-5 group:affinity_3 pool:affinity_4-5 concT:1 - --------------------------> 1 cpus/2cores 1 concTasks
node:1 slave:7 affinity:6-7 group:affinity_3 pool:affinity_6-7 concT:1 - --------------------------> 1 cpus/2cores 1 concTasks

node:2 slave:1 affinity:0-7 group:affinity_1 pool:affinity_0-7 concT:0 - --------------------------> 2 cpus/8cores 8 concTasks

node:2 slave:2 affinity:0-3 group:affinity_2 pool:affinity_0-3 concT:2 - --------------------------> 1 cpus/4cores 2 concTasks
node:2 slave:3 affinity:4-7 group:affinity_2 pool:affinity_4-7 concT:2 - --------------------------> 1 cpus/4cores 2 concTasks

node:2 slave:4 affinity:0-1 group:affinity_3 pool:affinity_0-1 concT:1 - --------------------------> 1 cpus/2cores 1 concTasks
node:2 slave:5 affinity:2-3 group:affinity_3 pool:affinity_2-3 concT:1 - --------------------------> 1 cpus/2cores 1 concTasks
node:2 slave:6 affinity:4-5 group:affinity_3 pool:affinity_4-5 concT:1 - --------------------------> 1 cpus/2cores 1 concTasks
node:2 slave:7 affinity:6-7 group:affinity_3 pool:affinity_6-7 concT:1 - --------------------------> 1 cpus/2cores 1 concTasks

node:3 slave:1 etc etc etc etc

So what is the best route here for me to take, with respect to job configuration and auto assigning a pool/group to be used for different jobs.
I currently use a custom script which uses image resolution to decide how many concurrent tasks should be used.

0-5999 pixels for longest side - 4 concurrent tasks
6000-9999 pixels for longest side - 2 concurrent tasks
10000-99999 pixels for longest side - 1 concurrent tasks

I can see from a previous thread that a pre-render script in the submission script can be used to drive affinity based on a job type:-
Reference code: (Not sure if there is a missing " after the Quicktime word?)

job = self.deadlinePlugin.GetJob() if job.JobName.lower().find("quicktime) != -1: self.ProcessAffinity = (0,)

So looking through the scripting reference, I think I can do something that calls a group based on the concurrent task setting. Is this correct?
Something like this:-

[code]job = self.deadlinePlugin.GetJob()
if job.JobConcurrentTask.lower().find("8) != -1:
self.set.Deadline.Statistics.JobEntry.Group = (affinity_1)

job = self.deadlinePlugin.GetJob()
if job.JobConcurrentTask.lower().find("2) != -1:
self.set.Deadline.Statistics.JobEntry.Group = (affinity_2)

job = self.deadlinePlugin.GetJob()
if job.JobConcurrentTask.lower().find("1) != -1:
self.set.Deadline.Statistics.JobEntry.Group = (affinity_3)
[/code]

Am I going down the correct path here?

Many thanks,

Tim.

rrussell · March 19, 2015, 1:40pm

Hey Tim,

With your current set up of 7 slaves per machine, what happens if a gigapixel job is picked up at the same time that 4 low res renders are being processed? The slaves on the machine work independently of each other, and thus aren’t aware of what the other slaves are rendering.

Are the jobs you’re submitting multi-task jobs, or single task jobs? If they are multi-task, you could make use of the job’s concurrent task property instead.

If they are single task jobs, then of course the concurrent task property isn’t an option…

Cheers,
Ryan

Tim_Ellis · March 19, 2015, 3:09pm

Hi Ryan,

The farm only ever process one job at a time, irrespective of concurrent tasks. However I see what you mean though I think, but pool priority and no ‘This job is interruptible’ should stop multiple jobs running concurrently.

Some jobs are animations, ie multi-task but only ever use 5 frames per task and single still renders are split, so my ten nodes get either 10 tasks cooperatively, 20 tasks or 40 tasks depending on the concurrent task set up. Even for animation jobs, I submit a single job per frame so the farm works together on one frame instead of all working on a different frame. This is Maxwell submissions if you have not guessed yet.
Each node will only run one pool/group at a time on one job at a time, at least that’s my intention. If I need to spin up slaves or kill them on the fly to avoid cross pollination of jobs, then I will have to go down that route. If by the time I get to testing this, the system fails when there is a queue, I can submit jobs as suspended and resume them once the previous completes. That may be another option.
The farm works in it’s current configuration without affinity, I just want to streamline and optimize cpu/core use, rather than have 4 instances of a plugin running off all cpus with no distinct allocation.

Cheers,

Tim.

rrussell · March 19, 2015, 5:39pm

Hey Tim,

Thanks for clearing that up. So if I’m understanding everything correctly, I think the best approach is to write an Event Plugin that handles the OnJobSubmitted event. It would check the concurrent task property of the job being submitted, and assign it to the appropriate group. Here’s the event plugin documentation for reference:
docs.thinkboxsoftware.com/produc … ugins.html

And the scripting API reference:
docs.thinkboxsoftware.com/produc … Reference/

Your dlinit file probably just needs the Enabled property:

Enabled=True

Your param file would have an entry to modify the Enabled property from the Configure Events menu item in the Monitor’s Tools menu:

[Enabled]
Type=boolean
Label=Enabled
Default=True
Description=If this event plug-in should respond to events.

Finally, your py file would probably look something like this:

from Deadline.Events import *
from Deadline.Scripting import *

######################################################################
## This is the function that Deadline calls to get an instance of the
## main DeadlineEventListener class.
######################################################################
def GetDeadlineEventListener():
    return MyEvent()

######################################################################
## This is the function that Deadline calls when the event plugin is
## no longer in use so that it can get cleaned up.
######################################################################
def CleanupDeadlineEventListener( deadlinePlugin ):
    deadlinePlugin.Cleanup()

######################################################################
## This is the main DeadlineEventListener class for MyEvent.
######################################################################
class MyEvent (DeadlineEventListener):

    def __init__( self ):
        # Set up the event callbacks here
        self.OnJobSubmittedCallback += self.OnJobSubmitted

    def Cleanup( self ):
        del self.OnJobSubmittedCallback

    def OnJobSubmitted( self, job ):
        if job.JobConcurrentTasks == 1:
            job.JobGroup = "affinity_1"
        elif job.JobConcurrentTasks == 2:
            job.JobGroup = "affinity_2"
        elif job.JobConcurrentTasks == 8:
            job.JobGroup = "affinity_3"
        
        RepositoryUtils.SaveJob( job )

In this case, the OnJobSubmitted function is checking the concurrent tasks property and assigning the group accordingly.

Hope this helps!
Ryan

Tim_Ellis · March 19, 2015, 9:32pm

Thankyou very much sir. I will give this a go and let you know how I get on.

Looking through the other affinity based threads, there would be quite a lot of use of this type of tool, more so with version 7 than before.

Tim.

Tim_Ellis · March 19, 2015, 9:39pm

Oh and current test of pools and groups for pre-set slaves’ affinity is working.

!!! Very important addition needed for Maxwell 3 though:- !!!
Commandline argument:-

-affinity:manual

This stops Maxwell controlling it’s threads. If threads is set to 0, Maxwell 3 will use all available threads and will ignore any affinity set by Deadline. Any thread number set for use is then started on cpuid 0. So if I set 4 threads, it would only ever use the first four cores, even if I set the slave to run on the last four. So the manual overide allows it to be controlled by Deadline’s affinity control.

Tim.

Tim_Ellis · November 10, 2015, 1:54pm

Hi Ryan, I’m trying to implement this with V6 final before I migrate to V7.

[code]
######################################################################

This is the main DeadlineEventListener class for MyEvent.

######################################################################
class MyEvent (DeadlineEventListener):

def __init__( self ):
    # Set up the event callbacks here
    self.OnJobSubmittedCallback += self.OnJobSubmitted

def Cleanup( self ):
    del self.OnJobSubmittedCallback

def OnJobSubmitted( self, job ):
    if job.JobConcurrentTasks == 3:
        job.JobGroup = "merge"
        job.JobPool = "merge"
    elif job.JobConcurrentTasks == 1:
        job.JobGroup = "affinity_1"
        job.JobPool = "affinity_1"
    elif job.JobConcurrentTasks == 2:
        job.JobGroup = "affinity_2"
        job.JobPool = "affinity_2"
    elif job.JobConcurrentTasks == 4:
        job.JobGroup = "affinity_2"
        job.JobPool = "affinity_2"
		
    RepositoryUtils.SaveJob( job )[/code]

This is the last section of my .py script, it is set to enabled in the Events. Param and init files are as you posted.
How can I check if this is being processed? Is there a logging/report function I can use? It seemed to be working but then stopped.

We custom build _job_info.job and _plugin_info.job files which we send to Deadline. Perhaps these are not accounting for my event as I fear we maybe creating jobs after the event would occur.

Can you shed some light on the situation please?

Cheers,

Tim.

rrussell · November 10, 2015, 2:04pm

Hey Tim,

Could you zip up the entire folder for your event and post it? We could then drop it in our repo and test it as is.

Thanks!
Ryan

Tim_Ellis · November 10, 2015, 2:18pm

AffinitySlaveControl.zip (1.31 KB)

Here you are sir.

Cheers,

Tim.

ryangagnon · November 10, 2015, 3:51pm

Hey Tim, if you want to log anything in a plugin in v6 you can use the ClientUtils.LogText() function. Here’s your plugin with log text added to print the affinity of the newly submitted job to a job report for that job.

from Deadline.Events import *
from Deadline.Scripting import *

######################################################################
## This is the function that Deadline calls to get an instance of the
## main DeadlineEventListener class.
######################################################################
def GetDeadlineEventListener():
    return MyEvent()

######################################################################
## This is the function that Deadline calls when the event plugin is
## no longer in use so that it can get cleaned up.
######################################################################
def CleanupDeadlineEventListener( deadlinePlugin ):
    deadlinePlugin.Cleanup()

######################################################################
## This is the main DeadlineEventListener class for MyEvent.
######################################################################
class MyEvent (DeadlineEventListener):

    def __init__( self ):
        # Set up the event callbacks here
        self.OnJobSubmittedCallback += self.OnJobSubmitted

    def Cleanup( self ):
        del self.OnJobSubmittedCallback

    def OnJobSubmitted( self, job ):
        ClientUtils.LogText( "Submitted job has affinity "+str(job.JobConcurrentTasks ))
        if job.JobConcurrentTasks == 3:
            job.JobGroup = "merge"
            job.JobPool = "merge"
        elif job.JobConcurrentTasks == 1:
            job.JobGroup = "affinity_1"
            job.JobPool = "affinity_1"
        elif job.JobConcurrentTasks == 2:
            job.JobGroup = "affinity_2"
            job.JobPool = "affinity_2"
        elif job.JobConcurrentTasks == 4:
            job.JobGroup = "affinity_2"
            job.JobPool = "affinity_2"
            
        RepositoryUtils.SaveJob( job )

A few comments about this plugin:

The case where concurrent tasks =2 and =4 assigns the job to the same pool and group. Not sure if that was intentional or not
If the pools or groups you are trying to assign the job to do not already exist, then the jobs pool/group property will be set to “none”.

As far as job submission goes, command line submission of the jobinfo and plugininfo will still trigger the OnJobSubmitted event, so you don’t need to worry about that.

Hope that helps!

Ryan G.

Tim_Ellis · December 11, 2015, 11:39am

Hi Ryan,

Thank you for your help, just wanted to let you know I got it all working.

Cheers,

Tim.

Tim_Ellis · March 1, 2016, 2:59pm

I have this working nicely thanks for your help.

I now wish to add another elif following on from my current script but I can’t get a positive result for Modo:-

from Deadline.Events import *
from Deadline.Scripting import *

######################################################################
## This is the function that Deadline calls to get an instance of the
## main DeadlineEventListener class.
######################################################################
def GetDeadlineEventListener():
    return MyEvent()

######################################################################
## This is the function that Deadline calls when the event plugin is
## no longer in use so that it can get cleaned up.
######################################################################
def CleanupDeadlineEventListener( deadlinePlugin ):
    deadlinePlugin.Cleanup()

######################################################################
## This is the main DeadlineEventListener class for MyEvent.
######################################################################
class MyEvent (DeadlineEventListener):

    def __init__( self ):
        # Set up the event callbacks here
        self.OnJobSubmittedCallback += self.OnJobSubmitted

    def Cleanup( self ):
        del self.OnJobSubmittedCallback

    def OnJobSubmitted( self, job ):
        ClientUtils.LogText( "Submitted job has affinity "+str(job.JobConcurrentTasks ))
			
        if job.JobGroup == "merge":
            job.JobGroup = "merge"
            job.JobPool = "merge"
        elif job.JobConcurrentTasks == 1:
            job.JobGroup = "affinity_3"
            job.JobPool = "affinity_3"
        elif job.JobConcurrentTasks == 2:
            job.JobGroup = "affinity_2"
            job.JobPool = "affinity_2"
        elif job.JobConcurrentTasks == 4:
            job.JobGroup = "affinity_3"
            job.JobPool = "affinity_3"
        elif job.JobPlugin == "Modo":
            job.JobGroup = "affinity_2"
            job.JobPool = "affinity_2"
            ClientUtils.LogText( "Submitted job is "+str(job.JobPlugin ))


        RepositoryUtils.SaveJob( job )

The Modo section is not working though, but this maybe my argument being in the wrong position, as Modo jobs are being assigned group and pool as per concurrent task values.

For Modo tile assembly jobs and Maxwell Render merge jobs, I want the tile assembly & merge job for co-op renders, to be assigned the Merge group and pool, but I need the main co-op job to be assigned to an affinity_pool as per concurrent taks.

Can I use:-

        elif DeadlinePlugin.IsTileJob == "true":
            job.JobGroup = "merge"
            job.JobPool = "merge"
        elif job.JobPlugin == "Maxwell" and job.JobType == "MergeJob":
            job.JobGroup = "merge"
            job.JobPool = "merge"

I ask because, this isn’t working either and concurrent tasks are still overriding the groups. Again perhaps I need to lead with plugin type, then job type and then concurrent tasks as a couple of separate arguments instead of trying to combine the lot to one if / elif argument.

Can you help please?

Cheers,

Tim.