Probably a stupid question, but is there a way to assign a spot instance to a pool upon creation? I thought the tags just MIGHT be what I was looking for, but apparently not. Just curious as this would help when I’m creating a fleet to pre-emptively assign to a pool so I can control which jobs get tossed up to AWS or not.
Thanks!
We have some code in a few places to do that (Balancer, I think the Spot plugin?..). The easy way at the moment is to make a simple OnSlaveStarted() event that looks around, sees that it’s in an EC2 instance (however you want to do that) and assign a particular pool to itself. There’s an example script here:
github.com/ThinkboxSoftware/Dea … veAutoconf
The pools and groups don’t need a Slave restart the same way as its listening port does.
Yup, this is definitely what I was looking for. For whatever reason though, my python isn’t behaving. Event is set to global enable, and the only thing I changed from the script is:
SLAVE_NAME_PREFIX = "ip-" # Example: "mobile-"
POOLS = ["aws"] # Example: ["one", "two", "three"]
LISTENING_PORT=27100 # or 27100
btw tried it as it comes (with the port slot empty). Not a big deal, but either way can’t seem to get it working right now.
Mmm… The Slave log will have the output for it if you have an easy way to connect into them. The new Slave streaming in 10.0 SP11 won’t catch it though as the startup events happen before the logging buffer has started up.
Specifically, it’ll look something like this:
Adding pool pool_name
The other bit is that you should see it populate with square brackets in the “pools” column in the Slave list first before the Slave properly applies it to itself and starts using it.
Just tried it again as I keep dipping in and out of AWS testing, and I see this from the slave (which seems to indicate its working, but its not?):
2018-02-27 14:59:12: SlaveAutoconf: Slave automatic configuration for ip-10-128-23-89
2018-02-27 14:59:12: SlaveAutoconf: Adding pool aws
2018-02-27 14:59:13: SlaveAutoconf: Configuring Slave to listen on port 27100
Date: 02/27/2018 14:59:12
Event Type: OnSlaveStarted
Slave Name: ip-10-128-23-89
Version: v10.0.11.1 Release (b0f2000be)
Operating System:
Machine User: ec2-user
IP Address:
MAC Address:
CPU Architecture:
CPUs: 0
CPU Usage: 0%
Memory Usage: 0.0 Bytes / -1.0 Bytes (0%)
Free Disk Space:
Video Card:
[/code]
I’m sure there is something simple that I’m overlooking, I just really can’t figure it out. I tried tweaking the options in the script (using “ip-” and “ip” as the prefix, etc etc).
I just noticed I chose “27100” for the Slave’s listening port. That’s a bit funny now that Deadline 10 is using it for the database…
As far as this not working, I wonder if there’s some kind of race condition where we’re updating the information and saving it, but something else is re-configuring things and saving the the old information over top. Are you using the AWS Portal tooling or other OnSlaveStarted events?
I’m trying to remember if we have an option to serialize how the events are fired…
I’m confused what you mean about the port, but if I left that blank, then I still don’t get the pool assignment and the log reads:
2018-02-27 16:55:55: SlaveAutoconf: Slave automatic configuration for ip-10-128-16-130
2018-02-27 16:55:55: SlaveAutoconf: Adding pool aws
2018-02-27 16:55:57: SlaveAutoconf: Configuring Slave to use random listening port
[/code]
I’m using the event plugin with the OnSlaveStarted you linked from github (copied below):
[code]‘’’
Stolen from Gavin’s great example on this forum thread:
Autoconfigure Groups/Pools? - Deadline - AWS Thinkbox Discussion Forums
‘’’
from System.Diagnostics import *
from System.IO import *
from System import TimeSpan
from Deadline.Events import *
from Deadline.Scripting import *
import re
import sys
import os
import subprocess
import traceback
import shlex
SLAVE_NAME_PREFIX = “ip” # Example: “mobile-”
POOLS = [“aws”] # Example: [“one”, “two”, “three”]
LISTENING_PORT=None # or 27100
def GetDeadlineEventListener():
return ConfigSlaveEventListener()
def CleanupDeadlineEventListener(eventListener):
eventListener.Cleanup()
class ConfigSlaveEventListener (DeadlineEventListener):
def init(self):
self.OnSlaveStartedCallback += self.OnSlaveStarted
def Cleanup(self):
del self.OnSlaveStartedCallback
# This is called every time the Slave starts
def OnSlaveStarted(self, slavename):
# Load slave settings for when we needed
slave = RepositoryUtils.GetSlaveSettings(slavename, True)
# Skip over Slaves that don't match the prefix
if not slavename.lower().startswith(SLAVE_NAME_PREFIX):
return
print("Slave automatic configuration for {0}".format(slavename))
# Set up the Pools we want to use
for pool in POOLS:
try:
print(" Adding pool {0}".format(pool))
RepositoryUtils.AddPoolToSlave(slavename, pool)
# Power management example:
# pmanage = RepositoryUtils.GetPowerManagementOptions()
# pmanage.Groups[0].SlaveNames.append(slavename)
except:
ClientUtils.LogText(traceback.format_exc())
# Set up the listening port
if LISTENING_PORT:
print(" Configuring Slave to listen on port {0}".format(LISTENING_PORT))
slave.SlaveListeningPort = LISTENING_PORT
slave.SlaveOverrideListeningPort = True
else:
print(" Configuring Slave to use random listening port".format(LISTENING_PORT))
slave.SlaveOverrideListeningPort = False
# Save any changes we've made back to the database
RepositoryUtils.SaveSlaveSettings(slave)
[/code]
Ah, sorry for my lack of clarity there. I was wondering if there were other OnSlaveStarted events. I know Charles has a working cloud setup that we can test this with. I’ll see if he’d be game to just copy the event over and give it a try.
We’ve had a lot of requests to bring in automatic pool / group assignment into the AWS Portal tooling, and I believe we’ll be making use of an event plugin to do that, so if there’s a bug in how this works we’ll find it pretty fast.