AWS Thinkbox Discussion Forums

Creating an AWS Portal Spot Fleet using deadlinecommand -CreateSpotFleet

Deadline
#1

According to the deadlinecommand arguments help, launching a Spot Fleet should be possible by providing a compatible JSON file with the keys

{“StackName”: “”, “Capacity”: 1, “InstanceTypes”: [], “SpotPrice”: 1, “AmiId”: “”, “AmiName”: “”, “KeyName”: “”, “AvailabilityZones”: [], “AWSPortalRequestType”: “”, “IdleTime”: 2, “Groups”: [], “Pools”: [], “InstanceTags”: {}, “Context”: “”, }

I have built a reasonably correct JSON file with all the required information including the Stack Name (automatically extracted from the prefix of the Placement Group of the Gateway instance), the instance types, the AMI ID, AMI Name and DashKey as used by my manually-launched Spot Fleet instances, the ‘AvailabilityZones’: [‘us-west-2a’, ‘us-west-2b’, ‘us-west-2c’] because my d Zone does not carry any GPU instances, and ‘AWSPortalRequestType’: ‘maintain’ which is an assumption, but hopefully correct.

I have also set the InstanceTags to the same values the AWS Portal would create.

The rest of the values like Pools, Groups and Context were left empty like in the template example.

When I run the command, I see in the Console the following:

2024-03-06 10:40:13:  PYTHON: Please enter your AWS Access Key ID:
2024-03-06 10:40:13:  PYTHON: Error reading spot fleet parameters, please ensure your input json file is in the format: {"StackName": "", "Capacity": 1, "InstanceTypes": [], "SpotPrice": 1, "AmiId": "", "AmiName": "", "KeyName": "", "AvailabilityZones": [], "AWSPortalRequestType": "", "IdleTime": 2, "Groups": [], "Pools": [], "InstanceTags": {},  "Context": "", }
2024-03-06 10:40:13:  PYTHON: Object reference not set to an instance of an object.

The log implies that

  • The AWS access key (and maybe the secret access key) need to be provided explicitly
  • The JSON file is somehow incorrect.
  • An error occurred internally when creating some object.

It is very possible that some of these are side effects of a single error. However, since the AWS Access Key ID message comes first, I suspect that it is the first issue to tackle.

So my question is - what is the correct way of providing credentials to this particular command? I have aws credentials configured on the machine and my boto3 scripts can perform pretty much any operation. I am willing to attempt to recreate the whole Spot Fleet logic myself (although it would require some reverse engineering to configure a Spot Fleet that completely meshes with AWS Portal), but given that the deadlinecommand offers such a command, it would be a waste of time.

That being said, I could imagine that the deadlinecommand -CreateSpotFleet is for internal use only, and the documentation simply fails to tell us that (esp. given the fact that I added that documentation chapter myself based on the command’s help and forgot to ask anyone while I could…)

Thanks in advance!

#2

I know folks have used that command, but I’ve never used it myself.

So with a little fiddling I’ve got it working, using this:

{
    "StackName": "stack12345",
    "Capacity": 1,
    "InstanceTypes": ["c4.xlarge"],
    "SpotPrice": 1,
    "AmiId": "ami-01e14aee9b97b73fc",
    "AmiName": "Deadline Worker Base Image Windows 10.3.1.3 with 3ds Max 2020 and V-Ray 4.30.02 and Arnold 4.0.4.36 2023-12-19T205011Z",
    "KeyName": "DashKey",
    "AvailabilityZones": ['us-east-1a'],
    "AWSPortalRequestType": "Maintain",
    "IdleTime": 2,
    "Groups": [],
    "Pools": [],
    "InstanceTags": {},
    "Context": ""
}

I ran it manually in a command prompt and had to input the access key and secret key for my AWSPortal IAM user. I think your problem is how you’re running the command, it appears we’re expecting to get those credentials via standard input versus pulling from configuration on the machine a-la aws cli or boto.

How are you running this? It looks like it’s being run in our scripting internals? It might need some extra wrapping so the keys can be provided over stdin if you want to fully automate it.

#3

Thanks Justin!

Yep, I would like to run it from a Python script.

If the problem is the credentials input, then the error message "Error reading spot fleet parameters, please ensure your input json file is in the format..." is truly misleading :slight_smile:

#4

I got it working using subprocess.Popen():

args = [dlcmd_exe, "-CreateSpotFleet", template_json_file]
input_data = b'<accesskeyhere>\n<secretaccesskeyhere>\n'
with subprocess.Popen(args, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE) as proc:
    stdout, stderr = proc.communicate(input=input_data)

where dlcmd_exe is the path to the deadlinecommand executable, and template_json_file is the .json file where the description of the fleet is saved.
The keys would be read from the .aws\config file or the environment to avoid hard-coding secrets in the source.

Thanks for the nudge in the right direction!

1 Like
#5

An understatement!

Glad you’re up and running, and thanks for sharing the snippet for future folks!

Privacy | Site terms | Cookie preferences