SEP not shutting down instances after render - IAM security credential error

[Walt]

Beginning a few days ago our Spot Event Plugin setup has ceased to shut down spot instances after rendering. We’ve made a ton of changes across the board so can’t easily identify the cause but all attempted solutions thus far have failed. The worker logs periodically complain about being “unable to get IAM security credentials from EC2 Instance Metadata Service” and being unable to load the
Event plugin ‘Spot’ from the repository, but these errors only seem to happen after the worker has been idle for 5-10 minutes after a render completes (right around the same time they should be automatically shutting down… coincidence?). Not sure how to even start debugging that issue since everything runs fine on the worker up until sometime after renders complete.

SEP otherwise is running fine… spot requests go in, worker instances get created and everything renders… the workers just stick around idle until manually killed.

Log file from a worker:
deadlineslave-ip-0A0D0088-2024-07-07-0000.log.zip (14.0 KB)

[Walt]

After a lot of digging we were able to solve the issue(s). Two separate issues, actually. The IAM error came from the IAM Instance Profile being removed from the launch template. Restoring DeadlineSpotWorker to the launch template solved that. The python errors all came from the workers being on 10.1.10.6 instead of 10.3.2.1 like the repository. Not sure where we ended up with mixed versions, but between 10.1.10.6 and 10.3.2.1 Python went from 2.7 to 3.10 which obviously makes a huge difference in what works and what doesn’t.