ArgumentException: The given SlaveInfo and SlaveSettings objects have an empty SlaveName property

Jonathan_Bouchard · February 5, 2024, 11:00pm

Hi,

I wish Deadline was open source, I could make more of the following server-side stack trace.

The issue manifests itself when listing workers from the monitor application or through deadlinecommand. The workers list on deadlinemonitor remains empty when it occurs. (The monitor console reports the HTTP call returned status 500)

I’ve looked into mongo and none of the SlaveInfo or SlaveSettings records have a missing or empty Name. Restarting the deadline server seems to fix the issue for a little while and it comes back. (We’re running version 10.2)

Does anyone have a clue what might cause this?

This started to happen out of a sudden. We are having a hard time to correlate it to anything in particular that might have changed.

Feb  5 16:51:27 deadline bash: ERROR: DeadlineLoggerMiddleware caught exception method=POST path=/db/slaves/modified remoteip=172.16.91.115 exception=System.ArgumentException: The given SlaveInfo and SlaveSettings objects have an empty SlaveName property.
Feb  5 16:51:27 deadline bash: at Deadline.Slaves.SlaveInfoSettings..ctor(SlaveInfo slaveInfo, SlaveSettings slaveSettings)
Feb  5 16:51:27 deadline bash: at Deadline.StorageDB.MongoDB.MongoSlaveStorage.GetModifiedSlaveInfoSettingsFromCache(SlaveInfoSettings[]& modifiedSlaveInfoSettings, String[]& deletedSlaveIds, Boolean& hasMore, DateTime settingsCutOff, DateTime infoCutOff, DateTime deletionCutOff)
Feb  5 16:51:27 deadline bash: at Deadline.ProxyServer.API.Commands.DB.ProxyWebService.GetModifiedSlaveInfoSettings(ModifiedSlaveInfoModel data, String transactionID)
Feb  5 16:51:27 deadline bash: at lambda_method1213(Closure , Object , Object[]

Thank you!

Jonathan_Bouchard · February 7, 2024, 5:45pm

This was actually caused by a worker machine in our environment that didn’t get its’ hostname assigned properly (it was blank).

If I can find a way to suggest a feature/bug report, I’ll suggest that Deadline should protect itself from blank hostnames / error out with some meaningful message instead of corrupting itself.

Cheers,

Justin_B · February 12, 2024, 9:56pm

Noted! We’ve seen issues when the name is empty in the database and causing issues in the Monitor.

We haven’t found a consistent cause of the hostname not being read and set correctly. Did a reboot of the machine correct the issue on the machine?

Jonathan_Bouchard · February 12, 2024, 10:15pm

In this case the worker hostname was truly empty. We didn’t notice that our install process failed to set it properly. Setting /etc/hostname fixed it.

The mongo database had no SlaveInfo or SlaveSettings document with an empty Name field. I think there must be an in-memory cache somewhere in Deadline that had it empty. Restarting the server fixed it momentarily. It started to fail again as soon as this nameless machine came online.

I don’t think there’s a bug in Deadline. It would be great though if the service that enrolls new workers/slaves would protect itself from evil machines with empty hostnames.

Justin_B · February 13, 2024, 2:37pm

Oh I gotcha. I’ve never seen the machine without a hostname, the issue was always in the machine adding itself to the database.

Either way, Deadline should be more resilient if it’s going to rely on the hostname as a primary key for the machines.