Also, it appears Deadline is still using (or at least thinking about using) a config file in the user’s home directory, in spite of this being previously-discussed as a horrible idea.
Our Pulse machine ate its own deadline.ini file today, and based on the timestamp on that file, this is what happened in the Launcher log at that time:
2015-03-18 11:10:05: Could not set deadline.ini file setting 'RestartStalledSlave' because: /tmp/deadline.ini does not exist (System.IO.FileNotFoundException)
2015-03-18 11:10:05: Auto Configuration: Set Local Slave Data Folder to '/var/lib/Thinkbox/Deadline6/slave-data'
2015-03-18 11:10:05: Launcher Thread - Automatic Updates is now disabled
It (the launcher?) blew away a bunch of specific settings (network root, license server, etc.) and replaced them all with one line setting the SlaveDataRoot. The upshot of this is that no one could submit any jobs, because we do all of our submissions using the REST API.
Would be really nice to have this resolved.
Here are some more error samples (these were all in a big chunk in the launcher log):
2015-03-18 08:09:19: Could not set deadline.ini file setting 'User' because: Could not create file "/tmp/deadline.ini" or "/Volumes/newhome/qmaster/Thinkbox/De
adline7/deadline.ini". File already exists. (System.IO.IOException)
2015-03-18 08:09:19: Could not set deadline.ini file setting 'LaunchSlaveAtStartup' because: Could not create file "/tmp/deadline.ini" or "/Volumes/newhome/qma
ster/Thinkbox/Deadline7/deadline.ini". File already exists. (System.IO.IOException)
2015-03-18 08:14:20: Could not set deadline.ini file setting 'User' because: Could not find file "/Volumes/newhome/qmaster/Thinkbox/Deadline7/deadline.ini". (S
ystem.IO.FileNotFoundException)
2015-03-18 08:29:28: Auto Configuration: Set Launch Slave At Startup value to 'false'
2015-03-18 08:34:30: Could not set deadline.ini file setting 'User' because: /Volumes/newhome/qmaster/Thinkbox/Deadline7/deadline.ini already exists (System.IO
.IOException)
2015-03-18 08:34:31: Could not set deadline.ini file setting 'LaunchSlaveAtStartup' because: Could not create file "/tmp/deadline.ini" or "/Volumes/newhome/qma
ster/Thinkbox/Deadline7/deadline.ini". File already exists. (System.IO.IOException)
2015-03-18 08:44:32: Auto Configuration: Set Launch Slave At Startup value to 'false'
2015-03-18 08:49:34: Could not set deadline.ini file setting 'User' because: Could not find file "/Volumes/newhome/qmaster/Thinkbox/Deadline7/deadline.ini" (Sy
stem.IO.FileNotFoundException)
2015-03-18 08:49:34: Using default value - could not get deadline.ini file setting 'LaunchSlaveAtStartup' because: Could not find file "/Volumes/newhome/qmaste
r/Thinkbox/Deadline7/deadline.ini". (System.IO.FileNotFoundException)
2015-03-18 08:49:34: Could not set deadline.ini file setting 'LaunchSlaveAtStartup' because: Could not find file "/Volumes/newhome/qmaster/Thinkbox/Deadline7/d
eadline.ini" (System.IO.FileNotFoundException)
Thanks for reporting this. Looks like this can still be an issue when the Launcher is updating the deadline.ini file with settings it retrieved from Auto Configuration. I wonder if there is a better way we could handle auto configuration settings (ie: like store them in a separate autoconfig.ini file that simply acts as an override). Ideally though, we’d want to figure out how this even happens in the first place. We’ll need to do some more investigation on our end.
Deadline still uses a config file in the user’s home directory for the current Deadline user name, and if the slave should launch at start up. System-wide settings like the repository path, license server, etc are stored in the system deadline.ini file. This is all covered here: docs.thinkboxsoftware.com/produc … onfig.html
I can’t see us changing this, because these settings are important for clients that have multiple users share a machine. Can you explain why you feel that this is a horrible idea?
Modifying of the local deadline.ini file does seem like it could be potentially bad if one application reads from it as another is writing it… Having the auto configuration settings in a separate file that functioned as an override seems like it would address this, as long as the settings reading functionality had a lot of error checking in place.
What are all those error messages about anyway?
Sorry, I should have been more specific. I think it’s a horrible idea to use a config file in the user’s home directory if it’s going to get updated by any kind of automated process. If it is only modified when the user actually changes some preference themselves, that should be fine. However, if 150 render nodes with launchers and slaves running as the same user all try to edit the same file, something is going to go horribly wrong.
We discovered some problematic logic in the updating of the ini file. It writes to a temp file first, and then copies it over the original. However, the temp file name was static, which meant that it was possible to multiple processes/threads to write or delete the temp file at the same time. This causes some of these errors, but not all of them. Changing to a dynamically generated temp file name helps a bit, but the problems still occur, so we’re trying to solve this issue at its core. If we can, then the separate autoconfig ini file won’t be necessary (although it’s still might be a good idea to do).
On windows, we can rely on how the OS locks the file when it’s in use to prevent these sorts of errors, but we don’t have that luxury on Linux and OSX…
Thanks for clarifying! Yes, this is currently the case with the settings that are stored in the user config file.
So over the last few days, this has silently happened to about half of our farm. I’m probably going to have to lock the perms on the deadline.ini file, but it gets worse.
The slaves whose deadline.ini files have been eviscerated have all started stealing licenses from our Melbourne studio, even though their license server has a completely different name than our LA license server, and its name isn’t in any auto-config setups or even in the deadline.ini files of the slaves that have suddenly decided to hit it up for licenses.
Where would these machines get the idea that they should connect to a license server they shouldn’t even be aware of?
Can I remove the THINKBOX_LICENSE_FILE line from that file to prevent this kind of stuff from happening in the future? If the slaves can’t connect to the license server, I want to know about it.
The code we get from Flexera handles all of that, so any time it successfully checks out a license from any source, it’ll store a little copy of the licensing path in either that file on the Unix-like machines, or in the registry on Windows. It will then forever use that as one of the places it checks for licenses.
We could open the file and purge it every time, but that’s not particularly safe.
I’d recommend blocking license traffic across facilities if that’s at all possible.