Heya,
We recently moved over to using scheduling with workers starting and stopping as per the schedule, this requires workers to be started and stopped, thus moving them between online and offline states. We noticed this week that offline workers do not show on the Monitors worker table when that Monitor instance is started whilst the worker is offline. While this makes sense, I would like to be able to see all worker in the network in the worker table in Monitor as there are time I need to start or stop workers on offline machines (not sure this is possible as workers are the ones that report to the repository and without a worker instance running on the machine…)? All the workers show up when polling the repository with RepositoryUtils.GetSlaveNames though I don’t know if this is due to some sort of caching. Any help would be great.
Cheers,
Stephen
Hello Stephen
This is not something the Worker scheduling should do normally. The Workers should not be deleted like this. You may have below setting configures in Deadline Monitor> Tools> Configure Repo Options> Worker settings> Delete Offline/Stalled Workers?
Hi Zainali,
Can confirm that that setting is not active. I assumed as Deadline workers report themselves to the repository it would be a case of the offline workers not reporting in (as they are offline) but obviously this is wrong. The behavior only occurs when a new instance of Monitor is opened up, if the worker goes offline while the monitor is active, the worker remains visible in the list. If that monitor is closed then opened again, the worker isn’t visible in the list any more. As mentioned, they are still there (listed as offline when viewing render candidates), just not in the list in the Workers tab. Any suggestions to prevent this?
Cheers,
Stephen
Ok, so I have figured out a way to get the offline workers to show up, All I do is loop through the workers and run RepositoryUtils.SaveSlaveSettings(slave_settings). This gets them to show up. I believe this is very similar to another issue I posted on the forum here: Free Disk Space Large Discrepancy
@zainali Is there any long term fix for this desyncing issue between the workers and the Repository or is this just a quirk of the decentralised nature of Deadline?
Hello
This seems to be a weird on off case of desync. We will need to further look into the issue. I have seen this before where the number of Workers show in the Monitor connected directly is different than shown in the Monitor connected via the RCS.
Are those Workers connected via RCS?
What version of Mondo DB are you running? Run below (Windows) to find out.
C:\DeadlineDatabase10\mongo\application\bin\mongo.exe --version
We may need to repair the DB, but I think We can get you DB dump and recreate the issue on our end. DB is has confidential information so I suggest you cut us a ticket for this issue.