Slave scheduling questions/issues

[CodeTag]

So, I have a couple of questions/issues related to Slave Scheduling. Instead of creating different posts, I thought I’d put both of them here:

1. We have a slave that shows up in the list of slaves in the slaves panel but it does not show up in either the “Unassigned Slaves” or “Slaves In Group” when trying to add it to a schedule. Unfortunately this particular machine is in use during business hours, hence the need for the scheduling. This just happened when we upgraded from version 9 to 10. Thoughts?

2. One quarter of our farm are workstations by day and render nodes by night. The workstations are scheduled to render from 19:00 to 7:30. For the most part this works pretty well, but each day there will be machines that are still rendering at 8 or 8:30. We do have “Allow Slave To Finish Its Current Task When Stopping” enabled but according to the monitor they’re picking up jobs after the scheduled stop time of 7:30.

Thanks.

[eamsler]

Hmm. For #1, have you tried deleting the Slave and then starting it? I’m curious if there’s some weird state there. If it’s blue, the database has the Slave Settings but not the Slave Info (meaning it’s never reported its status properly).

For #2, that’s controlled via the Launcher in that a special command is sent to the Slave (the ‘stop after current task completion’ you see in the ‘remote commands’). Can you check for that in the Launcher log, then check the Slave log to see if it was received? The timestamps from the Launcher log will hint where you should look for the Slave.

It’s either the Slave is broken, or (more likely) the comms channel didn’t work. The logs should prove / disprove both.

[CodeTag]

We’re still trying to figure out what’s going on with an issue (#1 above) that we have been having. This machine won’t appear in the list for slave scheduling. It worked okay in version 7 and 9, but this showed up when we migrated to 10. We install the client on the machine in the same manner as all of our machines, it shows up in the list of slave machines, it renders correctly when enabled, but it will not show up in the slave scheduling lists. Normally it wouldn’t be an issue but this particular machine is someone’s workstation. We have uninstalled/re-installed, it’s never showed up a blue, just then normal green when rendering, yellow when disabled, white when idle. Not sure where to go from here.

[eamsler]

I’m not sure either to be honest. I’m going to check around these parts to see if someone can show me how that list is populated. Is the name of the machine different than the others by chance? My list seems to mash host names down to all lower case, and I’m not sure what other filtering we may be doing here.

[CodeTag]

Not too much different. Some of our machine names are all over the place… it’s the rn-012 machine.

[eamsler]

Okay, so I went and dug pretty deep through the code and we’re grabbing the Slaves out of the database by asking Mongo for every “db.SlaveInfo.Name” it has on file.

I think this is pretty unlikely, but it may be that the Slave doesn’t have a name set. If you’ve already deleted the Slave and it was re-created I’d say that’s not going to be the problem, but just for fun can you run “C:\DeadlineDatabase10\mongo\application\bin\deadline_mongo.bat” and then throw this in there:

use deadline10db;
db.SlaveInfo.find({'Name': 'rn-012'}, {'_id': 1}).pretty();

For me, that ended up like so:

> db.SlaveInfo.find({'Name': 'MyMachine2'}, {'_id': 1}).pretty();
{ "_id" : "MyMachine2" }

Also, to grab all machine ids and Names, run this guy too:

db.SlaveInfo.find({}, {'_id': 1, 'Name': 1}).pretty();

on my test farm:

> db.SlaveInfo.find({}, {'_id': 1, 'Name': 1}).pretty();
{ "_id" : "MachineName", "Name" : "MachineName" }
{ "_id" : "MachineName-gamma", "Name" : "MachineName-gamma" }
{ "_id" : "MachineName-bravo", "Name" : "MachineName-bravo" }
{ "_id" : "MachineName-charlie", "Name" : "MachineName-charlie" }
{ "_id" : "MachineName-alpha", "Name" : "MachineName-alpha" }
{ "_id" : "MachineName-test", "Name" : "MachineName-test" }
{ "_id" : "MyMachine2", "Name" : "MyMachine2" }

[CodeTag]

So for the first command:

db.SlaveInfo.find({‘Name’: ‘rn-012’}, {’_id’: 1}).pretty();
{ “_id” : “rn-012” }

The second:

db.SlaveInfo.find({}, {’_id’: 1, ‘Name’: 1}).pretty();
{ “_id” : “am-14”, “Name” : “AM-14” }
{ “_id” : “rn-114”, “Name” : “rn-114” }
{ “_id” : “rn-01”, “Name” : “rn-01” }
{ “_id” : “rn-02”, “Name” : “rn-02” }
{ “_id” : “rn-012”, “Name” : “rn-012” }
{ “_id” : “rn-13”, “Name” : “rn-13” }

So it looks like it’s in the DB, it’s just not making it to the scheduling list.

[eamsler]

Okay. I’ll send you some code via a PM to try which uses undocumented APIs I don’t want people to become reliant on. It’s not especially secret, I just want to avoid future internal API changes bringing down someone’s farm.

Another question, are you using the RCS in your Deadline deployment?

[eamsler]

Update for others watching: It seems that the API call that builds a list of Slaves to populate those lists is working just fine, at least when run through deadlinecommand. Another test: Could you copy it into “[repo]/scripts/jobs”, enable it in “Configure Script Menus” -> “Job Script” and try running it by right-clicking a job? It will write out the the Monitor’s log that you can find by going to “Help” then “Explore Log Folder”. It’ll have similar output to when you ran it from deadlinecommand. We’re just looking for “rn-012” in there, and I don’t expect much to be different.

I am curious about the RCS setup though. We moved endpoints over a few versions back and while there doesn’t seem to be anything that would have this one Slave stand apart, I’m wondering if caching or something similar may be affecting getting this Slave.

[CodeTag]

I was out last last week for Thanksgiving so I haven’t had a chance to try your last request yet. We are not using RCS in our deployment.

[eamsler]

No worries. The test within the Monitor is just checking the caching layer (which should also be fine).

If that also works, I might get a dump of your Database so we can test over here, but I’ll likely want to move that conversation to a ticket instead of the forums.