AWS Thinkbox Discussion Forums

beta 18, slaves not dequeuing jobs

This is for beta18 slaves

Get this error in the log:
Scheduler Thread - Performing house cleaning…
Scheduler - Minor exception encountered while scanning for tasks and then getting job.
Scheduler - Job ID: 51688ab0a3f32e0d5893d6f4
Scheduler - exception: Object reference not set to an instance of an object. (System.NullReferenceException)
Scheduler - Unexpected Error Occurred

Exception Details
IndexOutOfRangeException – Index was outside the bounds of the array.
Exception.Data: ( )
Exception.TargetSite: Deadline.Jobs.Task DequeueNextTask(System.String, Int32, System.String, System.DateTime, Boolean ByRef)
Exception.Source: deadline
Exception.StackTrace:
at Deadline.StorageDB.MongoDB.MongoJobStorage.DequeueNextTask(String jobID, Int32 preferredTaskID, String slaveName, DateTime networkDateTime, Boolean& triggerStartEvent)
at Deadline.Controllers.DataController.DequeueTasks(SlaveState slaveState, Boolean verboseLogging)
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
Scheduler - Returning limit stubs not in use.
Scheduler - returning nuke
Scheduler - returning 51688ab0a3f32e0d5893d6f4
Scheduler Thread - Seconds before next job scan: 29

Mmm, looks like it’s a small bug in the new Task code, combined with a newer slave looking for a Task in an old Job. Have you guys completely rolled out Beta 18, or is this indeed a Job submitted with Beta 17 (or older)?

Either way, I’ll fix the bug I found in the current code.

You might have to temporarily have a separate pool for Beta 18 Jobs, and have that as top priority for the Beta 18 slaves so they don’t get stuck on these Beta 17 ones that are causing trouble.

-Jon

Thanks, i ended up suspending the offending jobs, and it fixed the issue.

I also keep a beta17 farm around just in case artists dont update their builds,… which might be for a while, as there is no way to force update them, and they dont start up the slaves overnight… :
So, i asked everyone to restart their monitors, but that usually falls on deaf ears :\

Gotcha. You should also be able to re-submit a Beta 17 job from a Beta 18 monitor (in the right-click menu), which should result in a Beta 18 job. Figured that could be useful in the transition, thought I’d mention it :slight_smile:

Cheers,

  • Jon

Thats a good tip, thanks!

We do get the occasional submission now from beta17, that hungs up the beta18 farm till that jobs finishes… im updating all workstations manually in vancouver, but i have a feeling that stragglers will remain all around the place for days :\

l

Christian here had a suggestion, maybe it would be a good option to add that deadline slave version has to match the submitting machine’s deadline version? To avoid this situation…

Privacy | Site terms | Cookie preferences