Did you guys change anything with the way limits work? I haven’t fully solved what’s going on but we are launching 16 concurrent nuke tasks with a limit of 12 which has always worked in the past. Pretty quickly we hit the FLEXlm license limit reached and then the license server seems to lock up and hold some licenses and never free them. If I lower our limit to 10, even though we have 12 it seems to go through no problem. Is Deadline launching a new task before the old ones have checked the license back in?
Also we noticed that machines that blow memory and stall, even if they still have nuke open and still have a license, will check their limit back in. I would imagine that if a machine stalls it should hold the license until it is reset to be sure the license is returned.
I don’t know if this is a Nuke issue or Deadline, but it just started happening when we upgraded to D6.2.
i’ve had cases in 6.1 where a tasks would bomb out but keep running. in fact, these tasks would even show up in the monitor slave window. the slave would pick up new jobs as well, so instead of saying “Idle” then switching to “Nuke” it would say “Houdini” then switch to “Nuke” then back to “Houdini”. so it kind of knew it was running… anyway, that is not new to 6.2.
what would be nice is to have deadline understand the licensing mechanism for each tool and do a more direct query of what’s available rather than having a hand-entered value. maybe have some kind of reservation system so you could set aside some number of tokens and leave the rest for the farm. that would take some work i’m guessing, so maybe more generally just let users specify a script to run that returns how many licenses are currently available for a specific tool.
So I am not sure how this is related to Concurrent Tasks but:
When I submit a job with Concurrent Tasks set to 16 and Limit Tasks to Slave’s Task Limit ON, and the slave task limit is set to 16 everything works fine and I do not receive any licensing problems from nuke.
When I submit a job with Concurrent Tasks set to 16 and Limit Tasks to Slave’s Task Limit OFF, and the slave task limit is set to 16, then I start failing on licenses.
When I submit a job with Concurrent Tasks set to 16 and Limit Tasks to Slave’s Task Limit OFF, and the slave task limit is set to 15, then everything seems to work again.
This really makes no sense to me and I am also talking to the Foundry about it as well. They said they have no limit on concurrent nuke sessions so could this some how be influencing the limits and not reporting properly?
It could be an issue between when the job is done and deadline goes looking for a new task, and the time it takes for the license to be given back. That would be my first guess. The new task search may be jumping ahead of the license return.
Would this be something that’s changed since 6.1? We never had a problem before. To your point though is there a delay at the end of a task that we could add… a “cooling off” period that might alleviate any race situations?
In thinking about this for a second I’m not totally sure how this would be the case since nuke would need to be shutdown before the task is completed, right? I asked the guys at the Foundry if they ensure that a license is returned before closing or is there a chance that nuke would be shutdown before the license server acknowledges the license being returned. Either way a cooling off period would be an easy and quick way to test this I would imagine.
So looking over all the stuff here fresh today, a few things stuck out at me. It seems like there was one option that seemed to work and use all your licenses, was that not a viable option for some reason? You could also use an options file as laid out here(media.3ds.com/support/simulia/pu … /chap5.htm) to ensure licenses get returned promptly. Hopefully one of these options works.
Well… that first scenario doesn’t work. It was working but I think the license server just happened to be keeping up. I’ve been talking to the foundry and they said they might have an issue with windows not returning the licenses immediately with the latest release which is causing the problem. If there was a way to add a delay to a task onComplete, it could be a workaround for now. Maybe it’s a simple as a python script that times out for a second.
I’m supposed to speak with the Foundry tomorrow and I’ll post up what comes of that.
