We had a strange issue early this morning. According to the webservice logs, it has not been doing anything since 5.33am (that was the last line in the log). However, in the process list, the webservice was going full tilt at 40-50% cpu usage.
Queries like this:
lascript0003.scanlinevfxla.com:8087/
Returned the expected string of:
This is the Deadline Pulse web service!
But actual queries would time out. Any ideas how that can happen? We checked the health status of the webservice by simply looking for the “This is the Deadline PUlse web service!” value being returned, so we didnt detect the outage for quite a while.
Are all other queries timing out? If you request the maximum priority does that hang? ( lascript0003.scanlinevfxla.com:8 … umpriority )
It could be a problem with the Web Service losing it’s connection to the repository. That will require some testing on our end. Are there other deadline applications running on the machine hosting that web service?
We think the machine might have run out of space. Since this is a critical service, we restarted it to recover operations, and a little while later another service member started cleaning up its drive due to drive full errors. (it got full due to webservice logs that dont seem to get deleted by deadline)
Ah that’s very helpful, thanks for the update! We’ll look into that.
We reconfigured our health checks to another low overhead test: lascript0003.scanlinevfxla.com:8 … sOnly=true