AWS Thinkbox Discussion Forums

mongo db crash

Hi there,

Not sure if this is a deadline issue, but about 20 mins ago, mongo stopped responding and the whole queue died.

Still beta 17.

Last lines from the mongo db (also shows the manual restart i initiated):

Fri Apr 12 13:24:57 [conn127688840] info DFM::findAll(): extent 0:67000 was empty, skipping ahead. ns:deadline6db.LimitGroups
Fri Apr 12 13:24:57 [conn127779157] query deadline6db.Jobs query: { Stat: 1, IsSub: true, Props.Pool: { $in: [ “2d”, “3d”, “python”, “none” ] }, Props.Grp: { $in: [ “maya”, “nuke”, “noassburner”, “none” ] } } ntoreturn:0 ntoskip:0 nscanned:1499 keyUpdates:0 numYields: 11 locks(micros) r:12657 nreturned:2 reslen:286 464ms
Fri Apr 12 13:24:57 [conn127690010] query deadline6db.Jobs query: { Stat: 1, IsSub: true, Props.Pool: { $in: [ “2d”, “3d”, “python”, “none” ] }, Props.Grp: { $in: [ “maya”, “python”, “nuke”, “noassburner”, “none” ] } } ntoreturn:0 ntoskip:0 nscanned:1499 keyUpdates:0 numYields: 11 locks(micros) r:11729 nreturned:2 reslen:286 466ms
Fri Apr 12 13:24:57 [conn127703342] command deadline6db.$cmd command: { $eval: function() { db.SlaveInfo.update( { _id : “lapro0252” }, { $set : { "L… } ntoreturn:1 keyUpdates:0 locks(micros) W:401022 reslen:45 401ms
Fri Apr 12 13:24:57 [conn127761032] command deadline6db.$cmd command: { $eval: function() { db.SlaveInfo.update( { _id : “lapro1230” }, { $set : { "L… } ntoreturn:1 keyUpdates:0 locks(micros) W:393282 reslen:45 393ms
Fri Apr 12 14:40:24 got signal 15 (Terminated), will terminate after current cmd ends
forked process: 5155
all output going to: /var/log/mongo/mongodb.log

***** SERVER RESTARTED *****

Fri Apr 12 14:40:29 [initandlisten] MongoDB starting : pid=5155 port=27017 dbpath=/data/db/ 64-bit host=deadline.scanlinevfxla.com
Fri Apr 12 14:40:29 [initandlisten] db version v2.2.2, pdfile version 4.5
Fri Apr 12 14:40:29 [initandlisten] git version: d1b43b61a5308c4ad0679d34b262c5af9d664267

Hey Laszlo,

Try upgrading mongo to 2.4.1 and see if the problem comes up again:
mongodb.org/downloads

Someone else had reported a similar issue with 2.2, and upgrading to 2.4 fixed the problem for them.

We’ll actually be shipping this version with the next beta instead of 2.2. There are some performance improvements that seem to make a noticeable difference as well.

Cheers,

  • Ryan

We just had it go down again… very odd, cause we didnt hav ea single mongo crash up till now, and now it crashed twice within an hour

It went down again :frowning:

Going to the deadline web site, i get this:

deadline.scanlinevfxla.com:28017/
error loading page: couldn’t get readlock to open admin db

3rd time :frowning:

Not sure what happened :frowning: But we are in delivery crunch, and deadline basically stopped working HELP

(mongodb went down again)

Have you guys tried moving to mongo 2.4.1 yet? You just have to stop mongodb, replace the binaries, and then start it up again (you don’t need to reconfigure the database or anything).

If that doesn’t help, please post the mongodb log and we’ll take a look.

I updated to 2.4.1 now, and it seems to be working again. Ill keep you posted if it goes haywire again.

Oh friday.

‘crisis friday’

always.

cb

Privacy | Site terms | Cookie preferences