AWS Thinkbox Discussion Forums

mongo DB crashed and needed repair

Deprecated Deadline 6 Beta Discussion
#1

This happened last week on beta 19, then over the weekend we updated to RC2. Again, this morning the DB crashed, but after a repair and restart, we’re back on track. Where can I look for a log on what’s causing these crashes?

Thanks.

-ctj

#2

Mongo should be writing to a log file that you can check. If you installed mongo with the Repository installer, the path to the log file will be one of the command line arguments being passed to mongo.

Also, do you know which version of mongo you are running? We were originally shipping with 2.2, but later in the beta we started shipping 2.4 because it seemed more stable.

#3

Found the logs, thanks. There’s also a web view at <your_server_here>:28017

I’m still unsure what caused the exception. I’m looking for the dump file now. I’m wondering if there was something to do with the update from beta 19 to RC2. Should we shut down mongodb while updating the repository?

I noticed the mongodb.lock file is from 04/15/2013, which is when we updated from v2.2 to v2.4.1, which is our current version.

#4

That shouldn’t have caused any problems. You actually need to keep mongodb running during the update so that the installer can update any collections in it if necessary.

Let’s keep an eye on it. If it crashes again, send us the mongodb log before you restart mongodb and we’ll take a look.

#5

Just fell down again. The log file is attached. This time, I deleted the lock file and restarted, no repair necessary…that I know of.

When mongo goes down, how does this effect current renders?

-ctj
log.txt (484 KB)

#6

Thanks for posting the log. It looks like you’re running the 32 bit version of mongodb, which we don’t recommend or support, especially in production. This thread here provides some more information:
viewtopic.php?f=86&t=8687&hilit=mongo+bit

I’m guessing that mongo is hitting the 2GB limit when it runs in 32 bit mode, and that’s why it’s falling over. I would recommend getting your mongo server on a 64 bit system to ensure this doesn’t happen again.

Cheers,

  • Ryan
#7

I rarely pass the buck…but this is one of those times. I didn’t setup Mongo, but I completely understand the issue.

Thank you. Now to download the 64bit mongo.

Suggestions on the best way to do this changeover with as little data lose as possible?

Thank you,

-ctj

#8

Here are some steps to migrate the database (from the D6 documentation on Migrating The Database or Repository):

  1. Shut down all the Slave applications running on your render nodes. You don’t want them making changes during the move.
    ** 2) Copy the Database folder from the original location to the new one.
  2. Start the mongod process and ensure that it points to the data and log folders in the database folder.
  3. Modify the dbConnect.XML file in the settings folder in the Repository to set the new database host name (if you moved it to another machine).
  4. Start up the Slaves and ensure that they can connect to the new Database.

** Now in your case, since you are also changing bitness, you’ll just want to copy the data folder that’s in the mongodb database folder.

Here is an example of how you would run mongod to point to the new data folder:

mongod --logpath "[DATAFOLDER]/logs/log.txt" --quiet --dbpath "[DATAFOLDER]"

Because the Clients use the dbConnect.xml file in the Repository to determine the database connection settings, you don’t have to reconfigure the Clients to find the new database.

Hope that helps!

  • Ryan
Privacy | Site terms | Cookie preferences