This has happened now twice within a day, and its starting to worry me. After a while, the mongo db just becomes inaccessible completely. I have to restart the service on the server to get it back up, for it to apparently it go inaccessible again within hours.
Any tips?
This is what the slaves/monitors get:
Connect to Deadline Repository
Deadline configuration error: An error occurred while trying to connect to the Database (deadline.scanlinevfxla.com:27017). It is possible that the Mongo Database server is incorrectly configured, currently offline, or experiencing network issues.
Full error: Unable to connect to server deadline.scanlinevfxla.com:27017: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host…
Might have figured it out, it appears that the ulimit settings reverted on machine reboot last week, so the centos machine went back to 1024 max threads per user.
Ah, that would totally do it. There must be a way to make those ulimit settings persist between reboots, or at the very least, you guys could write a shell script to set the ulimit values on boot up.