AWS Thinkbox Discussion Forums

MongoDBService stopping and won't restart

Hello everyone,

I’m trying to diagnose an issue with the Deadline10DatabaseService running on our main server (Windows Server 2019 up to date).

The Database service will run for about 24 hours, give or take, and then it will stop running. If I right click on the service > restart service, it starts briefly and then stops almost as quickly. Restarting the entire server does allow the service to restart on its own, but it will eventually fail again.

For context, we have ~24 machines on the network, 8 of which are workstations. Our Domain Controller has an SMB share on the network, and that’s where the repository is stored (so all machines can access it), so that’s where Mongo.exe is started from, and it’s run by the Domain Controller itself, along with Pulse. The farm behaves normally otherwise.

I’ve checked the logs, but it’s difficult to tell exactly what’s happening. I’m looking for an expert to help us out!

These are the last few entry lines in the log file:

{"t":{"$date":"2024-05-14T14:03:05.855-06:00"},"s":"I",  "c":"NETWORK",  "id":4915701, "ctx":"-","msg":"Initialized wire specification","attr":{"spec":{"incomingExternalClient":{"minWireVersion":0,"maxWireVersion":13},"incomingInternalClient":{"minWireVersion":0,"maxWireVersion":13},"outgoing":{"minWireVersion":0,"maxWireVersion":13},"isInternalClient":true}}}
{"t":{"$date":"2024-05-14T14:03:05.857-06:00"},"s":"I",  "c":"CONTROL",  "id":23285,   "ctx":"-","msg":"Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'"}
{"t":{"$date":"2024-05-14T14:03:06.419-06:00"},"s":"W",  "c":"ASIO",     "id":22601,   "ctx":"thread1","msg":"No TransportLayer configured during NetworkInterface startup"}
{"t":{"$date":"2024-05-14T14:03:06.420-06:00"},"s":"I",  "c":"NETWORK",  "id":4648602, "ctx":"thread1","msg":"Implicit TCP FastOpen in use."}
{"t":{"$date":"2024-05-14T14:03:06.425-06:00"},"s":"W",  "c":"ASIO",     "id":22601,   "ctx":"thread1","msg":"No TransportLayer configured during NetworkInterface startup"}
{"t":{"$date":"2024-05-14T14:03:06.426-06:00"},"s":"I",  "c":"REPL",     "id":5123008, "ctx":"thread1","msg":"Successfully registered PrimaryOnlyService","attr":{"service":"TenantMigrationDonorService","ns":"config.tenantMigrationDonors"}}
{"t":{"$date":"2024-05-14T14:03:06.426-06:00"},"s":"I",  "c":"REPL",     "id":5123008, "ctx":"thread1","msg":"Successfully registered PrimaryOnlyService","attr":{"service":"TenantMigrationRecipientService","ns":"config.tenantMigrationRecipients"}}
{"t":{"$date":"2024-05-14T14:03:06.426-06:00"},"s":"I",  "c":"CONTROL",  "id":5945603, "ctx":"thread1","msg":"Multi threading initialized"}
{"t":{"$date":"2024-05-14T14:03:06.426-06:00"},"s":"I",  "c":"CONTROL",  "id":23316,   "ctx":"thread1","msg":"Trying to start Windows service","attr":{"serviceName":"MongoDB"}}
{"t":{"$date":"2024-05-14T14:03:06.428-06:00"},"s":"I",  "c":"CONTROL",  "id":4615611, "ctx":"initandlisten","msg":"MongoDB starting","attr":{"pid":10020,"port":27100,"dbPath":"D:/AllSharedFiles/Projects/_render files/Deadline_repo/MongoDB/mongo/data","architecture":"64-bit","host":"DCSRV01"}}
{"t":{"$date":"2024-05-14T14:03:06.428-06:00"},"s":"I",  "c":"CONTROL",  "id":23398,   "ctx":"initandlisten","msg":"Target operating system minimum version","attr":{"targetMinOS":"Windows 7/Windows Server 2008 R2"}}
{"t":{"$date":"2024-05-14T14:03:06.428-06:00"},"s":"I",  "c":"CONTROL",  "id":23403,   "ctx":"initandlisten","msg":"Build Info","attr":{"buildInfo":{"version":"5.0.22","gitVersion":"302c19437ae65b7d459360e18c5ac5086f494989","modules":[],"allocator":"tcmalloc","environment":{"distmod":"windows","distarch":"x86_64","target_arch":"x86_64"}}}}
{"t":{"$date":"2024-05-14T14:03:06.428-06:00"},"s":"I",  "c":"CONTROL",  "id":51765,   "ctx":"initandlisten","msg":"Operating System","attr":{"os":{"name":"Microsoft Windows Server 2019","version":"10.0 (build 17763)"}}}
{"t":{"$date":"2024-05-14T14:03:06.428-06:00"},"s":"I",  "c":"CONTROL",  "id":21951,   "ctx":"initandlisten","msg":"Options set by command line","attr":{"options":{"config":"D:\\AllSharedFiles\\Projects\\_render files\\Deadline_repo\\MongoDB\\mongo\\data\\config.conf","net":{"bindIp":"*","ipv6":true,"port":27100,"tls":{"mode":"disabled"}},"security":{"authorization":"disabled"},"service":true,"storage":{"dbPath":"D:\\AllSharedFiles\\Projects\\_render files\\Deadline_repo\\MongoDB\\mongo\\data","engine":"wiredTiger"},"systemLog":{"destination":"file","path":"D:\\AllSharedFiles\\Projects\\_render files\\Deadline_repo\\MongoDB\\mongo\\data\\logs\\log.txt","quiet":true,"verbosity":0}}}}
{"t":{"$date":"2024-05-14T14:03:06.434-06:00"},"s":"W",  "c":"STORAGE",  "id":22271,   "ctx":"initandlisten","msg":"Detected unclean shutdown - Lock file is not empty","attr":{"lockFile":"D:\\AllSharedFiles\\Projects\\_render files\\Deadline_repo\\MongoDB\\mongo\\data\\mongod.lock"}}
{"t":{"$date":"2024-05-14T14:03:06.435-06:00"},"s":"W",  "c":"STORAGE",  "id":22302,   "ctx":"initandlisten","msg":"Recovering data from the last clean checkpoint."}
{"t":{"$date":"2024-05-14T14:03:06.436-06:00"},"s":"I",  "c":"STORAGE",  "id":22315,   "ctx":"initandlisten","msg":"Opening WiredTiger","attr":{"config":"create,cache_size=15681M,session_max=33000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),builtin_extension_config=(zstd=(compression_level=6)),file_manager=(close_idle_time=600,close_scan_interval=10,close_handle_minimum=250),statistics_log=(wait=0),verbose=[recovery_progress,checkpoint_progress,compact_progress],"}}
{"t":{"$date":"2024-05-14T14:03:06.477-06:00"},"s":"I",  "c":"STORAGE",  "id":22430,   "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":"[1715716986:476992][10020:140722393732016], txn-recover: [WT_VERB_RECOVERY_PROGRESS] Recovering log 352 through 353"}}
{"t":{"$date":"2024-05-14T14:03:06.548-06:00"},"s":"I",  "c":"STORAGE",  "id":22430,   "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":"[1715716986:547682][10020:140722393732016], txn-recover: [WT_VERB_RECOVERY_PROGRESS] Recovering log 353 through 353"}}
{"t":{"$date":"2024-05-14T14:03:06.636-06:00"},"s":"I",  "c":"STORAGE",  "id":22430,   "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":"[1715716986:635296][10020:140722393732016], txn-recover: [WT_VERB_RECOVERY_ALL] Main recovery loop: starting at 352/256 to 353/256"}}
{"t":{"$date":"2024-05-14T14:03:06.639-06:00"},"s":"I",  "c":"STORAGE",  "id":22430,   "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":"[1715716986:638281][10020:140722393732016], txn-recover: [WT_VERB_RECOVERY_PROGRESS] Recovering log 352 through 353"}}
{"t":{"$date":"2024-05-14T14:03:06.748-06:00"},"s":"I",  "c":"STORAGE",  "id":22430,   "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":"[1715716986:747797][10020:140722393732016], file:index-82--6041725008197237395.wt, txn-recover: [WT_VERB_RECOVERY_PROGRESS] Recovering log 353 through 353"}}
{"t":{"$date":"2024-05-14T14:03:06.815-06:00"},"s":"I",  "c":"STORAGE",  "id":22430,   "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":"[1715716986:814506][10020:140722393732016], file:index-82--6041725008197237395.wt, txn-recover: [WT_VERB_RECOVERY_ALL] Set global recovery timestamp: (0, 0)"}}
{"t":{"$date":"2024-05-14T14:03:06.815-06:00"},"s":"I",  "c":"STORAGE",  "id":22430,   "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":"[1715716986:815501][10020:140722393732016], file:index-82--6041725008197237395.wt, txn-recover: [WT_VERB_RECOVERY_ALL] Set global oldest timestamp: (0, 0)"}}
{"t":{"$date":"2024-05-14T14:03:06.838-06:00"},"s":"I",  "c":"STORAGE",  "id":22430,   "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":"[1715716986:838402][10020:140722393732016], WT_SESSION.checkpoint: [WT_VERB_CHECKPOINT_PROGRESS] saving checkpoint snapshot min: 36, snapshot max: 36 snapshot count: 0, oldest timestamp: (0, 0) , meta checkpoint timestamp: (0, 0) base write gen: 1200928"}}
{"t":{"$date":"2024-05-14T14:03:06.860-06:00"},"s":"I",  "c":"STORAGE",  "id":4795906, "ctx":"initandlisten","msg":"WiredTiger opened","attr":{"durationMillis":424}}
{"t":{"$date":"2024-05-14T14:03:06.860-06:00"},"s":"I",  "c":"RECOVERY", "id":23987,   "ctx":"initandlisten","msg":"WiredTiger recoveryTimestamp","attr":{"recoveryTimestamp":{"$timestamp":{"t":0,"i":0}}}}
{"t":{"$date":"2024-05-14T14:03:06.890-06:00"},"s":"I",  "c":"STORAGE",  "id":22262,   "ctx":"initandlisten","msg":"Timestamp monitor starting"}
{"t":{"$date":"2024-05-14T14:03:06.895-06:00"},"s":"I",  "c":"NETWORK",  "id":4915702, "ctx":"initandlisten","msg":"Updated wire specification","attr":{"oldSpec":{"incomingExternalClient":{"minWireVersion":0,"maxWireVersion":13},"incomingInternalClient":{"minWireVersion":0,"maxWireVersion":13},"outgoing":{"minWireVersion":0,"maxWireVersion":13},"isInternalClient":true},"newSpec":{"incomingExternalClient":{"minWireVersion":0,"maxWireVersion":13},"incomingInternalClient":{"minWireVersion":13,"maxWireVersion":13},"outgoing":{"minWireVersion":13,"maxWireVersion":13},"isInternalClient":true}}}
{"t":{"$date":"2024-05-14T14:03:06.897-06:00"},"s":"I",  "c":"STORAGE",  "id":5071100, "ctx":"initandlisten","msg":"Clearing temp directory"}
{"t":{"$date":"2024-05-14T14:03:06.931-06:00"},"s":"I",  "c":"CONTROL",  "id":20536,   "ctx":"initandlisten","msg":"Flow Control is enabled on this deployment"}
{"t":{"$date":"2024-05-14T14:03:07.205-06:00"},"s":"I",  "c":"FTDC",     "id":20625,   "ctx":"initandlisten","msg":"Initializing full-time diagnostic data capture","attr":{"dataDirectory":"D:/AllSharedFiles/Projects/_render files/Deadline_repo/MongoDB/mongo/data/diagnostic.data"}}
{"t":{"$date":"2024-05-14T14:03:07.205-06:00"},"s":"I",  "c":"REPL",     "id":6015317, "ctx":"initandlisten","msg":"Setting new configuration state","attr":{"newState":"ConfigReplicationDisabled","oldState":"ConfigPreStart"}}
{"t":{"$date":"2024-05-14T14:03:07.208-06:00"},"s":"I",  "c":"NETWORK",  "id":23015,   "ctx":"listener","msg":"Listening on","attr":{"address":"0.0.0.0"}}
{"t":{"$date":"2024-05-14T14:03:07.208-06:00"},"s":"I",  "c":"NETWORK",  "id":23015,   "ctx":"listener","msg":"Listening on","attr":{"address":"::"}}
{"t":{"$date":"2024-05-14T14:03:07.208-06:00"},"s":"I",  "c":"NETWORK",  "id":23016,   "ctx":"listener","msg":"Waiting for connections","attr":{"port":27100,"ssl":"off"}}
{"t":{"$date":"2024-05-14T14:03:07.209-06:00"},"s":"I",  "c":"CONTROL",  "id":20555,   "ctx":"initandlisten","msg":"Service running"}
{"t":{"$date":"2024-05-14T14:03:07.265-06:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn1","msg":"client metadata","attr":{"remote":"192.168.42.10:63589","client":"conn1","doc":{"driver":{"name":"mongo-csharp-driver|legacy","version":"2.13.3.0"},"os":{"type":"Windows","name":"Microsoft Windows 10.0.17763","architecture":"x86_64","version":"10.0.17763"},"platform":".NET 6.0.24"}}}
{"t":{"$date":"2024-05-14T14:03:07.266-06:00"},"s":"W",  "c":"COMMAND",  "id":5578800, "ctx":"conn1","msg":"Deprecated operation requested. The client driver may require an upgrade in order to ensure compatibility with future server versions. For more details see https://dochub.mongodb.org/core/legacy-opcode-compatibility","attr":{"op":"query","clientInfo":{"driver":{"name":"mongo-csharp-driver|legacy","version":"2.13.3.0"},"os":{"type":"Windows","name":"Microsoft Windows 10.0.17763","architecture":"x86_64","version":"10.0.17763"},"platform":".NET 6.0.24"}}}
{"t":{"$date":"2024-05-14T14:03:07.268-06:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn2","msg":"client metadata","attr":{"remote":"192.168.42.10:63590","client":"conn2","doc":{"driver":{"name":"mongo-csharp-driver|legacy","version":"2.13.3.0"},"os":{"type":"Windows","name":"Microsoft Windows 10.0.17763","architecture":"x86_64","version":"10.0.17763"},"platform":".NET 6.0.24"}}}
{"t":{"$date":"2024-05-14T14:03:07.268-06:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn3","msg":"client metadata","attr":{"remote":"192.168.42.10:63591","client":"conn3","doc":{"driver":{"name":"mongo-csharp-driver|legacy","version":"2.13.3.0"},"os":{"type":"Windows","name":"Microsoft Windows 10.0.17763","architecture":"x86_64","version":"10.0.17763"},"platform":".NET 6.0.24"}}}
{"t":{"$date":"2024-05-14T14:03:07.359-06:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn4","msg":"client metadata","attr":{"remote":"192.168.42.10:63592","client":"conn4","doc":{"driver":{"name":"mongo-csharp-driver|legacy","version":"2.13.3.0"},"os":{"type":"Windows","name":"Microsoft Windows 10.0.17763","architecture":"x86_64","version":"10.0.17763"},"platform":".NET 6.0.24"}}}
{"t":{"$date":"2024-05-14T14:03:07.361-06:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn5","msg":"client metadata","attr":{"remote":"192.168.42.10:63593","client":"conn5","doc":{"driver":{"name":"mongo-csharp-driver|legacy","version":"2.13.3.0"},"os":{"type":"Windows","name":"Microsoft Windows 10.0.17763","architecture":"x86_64","version":"10.0.17763"},"platform":".NET 6.0.24"}}}
{"t":{"$date":"2024-05-14T14:03:07.421-06:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn6","msg":"client metadata","attr":{"remote":"[fe80::9745:160:9531:4dae%6]:63584","client":"conn6","doc":{"driver":{"name":"mongo-csharp-driver|legacy","version":"2.13.3.0"},"os":{"type":"Windows","name":"Microsoft Windows 10.0.17763","architecture":"x86_64","version":"10.0.17763"},"platform":".NET 6.0.24"}}}
{"t":{"$date":"2024-05-14T14:03:07.424-06:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn7","msg":"client metadata","attr":{"remote":"[fe80::9745:160:9531:4dae%6]:63585","client":"conn7","doc":{"driver":{"name":"mongo-csharp-driver|legacy","version":"2.13.3.0"},"os":{"type":"Windows","name":"Microsoft Windows 10.0.17763","architecture":"x86_64","version":"10.0.17763"},"platform":".NET 6.0.24"}}}
{"t":{"$date":"2024-05-14T14:03:07.430-06:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn8","msg":"client metadata","attr":{"remote":"192.168.42.10:63586","client":"conn8","doc":{"driver":{"name":"mongo-csharp-driver|legacy","version":"2.13.3.0"},"os":{"type":"Windows","name":"Microsoft Windows 10.0.17763","architecture":"x86_64","version":"10.0.17763"},"platform":".NET 6.0.24"}}}
{"t":{"$date":"2024-05-14T14:03:07.500-06:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn9","msg":"client metadata","attr":{"remote":"192.168.42.10:63594","client":"conn9","doc":{"driver":{"name":"mongo-csharp-driver|legacy","version":"2.13.3.0"},"os":{"type":"Windows","name":"Microsoft Windows 10.0.17763","architecture":"x86_64","version":"10.0.17763"},"platform":".NET 6.0.24"}}}
{"t":{"$date":"2024-05-14T14:03:07.501-06:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn10","msg":"client metadata","attr":{"remote":"192.168.42.10:63595","client":"conn10","doc":{"driver":{"name":"mongo-csharp-driver|legacy","version":"2.13.3.0"},"os":{"type":"Windows","name":"Microsoft Windows 10.0.17763","architecture":"x86_64","version":"10.0.17763"},"platform":".NET 6.0.24"}}}
{"t":{"$date":"2024-05-14T14:03:07.616-06:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn11","msg":"client metadata","attr":{"remote":"[fe80::9745:160:9531:4dae%6]:63587","client":"conn11","doc":{"driver":{"name":"mongo-csharp-driver|legacy","version":"2.13.3.0"},"os":{"type":"Windows","name":"Microsoft Windows 10.0.17763","architecture":"x86_64","version":"10.0.17763"},"platform":".NET 6.0.24"}}}
{"t":{"$date":"2024-05-14T14:03:07.676-06:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn12","msg":"client metadata","attr":{"remote":"[fe80::9745:160:9531:4dae%6]:63588","client":"conn12","doc":{"driver":{"name":"mongo-csharp-driver|legacy","version":"2.13.3.0"},"os":{"type":"Windows","name":"Microsoft Windows 10.0.17763","architecture":"x86_64","version":"10.0.17763"},"platform":".NET 6.0.24"}}}
{"t":{"$date":"2024-05-14T14:03:07.676-06:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn13","msg":"client metadata","attr":{"remote":"192.168.42.10:63596","client":"conn13","doc":{"driver":{"name":"mongo-csharp-driver|legacy","version":"2.13.3.0"},"os":{"type":"Windows","name":"Microsoft Windows 10.0.17763","architecture":"x86_64","version":"10.0.17763"},"platform":".NET 6.0.24"}}}
{"t":{"$date":"2024-05-14T14:03:07.717-06:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn14","msg":"client metadata","attr":{"remote":"192.168.42.10:63597","client":"conn14","doc":{"driver":{"name":"mongo-csharp-driver|legacy","version":"2.13.3.0"},"os":{"type":"Windows","name":"Microsoft Windows 10.0.17763","architecture":"x86_64","version":"10.0.17763"},"platform":".NET 6.0.24"}}}
{"t":{"$date":"2024-05-14T14:03:08.001-06:00"},"s":"F",  "c":"CONTROL",  "id":6384300, "ctx":"ftdc","msg":"Writing fatal message","attr":{"message":"terminate() called. An exception is active; attempting to gather more information\n"}}
{"t":{"$date":"2024-05-14T14:03:08.001-06:00"},"s":"F",  "c":"CONTROL",  "id":6384300, "ctx":"ftdc","msg":"Writing fatal message","attr":{"message":"std::exception::what(): Unknown exception\nActual exception type: class std::exception\n\n"}}
{"t":{"$date":"2024-05-14T14:03:08.042-06:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn15","msg":"client metadata","attr":{"remote":"192.168.42.10:63598","client":"conn15","doc":{"driver":{"name":"mongo-csharp-driver|legacy","version":"2.13.3.0"},"os":{"type":"Windows","name":"Microsoft Windows 10.0.17763","architecture":"x86_64","version":"10.0.17763"},"platform":".NET 6.0.24"}}}
{"t":{"$date":"2024-05-14T14:03:08.140-06:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn16","msg":"client metadata","attr":{"remote":"192.168.42.10:63599","client":"conn16","doc":{"driver":{"name":"mongo-csharp-driver|legacy","version":"2.13.3.0"},"os":{"type":"Windows","name":"Microsoft Windows 10.0.17763","architecture":"x86_64","version":"10.0.17763"},"platform":".NET 6.0.24"}}}
{"t":{"$date":"2024-05-14T14:03:08.142-06:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn17","msg":"client metadata","attr":{"remote":"192.168.42.10:63600","client":"conn17","doc":{"driver":{"name":"mongo-csharp-driver|legacy","version":"2.13.3.0"},"os":{"type":"Windows","name":"Microsoft Windows 10.0.17763","architecture":"x86_64","version":"10.0.17763"},"platform":".NET 6.0.24"}}}
{"t":{"$date":"2024-05-14T14:03:08.148-06:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn18","msg":"client metadata","attr":{"remote":"192.168.42.10:63601","client":"conn18","doc":{"driver":{"name":"mongo-csharp-driver|legacy","version":"2.13.3.0"},"os":{"type":"Windows","name":"Microsoft Windows 10.0.17763","architecture":"x86_64","version":"10.0.17763"},"platform":".NET 6.0.24"}}}
{"t":{"$date":"2024-05-14T14:03:08.170-06:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn19","msg":"client metadata","attr":{"remote":"192.168.42.10:63602","client":"conn19","doc":{"driver":{"name":"mongo-csharp-driver|legacy","version":"2.13.3.0"},"os":{"type":"Windows","name":"Microsoft Windows 10.0.17763","architecture":"x86_64","version":"10.0.17763"},"platform":".NET 6.0.24"}}}
{"t":{"$date":"2024-05-14T14:03:08.203-06:00"},"s":"I",  "c":"CONTROL",  "id":31380,   "ctx":"ftdc","msg":"BACKTRACE","attr":{"bt":{"backtrace":[{"a":"7FF66F51EC31","module":"mongod.exe","file":".../src/mongo/util/stacktrace_windows.cpp","line":333,"s":"mongo::printStackTrace","s+":"41"},{"a":"7FF66F522007","module":"mongod.exe","file":".../src/mongo/util/signal_handlers_synchronous.cpp","line":244,"s":"mongo::`anonymous namespace'::myTerminate","s+":"D7"},{"a":"7FF66F5EE3A7","module":"mongod.exe","file":".../src/mongo/stdx/set_terminate_internals.cpp","line":88,"s":"mongo::stdx::dispatch_impl","s+":"17"},{"a":"7FF66F5EE389","module":"mongod.exe","file":".../src/mongo/stdx/set_terminate_internals.cpp","line":92,"s":"mongo::stdx::TerminateHandlerDetailsInterface::dispatch","s+":"9"},{"a":"7FFC7A83DE58","module":"ucrtbase.dll","s":"terminate","s+":"18"},{"a":"7FFC70AB1AAB","module":"VCRUNTIME140_1.dll","s":"_NLG_Return2","s+":"95B"},{"a":"7FFC70AB2317","module":"VCRUNTIME140_1.dll","s":"_NLG_Return2","s+":"11C7"},{"a":"7FFC70AB4119","module":"VCRUNTIME140_1.dll","s":"_CxxFrameHandler4","s+":"A9"},{"a":"7FF66F7F7C10","module":"mongod.exe","file":"D:/a/_work/1/s/src/vctools/crt/vcstartup/src/gs/amd64/gshandlereh4.cpp","line":86,"s":"__GSHandlerCheck_EH4","s+":"64"},{"a":"7FFC7E4A4C2F","module":"ntdll.dll","s":"_chkstk","s+":"11F"},{"a":"7FFC7E404CEF","module":"ntdll.dll","s":"RtlWalkFrameChain","s+":"14BF"},{"a":"7FFC7E408AE6","module":"ntdll.dll","s":"RtlRaiseException","s+":"316"},{"a":"7FFC7A4C1B29","module":"KERNELBASE.dll","s":"RaiseException","s+":"69"},{"a":"7FFC70A966C0","module":"VCRUNTIME140.dll","s":"CxxThrowException","s+":"90"},{"a":"7FF66F5EA219","module":"mongod.exe","file":".../src/third_party/boost/libs/filesystem/src/operations.cpp","line":724,"s":"boost::filesystem::detail::`anonymous namespace'::process_status_failure","s+":"129"},{"a":"7FF66F5EB5DF","module":"mongod.exe","file":".../src/third_party/boost/libs/filesystem/src/operations.cpp","line":2653,"s":"boost::filesystem::detail::status","s+":"6F"},{"a":"7FF66DE4E7A5","module":"mongod.exe","file":".../src/mongo/db/ftdc/file_manager.cpp","line":256,"s":"mongo::FTDCFileManager::recoverInterimFile","s+":"65"},{"a":"7FF66DE4D847","module":"mongod.exe","file":".../src/mongo/db/ftdc/file_manager.cpp","line":91,"s":"mongo::FTDCFileManager::create","s+":"387"},{"a":"7FF66DE51463","module":"mongod.exe","file":".../src/mongo/db/ftdc/controller.cpp","line":246,"s":"mongo::FTDCController::doLoop","s+":"2A3"},{"a":"7FF66DE5105C","module":"mongod.exe","file":"C:/Program Files (x86)/Microsoft Visual Studio/2019/Professional/VC/Tools/MSVC/14.29.30133/include/thread","line":56,"s":"std::thread::_Invoke<std::tuple<<lambda_726633d7e71a0bdccc2c30a401264d8c> >,0>","s+":"2C"},{"a":"7FFC7A7F268A","module":"ucrtbase.dll","s":"o_exp","s+":"5A"},{"a":"7FFC7C4A7AC4","module":"KERNEL32.DLL","s":"BaseThreadInitThunk","s+":"14"}]}}}
{"t":{"$date":"2024-05-14T14:03:08.203-06:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"ftdc","msg":"Frame","attr":{"frame":{"a":"7FF66F51EC31","module":"mongod.exe","file":".../src/mongo/util/stacktrace_windows.cpp","line":333,"s":"mongo::printStackTrace","s+":"41"}}}
{"t":{"$date":"2024-05-14T14:03:08.203-06:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"ftdc","msg":"Frame","attr":{"frame":{"a":"7FF66F522007","module":"mongod.exe","file":".../src/mongo/util/signal_handlers_synchronous.cpp","line":244,"s":"mongo::`anonymous namespace'::myTerminate","s+":"D7"}}}
{"t":{"$date":"2024-05-14T14:03:08.203-06:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"ftdc","msg":"Frame","attr":{"frame":{"a":"7FF66F5EE3A7","module":"mongod.exe","file":".../src/mongo/stdx/set_terminate_internals.cpp","line":88,"s":"mongo::stdx::dispatch_impl","s+":"17"}}}
{"t":{"$date":"2024-05-14T14:03:08.203-06:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"ftdc","msg":"Frame","attr":{"frame":{"a":"7FF66F5EE389","module":"mongod.exe","file":".../src/mongo/stdx/set_terminate_internals.cpp","line":92,"s":"mongo::stdx::TerminateHandlerDetailsInterface::dispatch","s+":"9"}}}
{"t":{"$date":"2024-05-14T14:03:08.203-06:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"ftdc","msg":"Frame","attr":{"frame":{"a":"7FFC7A83DE58","module":"ucrtbase.dll","s":"terminate","s+":"18"}}}
{"t":{"$date":"2024-05-14T14:03:08.203-06:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"ftdc","msg":"Frame","attr":{"frame":{"a":"7FFC70AB1AAB","module":"VCRUNTIME140_1.dll","s":"_NLG_Return2","s+":"95B"}}}
{"t":{"$date":"2024-05-14T14:03:08.203-06:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"ftdc","msg":"Frame","attr":{"frame":{"a":"7FFC70AB2317","module":"VCRUNTIME140_1.dll","s":"_NLG_Return2","s+":"11C7"}}}
{"t":{"$date":"2024-05-14T14:03:08.203-06:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"ftdc","msg":"Frame","attr":{"frame":{"a":"7FFC70AB4119","module":"VCRUNTIME140_1.dll","s":"_CxxFrameHandler4","s+":"A9"}}}
{"t":{"$date":"2024-05-14T14:03:08.203-06:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"ftdc","msg":"Frame","attr":{"frame":{"a":"7FF66F7F7C10","module":"mongod.exe","file":"D:/a/_work/1/s/src/vctools/crt/vcstartup/src/gs/amd64/gshandlereh4.cpp","line":86,"s":"__GSHandlerCheck_EH4","s+":"64"}}}
{"t":{"$date":"2024-05-14T14:03:08.203-06:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"ftdc","msg":"Frame","attr":{"frame":{"a":"7FFC7E4A4C2F","module":"ntdll.dll","s":"_chkstk","s+":"11F"}}}
{"t":{"$date":"2024-05-14T14:03:08.203-06:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"ftdc","msg":"Frame","attr":{"frame":{"a":"7FFC7E404CEF","module":"ntdll.dll","s":"RtlWalkFrameChain","s+":"14BF"}}}
{"t":{"$date":"2024-05-14T14:03:08.203-06:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"ftdc","msg":"Frame","attr":{"frame":{"a":"7FFC7E408AE6","module":"ntdll.dll","s":"RtlRaiseException","s+":"316"}}}
{"t":{"$date":"2024-05-14T14:03:08.203-06:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"ftdc","msg":"Frame","attr":{"frame":{"a":"7FFC7A4C1B29","module":"KERNELBASE.dll","s":"RaiseException","s+":"69"}}}
{"t":{"$date":"2024-05-14T14:03:08.203-06:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"ftdc","msg":"Frame","attr":{"frame":{"a":"7FFC70A966C0","module":"VCRUNTIME140.dll","s":"CxxThrowException","s+":"90"}}}
{"t":{"$date":"2024-05-14T14:03:08.203-06:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"ftdc","msg":"Frame","attr":{"frame":{"a":"7FF66F5EA219","module":"mongod.exe","file":".../src/third_party/boost/libs/filesystem/src/operations.cpp","line":724,"s":"boost::filesystem::detail::`anonymous namespace'::process_status_failure","s+":"129"}}}
{"t":{"$date":"2024-05-14T14:03:08.203-06:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"ftdc","msg":"Frame","attr":{"frame":{"a":"7FF66F5EB5DF","module":"mongod.exe","file":".../src/third_party/boost/libs/filesystem/src/operations.cpp","line":2653,"s":"boost::filesystem::detail::status","s+":"6F"}}}
{"t":{"$date":"2024-05-14T14:03:08.203-06:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"ftdc","msg":"Frame","attr":{"frame":{"a":"7FF66DE4E7A5","module":"mongod.exe","file":".../src/mongo/db/ftdc/file_manager.cpp","line":256,"s":"mongo::FTDCFileManager::recoverInterimFile","s+":"65"}}}
{"t":{"$date":"2024-05-14T14:03:08.203-06:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"ftdc","msg":"Frame","attr":{"frame":{"a":"7FF66DE4D847","module":"mongod.exe","file":".../src/mongo/db/ftdc/file_manager.cpp","line":91,"s":"mongo::FTDCFileManager::create","s+":"387"}}}
{"t":{"$date":"2024-05-14T14:03:08.203-06:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"ftdc","msg":"Frame","attr":{"frame":{"a":"7FF66DE51463","module":"mongod.exe","file":".../src/mongo/db/ftdc/controller.cpp","line":246,"s":"mongo::FTDCController::doLoop","s+":"2A3"}}}
{"t":{"$date":"2024-05-14T14:03:08.203-06:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"ftdc","msg":"Frame","attr":{"frame":{"a":"7FF66DE5105C","module":"mongod.exe","file":"C:/Program Files (x86)/Microsoft Visual Studio/2019/Professional/VC/Tools/MSVC/14.29.30133/include/thread","line":56,"s":"std::thread::_Invoke<std::tuple<<lambda_726633d7e71a0bdccc2c30a401264d8c> >,0>","s+":"2C"}}}
{"t":{"$date":"2024-05-14T14:03:08.203-06:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"ftdc","msg":"Frame","attr":{"frame":{"a":"7FFC7A7F268A","module":"ucrtbase.dll","s":"o_exp","s+":"5A"}}}
{"t":{"$date":"2024-05-14T14:03:08.203-06:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"ftdc","msg":"Frame","attr":{"frame":{"a":"7FFC7C4A7AC4","module":"KERNEL32.DLL","s":"BaseThreadInitThunk","s+":"14"}}}
{"t":{"$date":"2024-05-14T14:03:08.203-06:00"},"s":"F",  "c":"CONTROL",  "id":23134,   "ctx":"ftdc","msg":"Unhandled exception","attr":{"exceptionString":"0xE0000001","addressString":"0x00007FFC7A4C1B29"}}

Does anything jump out here to anyone? I can’t tell if there is a pattern to when the service stops, or why.

Thank you in advance!

Can you send Microsoft Event Error logs or check them?

  1. Client Compatibility:
  • Ensure that client drivers are compatible with the MongoDB server version. The Deprecated operation requested message suggests that the client driver may need an upgrade.

  1. Initialization and Configuration:
  • The logs show that the MongoDB server was initializing various configurations and services.
  • The id=4915701 message indicates that wire protocol specifications were initialized successfully.
  • The id=23285 message indicates that TLS 1.0 was automatically disabled.
  1. Warnings and Potential Issues:
  • Several warnings (id=22601) mention “No TransportLayer configured during NetworkInterface startup.” This indicates potential issues with the network interface configuration.
  • The id=22271 message indicates that an unclean shutdown was detected due to a non-empty lock file, suggesting that the previous MongoDB shutdown was not clean.
  1. Recovery Attempts:
  • Multiple WiredTiger message entries (id=22430) indicate attempts to recover data from logs and checkpoints.
  • The recovery process seems to progress normally, and the WiredTiger opened message (id=4795906) indicates successful initialization of the storage engine.
  1. Client Connections:
  • Multiple entries (id=51800) show client metadata being logged, indicating that clients were attempting to connect to the MongoDB server.
  1. Fatal Error:
  • The critical error occurs with id=6384300 and id=6384300 again, indicating that terminate() was called due to an active exception. The exception type is std::exception, which suggests a general exception occurred without specific details.
  • The backtrace provided (id=31380) lists the stack frames leading to the exception, involving MongoDB’s internal functions, standard C++ libraries, and the Visual Studio runtime.

Hey Derek,

Thanks for looking into this and for the info!

The Event Logs for Windows show the time of the Deadline failure, but the log doesn’t give any details, it just says “The Deadline 10 Database Service service terminated unexpectedly. It has done this 1 time(s).”

Following the log entries for Deadline, there are a handful of normal event logs for Background Intelligent Transfer Service and AppXSVC, but nothing really stands out in the entries leading up to the crash, just normal entries from the Service Control Manager.

The two potential issues you mention are interesting - is it possible that a backup service could cause MongoDB to crash due to an unavailable file? I found a few mentions of this possible reason for the issue online on various MongoDB-related forums. To test, I’ve added an exclusion in Microsoft Defender to avoid scanning the repo directories, and I’ve also excluded the repo from our backup sets.

Yes, so my next suggestion would be to definitely disabled any kind of virus scan.

And is your repo on this machine or is it located on a network attached storage?

Cuz it almost looks like it’s getting some sort of interruption when it’s trying to communicate which could be an antivirus or just a network error.

Hey Derek, just an update for you and for anyone else reading this from the future: Excluding the repository from Windows Defender antivirus and excluding it from our backup sets seems to have worked! The Deadline MongoDB service has not crashed since I made that change. I wish I knew which system was interfering with the MongoDB processes, but in the end it doesn’t really matter. What matters is that it works!

1 Like
Privacy | Site terms | Cookie preferences