Mongo service crashed. Need to find reason of crash

Question

Today morning I found the following error with mongo, on my server.

*** System restart required ***
You have mail.
ubuntu@ip-xxx-xx-xx-xx:~$ mongo
MongoDB sh         


        

Answer 1

I had the same issue. After some searching I guessed it was os that sacrificed mongod for memory. If kernel killed your mongod you can find the logs in kernel logs

Mostly kernel logs will be in /var/log/kern.log, The logs might be rotated.

So better to do a grep

grep mongod /var/log/kern.log*

if the logs are archived find -name \*kern.log.*.gz -print0 | xargs -0 zgrep "mongod"

./kern.log.1:Nov 12 12:55:04 ip-xxx-xxx-xxx-xxx kernel: [8413229.118361] mongod invoked oom-killer: gfp_mask=0x24201ca, order=0, oom_score_adj=0
./kern.log.1:Nov 12 12:55:04 ip-xxx-xxx-xxx-xxx kernel: [8413229.118365] mongod cpuset=/ mems_allowed=0
./kern.log.1:Nov 12 12:55:04 ip-xxx-xxx-xxx-xxx kernel: [8413229.118371] CPU: 0 PID: 1830 Comm: mongod Not tainted 4.4.0-1062-aws #71-Ubuntu
./kern.log.1:Nov 12 12:55:04 ip-xxx-xxx-xxx-xxx kernel: [8413229.118531] [ 1301]   112  1301   300787   181207     485       4        0             0 mongod
./kern.log.1:Nov 12 12:55:04 ip-xxx-xxx-xxx-xxx kernel: [8413229.118680] Out of memory: Kill process 1301 (mongod) score 355 or sacrifice child
./kern.log.1:Nov 12 12:55:04 ip-xxx-xxx-xxx-xxx kernel: [8413229.130161] Killed process 1301 (mongod) total-vm:1203148kB, anon-rss:724828kB, file-rss:0kB

Answer 2

According to MongoDB documentation

The getLog command returns a document with a log array that contains recent messages from the mongod process log.

Please try executing following command in mongo shell to return logs which consist of information related to errors or warnings while initialization of mongod service

db.adminCommand({getLog: "startupWarnings" })

For detailed description regarding getLog method please refer the documentation mentioned in following URL

https://docs.mongodb.org/manual/reference/command/getLog/

Answer 3

My server also crashed for no obvious reason. The other answers did not throw anything. But I found the following looking at the mongod.conf file in /etc/.

There are also logs inside /var/log/mongodb/mongod.log. It seems that by default it has a 500 Mb limit. But is still a big file.

There I found that my server crashed at a certain time, but couldn't find the reason.

2019-04-17T19:39:11.700+0000 I COMMAND  [conn460] command cge_info_system.expedientes command: find { find: "expedientes", filter: { NoExpediente: "090311-497449" }, limit: 1, $db: "cge_info_system", $readPreference: { mode: "primaryPreferred" }, lsid: { id: UUID("f18d952b-37e7-4a45-bf31-b89a596057d1") } } planSummary: COLLSCAN keysExamined:0 docsExamined:49360 cursorExhausted:1 numYields:386 nreturned:1 reslen:3224 locks:{ Global: { acquireCount: { r: 387 } }, Database: { acquireCount: { r: 387 } }, Collection: { acquireCount: { r: 387 } } } protocol:op_msg 1416ms
2019-04-23T16:35:19.578+0000 I CONTROL  [main] ***** SERVER RESTARTED *****
2019-04-23T16:35:19.599+0000 I CONTROL  [main] Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'
2019-04-23T16:35:19.958+0000 I CONTROL  [initandlisten] MongoDB starting : pid=741 port=27017 dbpath=/var/lib/mongodb 64-bit host=bb7dd22f-cb77-4c6b-9c4c-9fddcfc2cc00.clouding.host
2019-04-23T16:35:19.958+0000 I CONTROL  [initandlisten] db version v4.0.5
2019-04-23T16:35:19.958+0000 I CONTROL  [initandlisten] git version: 3739429dd92b92d1b0ab120911a23d50bf03c412
2019-04-23T16:35:19.958+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.1.0g  2 Nov 2017
2019-04-23T16:35:19.958+0000 I CONTROL  [initandlisten] allocator: tcmalloc
2019-04-23T16:35:19.958+0000 I CONTROL  [initandlisten] modules: none
2019-04-23T16:35:19.958+0000 I CONTROL  [initandlisten] build environment:
2019-04-23T16:35:19.958+0000 I CONTROL  [initandlisten]     distmod: ubuntu1804
2019-04-23T16:35:19.958+0000 I CONTROL  [initandlisten]     distarch: x86_64
2019-04-23T16:35:19.958+0000 I CONTROL  [initandlisten]     target_arch: x86_64
2019-04-23T16:35:19.958+0000 I CONTROL  [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1", port: 27017 }, processManagement: { timeZoneInfo: "/usr/share/zoneinfo" }, storage: { dbPath: "/var/lib/mongodb", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
2019-04-23T16:35:19.961+0000 W STORAGE  [initandlisten] Detected unclean shutdown - /var/lib/mongodb/mongod.lock is not empty.
2019-04-23T16:35:19.962+0000 I STORAGE  [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2019-04-23T16:35:19.962+0000 W STORAGE  [initandlisten] Recovering data from the last clean checkpoint.
2019-04-23T16:35:19.962+0000 I STORAGE  [initandlisten]

In there is also a reference to a directory called /var/lib/mongodb/diagnostic.data but the data in there are binary files. I haven't got much info from there.