How do the MongoDB journal file and oplog differ?

Question

I have recently started on MongodDb and I\'m trying to explore on replica sets and crash recovery.

I have read it like journal file are write a head redo log file. oplog

Answer 1

Oplog is just capped collection where MongoDB tracks all changes in its collections (insert, update, delete). It doesn't track read operations. MongoDB uses oplog to spread all changes within all nodes in a replica set. Secondary nodes copy and apply this changes.

Journal is a feature of underlying storage engine. Since MongoDB 3.2 default storage engine is WiredTiger and since MongoDB 4.0 you can't disable journaling for WiredTiger. All operations are tracked in the journal files. WiredTiger uses checkpoints to recover data in case of crash. Checkpoints are created every 60 secs. In case if a crash happens between checkpoints some data can be lost. To prevent this, WiredTiger uses journal files to apply all the changes after the last checkpoint.

In general, write flow in MongoDB looks like that:

• High-level - when a customer writes/updates/removes data, MongoDB applies it to proper collection, updates index and inserts the change to oplog. If any of these operations fails then other related operations must be rolled back to prevent inconsistency. For this MongoDB uses WiredTiger transactions:
  1. begin transaction
  2. apply change to collection
  3. update index
  4. add the change to the oplog
  5. commit the transaction
• Low-level - WiredTiger runs the transaction and adds the changes to journal file.

Answer 2

There must be a relationship between journal and oplog. when w=1 u commit to primary jouranl and also created an oplog entry for replset collection. I think at least in primary of a replSet - they both contain same update/delete/insert just in different format.

Answer 3

Oplog stores high-level transactions that modify the database (queries are not stored for example), like insert this document, update that, etc. Oplog is kept on the master and slaves will periodically poll the master to get newly performed operations (since the last poll). Operations sometimes get transformed before being stored in the oplog so that they are idempotent (and can be safely applied many times).

Journal on the other hand can be switched on/off on any node (master or slave), and is a low-level log of an operation for the purpose of crash recovery and durability of a single mongo instance. You can read low-level op like 'write these bytes to this file at this position'.

NOTE: Starting in MongoDB 4.0, you cannot turn journaling off for replica set members that use the WiredTiger storage engine. Source: https://docs.mongodb.com/manual/tutorial/manage-journaling/