This is part 4/6 of the MongoDB sharded cluster write path series.
Questions this page covers: When writes on the primary have stalled, why do killOp, maxTimeMS, replSetStepDown, and a clean shutdown (SIGTERM) have no effect? What should you do then, and in what order?
Reference version: MongoDB 4.2. The official documentationv4.2branch and server source tagr4.2.12were read. The latest versions were not checked.
Most of the ways MongoDB stops an operation are cooperative. They leave a kill mark and wait for the target thread to check that mark. A thread running inside the storage engine (WiredTiger) cannot check the mark, so neither killOp, replSetStepDown, nor a clean shutdown can stop that thread. The only remaining tool is kill -9, which the official documentation forbids. So you must keep the order: first stop the writers (API servers, batches, and consumers that send writes to the cluster), then take down the primary.
An interrupt point is a place where an operation can safely stop. This is the definition from the 4.2 glossary.
"A point in an operation's lifecycle when it can safely abort. MongoDB only terminates an operation at designated interrupt points."
In the server source, one operation is one OperationContext object. killOp only calls markKilled on this object. According to the source comment, from then on the operation thread gets the specified error code back every time it calls checkForInterrupt (src/mongo/db/operation_context.h:261-272 (r4.2.12)).
The killOp documentation puts it this way.
"Operations may not actually be terminated until they reach an appropriate interruption point."
You have to check again with db.currentOp() whether the operation has ended. The ok: 1 that killOp returns means the mark was set successfully.
Before a write operation enters the engine, it gets a write ticket (an admission pass for concurrent writes, 128 by default in 4.2). While waiting for a ticket, it can be interrupted. The 4.2 source passes the OperationContext to the ticket wait so that the kill mark is checked even while waiting (src/mongo/db/concurrency/lock_state.cpp:343-349 (r4.2.12)).
maxTimeMS works the same way. The documentation says that once the deadline passes, "MongoDB interrupts the operation at the next interrupt point".
What killOp and maxTimeMS cut off reaches only as far as operations waiting in the ticket line. They cannot cut off operations that have already received a ticket and entered the engine. WiredTiger's snapshot wait loop has no checkForInterrupt call (src/third_party/wiredtiger/src/txn/txn.c:196-235 (r4.2.12)). How this loop spins without end is covered in WiredTiger snapshot pause loop livelock.
In a sharded cluster there is one more pitfall. A killOp sent from mongos does not propagate write operations to the shards.
"The killOp command from the mongos does not propagate to the shards when the operation to be killed is a write operation."
A write operation has to be killed by connecting directly to that shard's primary mongod.
Start with two lock modes.
| Mode | Meaning | Can several operations hold it at once? |
|---|---|---|
| IX (intent exclusive) | A shared mode that declares an intent to write | Yes |
| X (exclusive) | Exclusive mode | Only one holder |
The side that wants X waits until every operation holding IX releases it. For example, if 128 operations hold IX, all 128 must release it before X is granted. Even if 127 finish, X is not granted while 1 remains.
In 4.2, for a primary to step down to secondary, it must hold the RSTL (ReplicationStateTransitionLock, the replication state transition lock) in X. Every read and write operation takes the RSTL in IX before taking the global lock (a lock over the whole mongod instance).
secondaryCatchUpPeriodSecs (default 10 seconds) is how long to wait for a secondary to catch up with the primary.
The stepDown flow in replication_coordinator_impl.cpp (r4.2.12) is as follows.
force, the deadline is secondaryCatchUpPeriodSecs; with force: true, it is the whole stepDown period (lines 2054-2058).Starting to kill user operations and puts a kill mark on operations that conflict with writes (lines 1887-1913, line 1946).LockTimeout (Unable to acquire lock ... within ...) (lock_state.cpp:964-966).The mark in step 3 is cooperative, just like killOp. A thread inside the engine cannot check the mark, so it does not release RSTL IX. So step 4 hits the deadline. force: true only extends the deadline; it still needs RSTL X.
According to the documentation, stepDown also returns an error when no secondary catches up within secondaryCatchUpPeriodSecs. If the error message is Unable to acquire lock, it was blocked on RSTL X, not on catch-up.
The 4.2 documentation says that when a primary receives SIGTERM, it tries to step down before shutting down. Even if stepDown fails, the shutdown continues (introduced in 3.6.15, 4.0.8, and 4.1.10 by SERVER-38994).
The shutdown path in src/mongo/db/db.cpp (r4.2.12) proceeds in this order.
Enqueuing the ReplicationStateTransitionLock for shutdown # RSTL X 요청
Killing all operations for shutdown # 전체 종료 표시
Acquiring the ReplicationStateTransitionLock for shutdown # RSTL X 대기, Date_t::max() (1013-1029행)
Acquiring the global lock for shutdown # 글로벌 X 대기, Date_t::max() (1102-1106행)
Date_t::max() is a value with no time limit. So if a thread inside the engine does not release the RSTL and the global lock, the clean shutdown never finishes. This is an inference drawn from reading the source.
When you send SIGKILL, the operating system ends the process immediately without asking for cooperation. The 4.2 documentation says this in a warning box.
"Never use kill -9 (i.e. SIGKILL) to terminate a mongod instance."
The journal recovery behavior after kill -9 was not checked. So kill -9 is kept as a last resort, used only after diagnosis (step 1 of "Response order" below) is done.
| Command | What it does | Does it reach a thread inside the engine? | What it cuts off instead |
|---|---|---|---|
killOp |
Puts a kill mark on the target operation | No | Operations waiting for a ticket |
maxTimeMS |
Stops at the next interrupt point after the deadline | No | Operations in the ticket line |
replSetStepDown |
Puts a kill mark on write operations, then waits for RSTL X | No (LockTimeout after the deadline) |
Write operations that reach an interrupt point |
| SIGTERM | Tries stepDown, marks everything for kill, waits for locks with no limit | No (the shutdown does not finish, inference) | Operations that reach an interrupt point |
kill -9 |
The operating system ends the process immediately | Yes | Every operation in progress |
When a cooperative tool cannot stop the target thread, it looks like this.
ok: 1, but the same opid stays in db.currentOp().replSetStepDown fails after the deadline (by default secondaryCatchUpPeriodSecs). The error is LockTimeout, and the log contains Starting to kill user operations.If the primary appears to be stuck inside the engine, the operator follows the order below.
kill -9 to the stuck primary mongod. Confirm it by checking that each writer's processing metric has dropped to 0. If you are not sure, hold off on kill -9 and wait. After kill -9 the heartbeat stops, so the secondaries start an election (electionTimeoutMillis default 10 seconds).rs.status() that another member is PRIMARY. Then turn the writers on one at a time. Each time you turn one on, check write ticket headroom and throughput before moving to the next writer.There are three things not to do.
Do not take down only the primary while writes remain. If you skip step 2 and start with kill -9, the same stall can move to the new primary.
Keep a one-page list ready in advance: "who writes to this cluster, and how do you stop each one". Below is a format example, and all values are hypothetical.
| Writer | Type | How to stop it | Resume order |
|---|---|---|---|
store-inventory-writer |
Kafka consumer | Scale the consumer replicas down to 0 | 2 |
| Change event CDC connector | CDC | Pause the connector | 1 |
Also write down in the list the metric that confirms each one has stopped. A writer missing from the list keeps writing to the new primary even during the response.
This procedure was not verified by running it. The results below are expectations based on the documentation and the source.
Reproducing a spin inside the engine is covered in part 3/6, WiredTiger snapshot pause loop livelock. Here, on a 3-member test replica set, we look at how each tool responds to a long write that passes through interrupt points.
Insert 1 million documents (example value) into a collection.
use test
for (let b = 0; b < 100; b++)
db.store_inventory.insertMany(Array.from({ length: 10000 },
(_, j) => ({ _id: { productId: j % 1000, storeId: b * 10000 + j }, qty: 0 })))
In shell A, run a long write.
db.store_inventory.updateMany({}, { $inc: { qty: 1 } })
In shell B, find the opid and send killOp.
db.currentOp({ op: "update", ns: "test.store_inventory" }).inprog.map(o => o.opid)
db.killOp(<opid>)
Expected result: killOp returns ok: 1. The updateMany in shell A ends with an interruption error at the next interrupt point.
Run step 2 again and send stepDown from shell B.
db.adminCommand({ replSetStepDown: 60, secondaryCatchUpPeriodSecs: 10 })
Expected result: Starting to kill user operations appears in the log. updateMany receives the kill mark and ends. The primary steps down to secondary.
Run step 2 again and send SIGTERM (kill -15) to that mongod.
Expected result: the four lines of the shutdown path appear in the log in order, and the process exits.
On a sharded cluster, try steps 2-3 from mongos. The expected result is that the operation remains on the shard primary.
- How far mongod recovers from the journal and the last checkpoint after
kill -9was not checked.- The conclusion that a SIGTERM shutdown does not finish is an inference from reading
db.cpp. Actual logs were not examined.- The stepDown and shutdown paths in versions after 4.2 were not checked.