This is part 5/6 of the MongoDB sharded cluster write path series.
Questions this page covers: Why does automatic failover not happen when the primary cannot finish a single write? Why do Flow Control and Storage Node Watchdog fail to catch this state? What should you monitor, then?
Reference version: MongoDB 4.2 (official documentation
v4.2branch, server source tagr4.2.12). Changes in 7.0 are noted separately in the relevant sections.
In MongoDB 4.2, three mechanisms react automatically to primary problems. The three look at whether the primary is alive and whether secondaries have fallen behind. A state in which the primary replies but cannot finish writes does not meet any of their criteria.
| Mechanism | What it looks at | When it sees a primary whose writes have stalled |
|---|---|---|
| Automatic failover (election) | Whether heartbeats get a response | Heartbeats get a response, so there is no election |
| Flow Control | majority commit delay (lag) | lag is close to 0, so it judges the state as normal |
| Storage Node Watchdog | Filesystem unresponsiveness | The filesystem is fine, so it does not trigger |
So the operator has to monitor progress directly. The question to monitor is not "does it reply?" but "does it finish writes?".
The cause of this state lies inside the storage engine WiredTiger (the engine, below). Specifically, it is the livelock of the WiredTiger snapshot pause loop. A livelock is a state where threads keep spinning and using CPU but the work never finishes. That cause is covered in part 3/6. This page covers why the three mechanisms do not react to this state and how to design monitoring.
Replica set members send heartbeats to each other. The 4.2 documentation says:
Replica set members send heartbeats (pings) to each other every two seconds. If a heartbeat does not return within 10 seconds, the other members mark the delinquent member as inaccessible.
| Item | Value |
|---|---|
| Heartbeat interval | 2 seconds |
| Time without a response before a member is considered inaccessible | 10 seconds |
| Loss-of-contact time that triggers an election | electionTimeoutMillis, default 10 seconds |
An election starts when a secondary loses contact with the primary for longer than electionTimeoutMillis. The election rule has one input: "do heartbeats come back?". Whether the primary is processing writes is not an input.
The code that handles the heartbeat command takes neither the global lock nor a ticket. A ticket is an admission pass that limits how many operations can enter the engine at the same time. The handler is CmdReplSetHeartbeat::run in src/mongo/db/repl/repl_set_commands.cpp (r4.2.12). After parsing the arguments, this function only calls processHeartbeatV1.
Ordinary writes are different. insert, update, delete, and findAndModify take one write ticket while taking the global lock in IX (intent to write) mode (src/mongo/db/concurrency/lock_state.cpp:262-266 (r4.2.12)). The default number of write tickets in 4.2 is 128 (src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp:474-475 (r4.2.12)).
Even if all 128 operations holding write tickets are stuck inside the engine, heartbeats do not wait for a ticket. So heartbeats can respond normally even when the engine has stalled. This conclusion is an inference from reading the source. It was not reproduced directly.
For example, while not a single write from the writer service that writes to the store_inventory collection finishes, the secondaries still receive a heartbeat response every 2 seconds. Nothing changes in the inputs of the election rule.
Flow Control is a feature introduced in 4.2. The majority commit delay is how far behind the confirmation of a write by a majority of members is, relative to the primary applying it. Flow Control slows the primary's write rate to keep this delay below a set value.
With flow control enabled, as the lag grows close to the flowControlTargetLagSeconds, writes on the primary must obtain tickets before taking locks to apply writes.
The defaults are enableFlowControl: true and flowControlTargetLagSeconds: 10. It works only when FCV (featureCompatibilityVersion) is 4.2 and majority read concern is enabled.
The lag formula is in the source (src/mongo/db/storage/flow_control.cpp:77-82 (r4.2.12)).
return static_cast<std::uint64_t>(durationCount<Milliseconds>(myLastApplied - lastCommitted));
myLastApplied is the time of the last write the primary applied. lastCommitted is the time of the last write confirmed by a majority of members. If this difference is smaller than the threshold (flowControlTargetLagSeconds), Flow Control judges the state as normal and increases tickets (lines 328-350 of the same file). It also judges the state as normal when there is almost no work between the two times.
When the primary cannot write, things go like this.
myLastApplied stops.lastCommitted becomes equal to myLastApplied.The flow above is an inference drawn from the formula. Work through it with example values. The primary applied its last write at 10:00:00. A secondary replicated it, so lastCommitted also became 10:00:00. If the primary applies nothing after that, lag is fixed at 10:00:00 − 10:00:00 = 0초. That is far below the 10-second threshold.
Even when lag grows and Flow Control slows writes, this state is not released. Flow Control reduces the amount of writes coming in at the engine entrance. It cannot touch operations that have already received a ticket and entered the engine.
Storage Node Watchdog (Watchdog, below) is also in the Community edition starting with 4.2. What it monitors is filesystem unresponsiveness in the following directories.
--dbpath directoryjournal directory inside --dbpath (if journaling is enabled)--logpath file--auditPath fileIf the filesystem stops responding, Watchdog terminates mongod (exit code 61). If that mongod was the primary, a failover happens.
Watchdog is off by default. The default of watchdogPeriodSeconds is -1 (disabled), and to turn it on you must give an integer of 60 or more when starting mongod.
Even while the CPU is tied up in a loop inside the engine, the disk and filesystem respond normally. So even if Watchdog is on, it does not trigger in this situation. This conclusion is an inference drawn from the monitored targets in the documentation.
No other feature that demotes the primary when the engine cannot make progress was found in the 4.2 documentation. The documentation and parameters were not checked exhaustively.
The structure of "replies, but cannot finish work" also shows up often with Kubernetes liveness probes. Say a probe checks only /health. If the HTTP thread replies, the probe judges the state as normal even when the work queue has stalled. The MongoDB heartbeat plays the same role as this /health probe.
To see progress, look at three signals together.
Looking at only one gives false alarms. Looking only at throughput, it fires in the early morning when writes are naturally few. Looking only at the queue, it fires even on a normal momentary burst.
The paths below are paths in the result of db.serverStatus(). Read them on each shard's primary.
| serverStatus path | Meaning | Expected shape on a stuck primary |
|---|---|---|
wiredTiger.concurrentTransactions.write.available |
Number of write tickets left | Stays near 0 |
opcounters.insert·.update·.delete |
Cumulative processed count | Stops growing or drops sharply |
opLatencies.writes.latency·.ops |
Cumulative write latency and count | Interval average latency spikes |
globalLock.currentQueue.writers |
Number of writes waiting | Keeps growing |
There are three expected shapes on the host.
When iowait is close to 0, the CPU is being used to execute instructions, not waiting on disk. Context switches drop because the pause loop makes no system calls, so threads do not give up the CPU on their own. These two interpretations are inferences drawn from the engine source analysis in part 3/6.
globalLock.currentQueue.writers counts lock waits and ticket waits together. In the r4.2.12 source, this value is the number of clients whose state is kQueuedWriter (src/mongo/db/stats/lock_server_status_section.cpp:76-83). In lock_state.cpp, there are three cases that lead to this state.
Do not conclude lock contention just because this value rose. The same value rises when tickets run out.
Below is pseudocode. The metric names are written as serverStatus paths. The actual metric names emitted by the collector (exporter) were not checked, so adapt them to your environment. All thresholds are example values.
# 의사 코드. 각 shard primary마다 평가한다. 임계치는 예시값.
ALERT MongoPrimaryWriteStalled
WHEN
# 1) 엔진 입구가 막혔다
wiredTiger.concurrentTransactions.write.available <= 2
# 2) 들어오려는 쓰기가 있다
AND globalLock.currentQueue.writers > 10
# 3) 처리량이 늘지 않는다 (1분 증가율이 지난주 같은 시각의 10% 미만)
AND rate(opcounters.insert + opcounters.update + opcounters.delete, 1m)
< 0.1 * baseline_same_hour_last_week
# 4) CPU는 명령 실행에 묶여 있다
AND host.cpu.user >= 90%
AND host.cpu.iowait < 5%
FOR 2m
SEVERITY page
NOTE "heartbeat가 정상이어도 자동 failover는 일어나지 않는다. 대응 절차 문서를 연다."
The interval average write latency is computed from the difference of cumulative values.
avg_write_latency_us(1m) =
delta(opLatencies.writes.latency, 1m) / delta(opLatencies.writes.ops, 1m)
If not a single write finishes, the denominator is 0. In that case, use the fact that the completed count is 0 directly as the signal.
From 7.0, condition 1 cannot be used as is. 7.0 adjusts the number of tickets dynamically. So the 7.0 documentation says that the number of tickets left (ticket availability) staying at 0 for a long time is not considered an overload signal. In 6.0 and earlier, the same situation is an overload signal. On 7.0 and later, focus on conditions 2-4.
serverStatus metrics are indirect signals. A probe that periodically sends a small write and measures whether it finishes is more direct. Below is a mongo shell example that was not verified by running it.
// 예시: 외부 감시 프로세스가 30초마다 실행한다.
// ops_probe 데이터베이스의 probe 컬렉션에 문서 한 건을 갱신한다.
const started = new Date();
db.getSiblingDB("ops_probe").probe.updateOne(
{ _id: "writer-probe" },
{ $set: { at: started } },
{ upsert: true }
);
const elapsedMs = new Date() - started;
// elapsedMs가 예시 임계치(2000ms)를 넘거나 클라이언트 타임아웃으로 실패하면 알린다.
maxTimeMS is a cooperative interruption. An operation stops only when it checks the stop request for itself, so it cannot cut off an operation stuck inside the engine (see part 4/6).Problems often show up in the application before they show up in primary metrics. The order expected in general terms is as follows.
How application retries pile more waiting work onto the server is covered in part 2/6.
- Do not rule out the DB just because heartbeats and
rs.status()show green. Heartbeats do not go through the engine, so the primary status display also stays normal (inference based on the source).- Changing Flow Control settings does not release this state. Flow Control only slows writes at the entrance when lag is large.
- Turning on Watchdog does not detect this state. Watchdog looks only at filesystem unresponsiveness.
- An alert on throughput drop alone produces many false alarms. Combine it with queue growth.
- Whether 4.2 really has no feature that demotes the primary based on engine progress was not checked exhaustively against the documentation and parameters.
- The conclusion that heartbeats respond while the engine is stalled is an inference from reading the r4.2.12 source. It was not reproduced directly.
- The field definitions of
opcountersandopLatencies.writeswere not checked against the original 4.2 documentation.- The exact code and name of the error mongos returns when it cannot get a shard connection were not checked.
- The actual metric names used by the collector (exporter) were not checked.
- The concrete way to keep a write probe per shard in a sharded cluster, and its side effects, were not verified.
- Whether the heartbeat handling path and the Flow Control formula changed in 7.0 and later was not checked.