This is part 3/6 of the MongoDB sharded cluster write path series.
Question this page covers: MongoDB's CPU is at 100%, yet not a single write finishes. The disk is idle and it is not a deadlock. What is using the CPU?
Short version: WiredTiger builds a snapshot every time a transaction starts. If another session is in the middle of receiving a transaction ID at that moment, the thread building the snapshot waits by repeating only the x86pauseinstruction. If the thread that was receiving the ID is pushed off the CPU, the waiting threads take over the CPUs, making it hard for that thread to come back. Everyone is busy, but no one makes progress: a livelock.
Reference version: MongoDB 4.2 (source tagr4.2.12, bundled WiredTigermongodb-4.2branch). The WiredTigerdevelopbranch was checked on 2026-11-05.
A write ticket is the upper limit on how many operations MongoDB lets into the storage engine (WiredTiger) at the same time. Like an admission pass, only operations holding a ticket enter the engine, and the rest wait in line.
| Item | As of 4.2 |
|---|---|
| Default | 128 for writes, 128 for reads (wiredtiger_kv_engine.cpp:474-475) |
| Setting | wiredTigerConcurrentWriteTransactions |
| When acquired | When the global lock is acquired (lock_state.cpp:262-266) |
| Metric | wiredTiger.concurrentTransactions.write |
The setting and metric names carry "WiredTiger", but MongoDB is what manages them. The 4.2 official documentation also says "These settings are MongoDB-specific." The setting can be changed at runtime with setParameter.
Operations that take an intent-write lock (IX), such as insert, update, and delete, each use one write ticket.
wiredTiger.concurrentTransactions.write in serverStatus reports three values.
out: the number of tickets currently in useavailable: the number of tickets lefttotalTickets: the total number of ticketsThe number of operations waiting in line is seen in globalLock.currentQueue. In the 4.2 source, this value counts operations waiting for a ticket and operations waiting for a lock together (lock_state.cpp:296-303, lock_state.cpp:341). This value alone cannot tell the two apart.
A snapshot is the list of transaction IDs that were not yet committed at the moment a transaction started. A transaction does not see values written by the transactions in this list. Read transactions also build a snapshot when they start.
The function that builds a snapshot is __wt_txn_get_snapshot (txn.c:148-248, r4.2.12). The steps are as follows.
txn_global->rwlock) of the global transaction state txn_global.If there is no write transaction in progress, steps 2-3 are skipped (source comment "For pure read-only workloads, avoid scanning."). Under a load that mixes in writes, the whole array is scanned every time.
A write transaction receives its ID without a lock (txn.i:912-918).
is_allocating flag in its own session state.is_allocating flag.While scanning the array, the thread building a snapshot waits in place when it finds a session with is_allocating raised. Below is an excerpt from src/third_party/wiredtiger/src/txn/txn.c (r4.2.12).
/* We're going to scan the table: wait for the lock. */
__wt_readlock(session, &txn_global->rwlock);
…
for (i = 0, s = txn_global->states; i < session_cnt; i++, s++) {
…
while (s != txn_state && (id = s->id) != WT_TXN_NONE && WT_TXNID_LE(prev_oldest_id, id) &&
WT_TXNID_LT(id, current_id)) {
/*
* If the transaction is still allocating its ID, then we spin here until it gets its
* valid ID.
*/
WT_READ_BARRIER();
if (!s->is_allocating) {
…
if (id == s->id) {
txn->snapshot[n++] = id;
…
break;
}
}
WT_PAUSE();
}
}
On x86, WT_PAUSE() is a single pause instruction (gcc.h:160).
#define WT_PAUSE() __asm__ volatile("pause\n" ::: "memory")
This loop has no yield (a call that gives the CPU back to the operating system), no sleep, and no cap on the number of iterations. So the waiting thread keeps holding the CPU and repeating pause until the operating system forcibly takes it off. __txn_oldest_scan, which computes the oldest transaction ID, has the same loop (txn.c:278-299).
Even within WiredTiger, the code that waits for the rwlock itself gives up the CPU in stages (mtx_rw.c:244-253).
| What it waits for | Stage 1 | Stage 2 | Stage 3 |
|---|---|---|---|
| Acquiring the rwlock | pause 1,000 times |
yield 200 times | Repeated condition variable waits (up to 10ms) |
| Another session's ID allocation | pause repeated forever |
None | None |
ID allocation takes only a few instructions. Normally, the thread building a snapshot also leaves the loop after a few pauses. When there are more runnable threads (threads in the runnable state waiting for their turn on a CPU) than cores, the following sequence can occur.
is_allocating.pause loop.is_allocating. But the threads occupying the cores are all waiting for A.When A gets a CPU again, the livelock is released. But if steps 4-5 repeat, a state with near-zero throughput can last a long time.
This sequence was written down by the author of WiredTiger issue WT-8670. The original is a hypothesis that starts with "My understanding is that".
"if an allocating thread gets scheduled out, it could take a long time for it to resume execution because all threads creating a snapshot will be spinning on that loop"
The measurement in the same issue is a throughput loss of about 4%. The comparison is against serializing snapshot creation with a mutex. The conditions were YCSB 50read50update, 32 threads, 16 CPUs. Under these conditions throughput did not collapse. No public data was found that measures whether the sequence above leads to a throughput collapse when threads far outnumber cores.
| Aspect | Deadlock | The livelock on this page |
|---|---|---|
| How it waits | Asleep, waiting for the other side's lock | Awake, waiting in a pause loop |
| CPU | Barely used | All cores used |
| Lock cycle | Present | None (what it waits for is the is_allocating flag) |
| Condition for release | One side must be forcibly cut off | Released when the displaced thread gets a CPU again |
If the CPU is at 100% but no circular lock wait is visible, suspect this livelock.
Below are general tendencies derived from the mechanism. No numeric thresholds are set.
pause is a user-space instruction that does not wait on disk.pause loop makes no system calls, so it never gives up the CPU on its own. If it were an rwlock wait, context switches should have increased because of the yields and condition variable waits.pause loop and stays runnable. Look at the r column of vmstat side by side with concurrentTransactions.write.out.currentQueue can come later than CPU saturation. That is because unfinished operations are holding tickets.When interpreting, keep two things in mind together.
perf or pstack to check whether many threads are in __wt_txn_get_snapshot.txn global lock wait metric can rise too. The thread building a snapshot spins in the pause loop while holding the txn_global->rwlock read lock. The side updating the oldest ID waits for the write lock on the same rwlock (txn.c:392-396). Do not conclude from this metric alone that rwlock contention is the cause.From MongoDB 7.0, the number of tickets is no longer fixed; it is adjusted by watching throughput. The algorithm is throughputProbing, the default of storageEngineConcurrencyAdjustmentAlgorithm. The upper limit is 128. The official documentation states its purpose as "to optimize database throughput during overload".
No official statement or measurement was found saying this feature reduces the livelock on this page. With fewer tickets, fewer threads build snapshots at the same time, so the chance of a livelock may go down. This is an inference.
As of 2026-11-05, the fix status of the pause loop was as follows.
is_allocating is still in src/txn/txn.c on the WiredTiger develop branch.Support for MongoDB 4.2 ended on 2023-04-30 (EOL).
perf top -p <mongod pid> to see the share of __wt_txn_get_snapshot.Do not conclude that ticket exhaustion is the cause. Ticket exhaustion can be a result that follows CPU saturation. Simply adding tickets can increase the number of threads spinning in the
pauseloop and make things worse (inference).
maxTimeMSandkillOpcannot stop a thread inside thepauseloop. Both work only when the operation checks for itself whether to stop, and this loop has no such check (part 4/6).
A failover may not happen. Heartbeats do not go through the engine (part 5/6).
This procedure was not verified by running it. The results below are expectations derived from the source.
Start a single MongoDB 4.2-series mongod on a VM with few cores (e.g., 2-4 vCPU, example value).
Use YCSB to apply far more concurrent writes than there are cores (e.g., 256 threads, example value).
./bin/ycsb load mongodb -s -P workloads/workloada -p mongodb.url=mongodb://<host>:27017/ycsb
./bin/ycsb run mongodb -s -P workloads/workloada -p mongodb.url=mongodb://<host>:27017/ycsb -threads 256
In another terminal, look at hotspots with sudo perf top -p $(pgrep -x mongod).
In yet another terminal, run vmstat 1 and watch the r (runnable) and cs (context switch) columns.
For comparison, lower wiredTigerConcurrentWriteTransactions to around the core count (setParameter) and apply the same load again.
The expected results are as follows.
__wt_txn_get_snapshot shows up near the top of perf top.vmstat, r is much larger than the core count and wa is close to 0. cs drops below its level at the start of the load.__wt_txn_get_snapshot and r go down.As in the WT-8670 measurement, the loss may stay at a few percent without reaching a livelock.
- Public measurements of how many times more threads than cores it takes for this loop to cause a throughput collapse
- Whether MongoDB 7.0
throughputProbingactually reduces this livelock- The corresponding code in the WiredTiger bundled with the MongoDB 7.0 and 8.0 releases. This page checked only 4.2.12 and WiredTiger
develop- The actual symbol name seen in
perf. If the compiler inlines__wt_txn_get_snapshot, it may show up under a different function name