This is part 6 (the last part) of the MongoDB sharded cluster write path series.
Question this page covers: If you route reads to secondaries with
secondaryPreferred, are queries safe while the primary is stuck? To give the conclusion first: they are not. In a sharded cluster, a secondary gets the ownership list (routing metadata) that records "which data belongs to which shard" through a write on its own primary. So a secondary restarted on a shard whose primary cannot write cannot get the ownership list and refuses reads withStaleConfig.
Reference version: MongoDB 4.2 (source tagr4.2.12). The latest versions were not checked.
A sharded cluster splits a collection into chunks, which are ranges of the shard key, and places them on several shards. The balancer moves chunks to even out the amount of data between shards. The order of a move is "copy to the new shard → hand over ownership → delete from the old shard".
Until the final delete finishes, the old shard still holds documents whose ownership has already moved. Such documents are called orphan documents. For example, say the chunk containing document X of the store_inventory collection moved from shardA to shardB. The state at each point is as follows.
| Point | shardA | shardB | Owner |
|---|---|---|---|
| Before the move | X | None | shardA |
| After copy, before delete | X (orphan) | X | shardB |
| After delete | None | X | shardB |
If a query that asks every shard arrives at the middle point, both shards have X. If each shard returns it without checking ownership, X appears twice in the response. So before responding, a shard must check "is this document mine?" and filter out orphan documents.
To filter, a node has to hold the ownership list. The ownership list is information that records "which chunk belongs to which shard". A node keeps the ownership list and its version in memory. They map to MongoDB terms like this.
From here on, this page uses only the name ownership list.
When mongos sends a read to a shard, it attaches the shard version it knows. The receiving node compares that version with its own (collection_sharding_state.cpp:201-278). The 3.6 release notes say that secondaries, not only the primary, maintain chunk metadata so that secondary reads do not return orphan documents.
"Starting in 3.6, all members of a shard replica set, not just the primary, maintain the metadata regarding chunk metadata."
The check has three outcomes.
| Node state | What the node does |
|---|---|
| Version equals mongos's | Reads, filters out orphan documents, and responds |
| Version differs | Refuses. The stale side fetches the latest ownership list |
| No ownership list (UNKNOWN) | Refuses and goes to fetch the ownership list |
The error used when refusing is StaleConfig. Its code is 13388 (error_codes.err:319). The source calls the state with no ownership list UNKNOWN and defines it like this (collection_sharding_runtime.h:85-86).
"all attempts to check for shard version match will fail with StaleConfig errors in order to trigger an update."
In other words, StaleConfig is a signal that says "my ownership list is stale or missing, so let me fetch a new one and try again". Normally the refresh finishes right away and mongos retries, so the application never sees this error.
The primary and secondaries fetch the ownership list differently. The primary gets it directly from the config server. A secondary does not ask the config server; it goes through its own primary. The order in the 4.2 source is as follows (shard_server_catalog_cache_loader.cpp:268-290, 428-431, 585-591).
_flushRoutingTableCacheUpdates command to its own primary. The source sends this command under its old name forceRoutingTableRefresh. The old name is an alias kept for 3.6.0 compatibility (flush_routing_table_cache_updates_command.cpp:60-63). The time limit is 30 seconds (Seconds{30}).config.cache.* collections.waitUntilOpTimeForRead). When that write arrives through replication, it reads the local config.cache.* and fills the ownership list in memory.For one secondary read to refresh the ownership list, one write on the primary and its replication must finish first. That is because step 2 is a write.
The source comments do not say why a secondary does not ask the config server directly. This page reads this structure as a design that keeps replicated data and the ownership list at the same point in time. If a secondary learned about a chunk move through the ownership list before it had replicated that move, the documents and the ownership list would disagree. Receiving both through the same replication stream avoids that disagreement. This interpretation is an inference drawn from the source.
The ownership list lives in memory. When you restart mongod, the in-memory ownership list starts empty (metadata_manager.cpp:246-252). config.cache.* on disk remains, but the in-memory ownership list is filled at the first refresh. So a restarted node refuses the first read for each collection with StaleConfig and refreshes the ownership list.
Now layer on a situation where the primary cannot write. A primary whose process is alive but whose writes have stalled was covered in part 5 of this series. Comparing that shard's secondaries in three cases gives the following.
| Secondary | Ownership list in memory | Read result |
|---|---|---|
| Not restarted, collection it was already reading | Present | Keeps reading |
| Not restarted, collection read for the first time | Absent | Refused because the refresh fails |
| Restarted | Absent for every collection | Refused on the first read of each collection because the refresh fails |
The restarted secondary sends a command to the primary to refresh. But the primary cannot write to config.cache.*. The secondary fails the refresh, either by hitting the 30-second limit or while waiting for replication. The ownership list stays UNKNOWN, so the next read takes the same path. Until a healthy primary is up, this node cannot serve a single read for that shard.
A readPreference such as secondaryPreferred is a rule for choosing which node to send a read to. It does not decide where that node gets its ownership list. Even if you route reads to secondaries, the ownership list refresh is still tied to writes on the primary.
StaleConfig is an error exchanged between mongod and mongos. When mongos receives this error, it waits for the refresh and retries. So the application may not see StaleConfig as is. While mongos retries, the application's request waits for a response. On the application side it may show up only as a connection timeout or a socket timeout.
Signals to look for in this situation are as follows.
forceRoutingTableRefresh or _flushRoutingTableCacheUpdates failure.StaleConfig (13388) repeats in the mongos log.this shard no longer contains chunks for <ns> (collection_sharding_state.cpp, inference based on the source).If you read with readConcern: "available", the node skips the shard version check. The invariant at collection_sharding_state.cpp:210-212 confirms this premise. The node responds to reads even without an ownership list.
The price is orphan documents. The 3.6 release notes say that a secondary's ownership list is what prevents orphan documents, and that with "available" this protection is gone. At the middle point in the table above, document X can appear twice.
// mongos에 연결한 셸 예시
db.store_inventory.find({ "_id.productId": 1001 })
.readPref("secondaryPreferred")
.readConcern("available")
Whether to use it depends on how accurate the query result must be. You can choose it for a screen where duplicates or documents that have already moved are acceptable (e.g., a rough inventory display). Do not use it for queries where counts must be exact, such as payment or settlement.
readConcern: "available" only for queries that can tolerate orphan documents.Pitfall: Restarting a secondary is a common "let's just try reviving it" action. In this situation, it turns a secondary that was serving reads into one that cannot. A restart empties the in-memory ownership list. Filling that ownership list requires a write on the stuck primary.
This procedure was not verified by running it. The procedure and results below are expectations based on the source.
You need MongoDB 4.2 binaries. The operator starts, locally, 1 config server replica set, 2 shards (each a 3-node replica set), and 1 mongos.
store_inventory collection on an _id hashed shard key and insert documents.sh.enableSharding("demo")
sh.shardCollection("demo.store_inventory", { _id: "hashed" })
for (let p = 1; p <= 100; p++)
for (let s = 1; s <= 20; s++)
db.getSiblingDB("demo").store_inventory.insertOne({ _id: { productId: p, storeId: s }, qty: 10 })
{ node: "a1" }, { node: "a2" }). In the same reconfig, set the priority of both secondaries to 0. This setting keeps an election from happening when the primary is stopped.find(...).explain().db.store_inventory.find({ _id: { productId: 1, storeId: 1 } })
.readPref("secondary", [{ node: "a1" }]).maxTimeMS(60000)
kill -STOP <pid> to the PID of the shardA primary.kill -CONT <pid> and send the a1 query again.The expected results are as follows.
| Step | a1 (restarted) | a2 (not restarted) |
|---|---|---|
| Step 7 | Responds slowly, then fails. A refresh failure in the a1 log, StaleConfig in the mongos log |
Responds right away |
| Step 8 | Responds | Responds |
Limits of this reproduction:
SIGSTOPstops the whole process. Heartbeat responses also stop, so the replica set sees the primary as "unreachable". A primary whose engine is in a livelock, by contrast, still replies to heartbeats and only fails to write (part 5). priority 0 prevents the election, but from the step where a1 looks for the primary onward, it differs from the real case. So the error in step 7 may come out as a "primary not found" kind of error rather than the 30-second limit. Use it to see the difference that only the restarted node gets blocked.
- Whether the error mongos returns to the application in this situation is
StaleConfigor a timeout kind of error was not confirmed. "It may show up as a timeout in the application" is a general statement.- The reason a secondary does not ask the config server directly is not written in the source. The explanation above is an inference.
- Whether an unsharded collection takes the same path right after a restart was not checked separately.
- Whether the secondary refresh path changed in versions after 4.2 was not checked.
- "Reproduce it yourself" was not run.