MongoDB Replica Set Election Hangs on Stale Oplog Entry

Quick answer

Run rs.stepDown() on the current primary. If that hangs, check the oplog on each secondary with rs.printSecondaryReplicationInfo() and restart the secondary with the highest lag. Then force re-election with rs.reconfig() using force: true.

Why this happens

You're running a MongoDB replica set, probably a 3-node or 5-node cluster, and suddenly you can't elect a new primary. The current primary is stuck, or all nodes are secondaries with no primary. You check logs and see "waiting for primary heartbeat" or "election failed, no candidate." The real culprit is often a stale oplog entry on one or more secondaries.

The oplog (operations log) is a capped collection that records every write. Each secondary replays these entries to stay in sync. If a secondary has an oplog entry that's older than the other members' oldest entry, that secondary is considered "stale." The replica set will not hold an election until all members agree on a consistent oplog point. That's the safety net. But when a secondary's oplog has a corrupt or orphaned entry — maybe from a failed resync or a network split — the election can hang indefinitely.

I've seen this happen after a power outage where one secondary rebooted but its oplog got truncated wrong. Or after a failed rs.initiate() on a node that was already part of a set. The symptom is the same: the set cannot agree on a primary.

Step-by-step fix

1. Check the current state of all nodes. Connect to any node with mongosh and run rs.status(). Look at each member's stateStr. If you see a primary, note its name. If all are SECONDARY or UNKNOWN, that's the stuck situation.
2. Identify the worst lagging secondary. Run rs.printSecondaryReplicationInfo() from any node. Look at the syncedTo values. The node with the oldest syncedTo timestamp is the one causing the problem. Write down its hostname.
3. Access the stale secondary directly. Connect to that host with mongosh and check if it's still accepting writes. If it's a secondary, it shouldn't, but sometimes a misconfigured node thinks it's still the primary. Run rs.isMaster() to confirm its role.
4. Kill the replication worker on the stale node. On the stale secondary, run db.adminCommand({replSetAbortPrimaryCatchUp: 1}). This forces it to stop trying to catch up. Then run rs.syncFrom() to point it to the current primary (or the healthiest secondary). Use rs.syncFrom("primary-hostname:27017").
5. Force a re-election from the current primary. If there's still a primary, connect to it and run rs.stepDown(300, 30). The first number is seconds to wait for a new primary to catch up; the second is how long to remain ineligible. If the primary is unresponsive, skip to step 6.
6. Force reconfig with force: true. If no primary exists, connect to any secondary and run:

   var cfg = rs.conf()
   cfg.members[0].votes = 1
   cfg.members[1].votes = 1
   cfg.members[2].votes = 1
   rs.reconfig(cfg, {force: true})

   This forces a new election using the current member's view of the set. After a few seconds, run rs.status() again — you should see a new primary.
7. Resync the stale node if it still lags. If the node you fixed in step 4 is still behind, you'll need to resync it entirely. Stop the mongod process, delete the data directory (make a backup first if you care), restart the process, and let it do a full initial sync. This can take hours on large datasets. That's fine — it's better than a stuck election.

Alternative fixes that sometimes work

• Restart all secondaries at once. Shut down all secondaries, wait 30 seconds, then start them one by one. This can clear transient oplog issues. But it won't fix corruption.
• Bump protocol version. If you're on an older set, run rs.reconfig({protocolVersion: 1, writeConcernMajorityJournalDefault: true}, {force: true}). PV1 handles elections better.
• Use rs.initiate() with the correct config. If the config is completely broken, you can start fresh on one node, then add others. But this is a last resort — it blows away existing replication state.

Prevention tip

Set a small oplog size (like 1GB) and monitor replLagSeconds on all nodes. If any secondary's lag exceeds 10 minutes, investigate immediately. Also, never run rs.initiate() on a node that's already part of a replica set — that's how stale entries get created. And always use writeConcern: majority for critical writes to force consistent replication.