Fix PostgreSQL Replication Slot WAL Retention

When This Error Hits

You're running PostgreSQL 13 or newer with streaming replication. A standby server went offline for a few hours — maybe a network blip, maybe a reboot. Suddenly your primary server's disk fills up fast. The logs show FATAL: could not access status of transaction or just ERROR: out of disk space. Check pg_stat_replication and you see a slot with active = false but sent_lsn far behind pg_current_wal_lsn(). That's the culprit. The primary is holding WAL files for that dead slot because the slot claims a standby still needs them.

Root Cause

Replication slots are great — they guarantee a standby won't miss WAL segments during reconnection. But they're also a trap. When a slot goes inactive (standby disconnects), PostgreSQL keeps every WAL file generated after the slot's restart_lsn. If your database writes a lot (say 1-2 GB/hour), a 4-hour outage means 4-8 GB of WAL pinned on disk. No other process can delete those files. The pg_wal directory swells until you get a full disk or hit max_wal_size (default 1 GB). Worst case: the disk fills, the primary crashes, and recovery is a nightmare.

The real issue? Nobody monitors replication slots. They're invisible until the disk alarm goes off. And the default settings (wal_keep_segments or wal_keep_size) don't override slot retention — slots always take priority.

The Fix: Clear the Blocked Slot

Here's the exact sequence. Run these commands on the primary as a superuser (typically postgres).

Identify the offending slot
```
SELECT slot_name, active, restart_lsn, wal_status FROM pg_replication_slots;
```
Look for active = false and wal_status = 'lost' or wal_status = 'reserved'. The restart_lsn will be far behind the current WAL position. Note the slot_name.
Verify it's safe to drop

Can't recover that standby? Or the standby already got rebuilt with a new slot? Then drop it. If the standby might come back, you can't drop the slot — you'll need to reconnect it first. But in practice, if the slot's inactive for hours and disk is critical, drop it. The standby can reinitialize with a fresh slot.
Drop the slot
```
SELECT pg_drop_replication_slot('your_slot_name_here');
```
This deletes the slot and signals PostgreSQL to release all held WAL files. The cleanup happens lazily — it won't remove WAL files still needed by other slots or by wal_keep_size. But within a few seconds, you'll see pg_wal start shrinking. Run a checkpoint if you want it faster:
```
CHECKPOINT;
```
Free the space

PostgreSQL won't delete WAL files that are still ahead of wal_keep_size. If your wal_keep_size is 1 GB, you might have 1 GB stranded. That's fine — it'll recycle naturally as new WAL is written. For an immediate fix, you can lower wal_keep_size temporarily and restart, but that's heavy-handed. Better to just monitor pg_wal size drop over 5-10 minutes.

If Disk Still Full

Your disk might have filled completely — PostgreSQL stopped writing WAL, which means the database is down. In that case:

Kill the PostgreSQL process if it's stuck (pg_ctl stop -m immediate).
Manually delete old WAL files from pg_wal in the data directory. Never delete the current WAL segment (the one with the highest number). Keep at least 2-3 segments before the current one. This is ugly but it buys you a restart.
After restart, drop the slot as above.

If you can't even start PostgreSQL because of a missing WAL file (you deleted too many), restore from backup. That's why I say keep a few segments.

Preventing This Long-Term

You've got to automate slot monitoring. I add this to our monitoring every time:

SELECT slot_name, active, pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS retained
FROM pg_replication_slots
WHERE NOT active;

Alert if any row returns a value. Or set a cron job that drops slots inactive for more than, say, 6 hours — but only if you're fine losing that standby's catch-up ability. Better to page a human.

Also check wal_keep_size. On high-write databases, set it to 2-4 GB. That way, even without slots, a disconnected standby has a fighting chance. And always set max_slot_wal_keep_size (PostgreSQL 13+) to cap how much WAL a single slot can hold. Something like 4 GB per slot:

max_slot_wal_keep_size = 4096
wal_keep_size = 2048

This limits the damage. If a slot goes rogue, it can't eat more than 4 GB before the system marks it as lost and stops reserving WAL. That gives your standby a limited window, but it beats a full disk.

What About the Standby?

After dropping the slot, you need to reinitialize the standby. Use pg_basebackup with --slot=your_new_slot_name. Create a new slot first on the primary:

SELECT pg_create_physical_replication_slot('standby1_slot');

Then on the standby, run:

pg_basebackup -h primary_ip -D /var/lib/postgresql/data -S standby1_slot -X stream -P -v

That's it. The standby reconnects with a fresh slot, and you've got replication running clean again.

One last thing: never set max_replication_slots higher than you need. Slots are cheap metadata, but each one is a potential disk bomb. Keep them to a minimum — one per standby is plenty.