Object Storage Consistency Check Failed: Quick Fixes

1. Network Time Sync Drift

This is the number one cause. I've seen it in Ceph, MinIO, and even on-prem object stores. When your storage nodes have clock drift—even by a few seconds—they start arguing about which version of an object is the right one. The consistency check fails because timestamps don't match.

How to spot it: Check /var/log/messages or your storage daemon logs for entries like "clock skew" or "time mismatch". You'll also see the error pop up after a reboot or after a power outage when NTP hasn't fully synced yet.

The fix:

1. Check current time on all nodes: Run date on each node and compare. If they're more than half a second off, that's your problem.
2. Force NTP sync immediately: Run sudo systemctl restart ntp (or chronyd if you're on RHEL/CentOS 7+). Then wait 30 seconds and run ntpq -p to confirm sync.
3. If you're in a VM: Disable VMware time sync or Hyper-V time sync integration services. They often fight with NTP. Set tools.syncTime = "FALSE" in your VMX file for VMware.
4. Permanent fix: Add server pool.ntp.org iburst to /etc/ntp.conf and make sure NTP starts on boot: sudo systemctl enable ntp.

After fixing time, run the consistency check again. 8 times out of 10, it passes immediately.

2. Corrupted Object or Stale Checksum

If time is fine, the next culprit is a corrupted object or a stale checksum file. This happens when a write operation gets interrupted—think sudden power loss, a full disk, or a network timeout during a large file upload. The metadata says the object is one size or hash, but the actual data on disk is different.

How to spot it: Look at the storage node logs for checksum mismatch or object size mismatch. In MinIO, you'll see md5 mismatch errors. In Ceph, scrub error messages point to specific OSDs and PGs.

The fix:

1. Identify the bad object: Use your storage system's tool to list inconsistent objects. For Ceph: ceph pg scrub {pg-id}, then ceph pg repair {pg-id}. For MinIO: use mc admin heal myminio.
2. Force a recheck: Don't assume it fixes itself. Run a deep scrub if you can. On Ceph: ceph pg deep-scrub {pg-id}.
3. If repair fails: You may need to delete and re-upload the object. That's rare—like 1 in 50 cases—but it happens. Check if the object is critical, then copy it from a backup or regenerate it.

A lot of admins skip step 2. Don't. A deep scrub actually reads the data and compares it to the checksum. That's what catches the corruption.

3. Cluster Rebalance Stuck or Hung

This one's sneaky. The consistency check fails because the cluster is mid-rebalance and can't settle on which node owns the object. I see this most often when you add a new node or replace a failed disk. The rebalance process gets stuck on a specific bucket or object.

How to spot it: Check the cluster status. In Ceph, ceph -s shows rebalancing or degraded state. In MinIO, mc admin info shows incomplete uploads or pending rebalance. Also, the consistency check error might only affect one bucket or a set of objects.

The fix:

1. Let it finish or force it: If the rebalance is slow, increase the rebalance speed. For Ceph: ceph tell osd.* injectargs '--osd-max-backfills 4'. For MinIO: set MINIO_REBALANCE_THROTTLE=0 (removes throttling).
2. If it's actually stuck: Restart the stuck node. For Ceph OSD: sudo systemctl restart ceph-osd@{id}. For MinIO: sudo systemctl restart minio on the node that's lagging.
3. Worst case: Remove and re-add the node. I've had to do this twice in 14 years. Export the bucket data first with mc mirror or rados tools, then remove the node, let the cluster heal, then add it back.

Pro tip: before restarting anything, check disk space on each node. A full disk will stop rebalance cold.

Quick-Reference Summary Table

That's it. Fix time first, then check for corruption, then deal with rebalance. You'll clear this error in under 30 minutes.