Cluster Resource Scheduler Stuck After Node Reboot

Quick answer

Run pcs status and check for Online: [...] missing your rebooted node. If quorum is lost, force reestablish with pcs cluster quorum unblock (Pacemaker 2+) or restart corosync on the affected node.

Here's the thing: after a node comes back from reboot, the cluster stack (corosync + pacemaker) starts fresh. The scheduler — that's pacemaker's brain — waits for quorum before making decisions. If the node can't rejoin the quorum ring, the scheduler sits idle. Your resources don't move, services stay down. People panic and restart the entire cluster. You don't need to.

What's actually happening here is the corosync ring protocol gets confused. The rebooted node might have a different IP or hostname, or the cluster config on disk is stale. The scheduler sees "no quorum" and just stops scheduling. It's a safety feature, not a bug.

Fix steps

1. Check cluster status
   pcs status — look for the rebooted node under "Online:". If it's missing, you have a connectivity or quorum problem.
2. Verify node is reachable
   ping <node-ip> from another cluster node. If ping fails, check network and hostname resolution. Common trap: the node's interface came up with a different MAC or IP after reboot due to DHCP.
3. Force quorum if needed
   On the rebooted node, run:
   pcs cluster quorum unblock
   This tells pacemaker to start scheduling even if quorum is missing (set no-quorum-policy to ignore). Do this only if you're sure the node is healthy.
4. Reload corosync config
   corosync-cfgtool -R on the rebooted node — reloads ring config without restarting the service. This fixes most IP/interface mismatches.
5. Restart pacemaker (not corosync)
   systemctl restart pacemaker on the rebooted node. Corosync stays up, so the ring doesn't break. Pacemaker will rejoin and the scheduler resumes.
6. Verify resources
   pcs resource cleanup — clears transient resource failures that happened while the node was offline. Then pcs status again.

If the main fix fails

Sometimes the node just won't rejoin. Try these in order:

• Check corosync logs — journalctl -u corosync -n 50 on the rebooted node. Look for "join failed" or "token lost". That tells you the real cause: wrong mcast port, firewall blocking, or mismatched cluster name.
• Re-add the node manually
  On another cluster node: pcs cluster node add <hostname> — this forces the cluster to accept it. Yes, even if it was already there. Sometimes the node's fingerprint changes after reboot.
• Restart corosync on all nodes
  Only as last resort. On each node in sequence: systemctl restart corosync. Wait for the node to come back online before moving to the next. This rebuilds the ring from scratch.

Prevention tip

Set no-quorum-policy=ignore in your cluster config (only if you have <3 nodes or use shared storage like DRBD). This way, a single node reboot won't lock the scheduler. Also, pin static IPs or use predictable DHCP for cluster interfaces — changing IPs after reboot is the #1 cause of this issue.