Oracle RAC Node Eviction: Reboot Loop Fix

Quick Answer

Check /var/log/messages and $GRID_HOME/log/<hostname>/cssd/ocssd.log for network heartbeat loss. Also verify disk timeouts on shared storage. Fix the root cause, then reboot the node.

Why This Happens

Oracle RAC node eviction is the cluster's safety mechanism. When a node stops responding to heartbeats (CSSD daemon), the cluster kills it to protect data integrity. The culprit here is almost always one of two things: network issues on the private interconnect or disk I/O hangs on the shared storage. I've seen this on RHEL 7 and Oracle Linux 7 with Grid Infrastructure 12c and 19c. The node goes into a reboot loop because the CSSD daemon triggers a reboot on eviction, and the root cause hasn't been fixed yet.

Don't bother checking all logs at once. Start with the two most important: ocssd.log for CSSD messages, and alert.log for the clusterware stack. A common trigger is a switch failover or a disk path loss on a SAN. Also watch for high CPU or memory pressure that causes the node to miss heartbeats.

Fix Steps

1. Check CSSD logs first
   Run this on the evicted node after it comes back up:

   tail -100 $GRID_HOME/log/$(hostname)/cssd/ocssd.log | grep -i evict

   Look for Member killed or Node evicted messages. That tells you if it's a heartbeat loss.
2. Check network heartbeat
   Verify the private interconnect is working. Ping the other nodes using the private IPs. Also run:

   netstat -i | grep <interface_name>

   Look for errors, collisions, or drops. If you see errors, that's your issue. Increase MTU to 9000 on all private interconnects.
3. Check disk timeouts
   Shared storage I/O hangs cause the node to miss heartbeats. Check disk timeouts:

   cat /sys/block/sd*/device/timeout

   Should be 60 or higher for Oracle RAC. Also check multipath configurations with multipath -ll. Look for path failures.
4. Check OS logs for resource pressure
   Memory or CPU spikes can cause missed heartbeats. Check /var/log/messages for OOM killer events or high load averages. If you see that, add memory or CPU to the VM or physical node.
5. Test clusterware restart
   After fixing the root cause, restart clusterware on the node:

   crsctl stop crs
   crsctl start crs

   Then check the cluster status with crsctl stat res -t. The node should rejoin without eviction.

Alternative Fixes If Main One Fails

Sometimes the root cause isn't obvious. If the node still gets evicted:

• Check for network bonding issues — bond0 misconfiguration can cause intermittent drops. Try splitting private interconnect across two different switches with VLANs.
• Check ASM disk group timeouts — If the disk group has disk_repair_time set too low, the node may get evicted during short path failures. Increase it to 3.6 hours.
• Disable firewall on private interconnect — iptables or firewalld can drop ICMP packets. Disable them on the private subnet.
• Check NTP sync — Time drift over 500ms between nodes can cause eviction. Use ntpq -p to check. Fix with chronyd.

If none of that works, try a full clusterware reconfiguration — but that's rare. Usually one of the above fixes it.

Prevention Tip

Set up monitoring for the private interconnect latency and shared storage I/O latency. Use tcpdump on the private interface for 24 hours and look for retransmissions. Also configure misscount in CSSD to 60 seconds instead of the default 30 — gives the node a bit more time to recover from short hangs. To change it:

crsctl set css misscount 60

Also set disk timeouts to 120 seconds on all shared storage paths. I've never seen a node evicted with these two changes.