Oracle RAC Node Eviction: CRS-4535 Insufficient Memory

You're running Oracle RAC on Linux — maybe Oracle 12c or 19c — and suddenly one node gets evicted. The alert log shows CRS-4535: Insufficient memory. Other times you see CRS-4546: Node eviction due to memory pressure. This always happens during peak load, like when a batch job kicks off or a big query runs. The node just drops off the cluster, and you're scratching your head.

Why It Happens

The culprit here is almost always Linux running out of physical memory. Oracle Clusterware uses a memory reservation system — it expects a certain amount of RAM to be available. When the OS starts swapping heavily because memory is tight, the clusterware heartbeat misses a beat. The cluster thinks the node is dead, and poof — eviction.

Don't bother blaming Oracle. It's a kernel misconfiguration or over-committed memory 9 times out of 10. Specifically, hugepages are set too low, or the vm.nr_hugepages parameter isn't matching what Oracle expects. Also, if you're running Oracle on a VM, the hypervisor might be stealing memory via ballooning.

The Fix

Here's the step-by-step. Run each command as root or oracle user with sudo.

Step 1: Check Current Memory and Hugepages

cat /proc/meminfo | grep -E 'MemTotal|HugePages_Total|HugePages_Free|HugePages_Rsvd'

You want HugePages_Free close to HugePages_Total. If HugePages_Rsvd is big, the database is claiming but not using pages. If HugePages_Total is zero or too small, that's your problem.

Step 2: Calculate Correct Hugepages

Oracle database uses hugepages for its SGA. The formula is simple: take the total SGA size (in GB) and divide by the hugepage size (default 2MB on x86_64). Add 10% headroom. So if SGA is 16 GB:

16 GB * 1024 MB/GB / 2 MB = 8192 hugepages
Add 10%: 8192 * 1.1 = 9011

Set that in /etc/sysctl.conf:

vm.nr_hugepages = 9011

Then apply:

sysctl -p

Step 3: Disable Transparent Hugepages

Oracle hates transparent hugepages. Turn them off. Edit /etc/default/grub and add to GRUB_CMDLINE_LINUX:

transparent_hugepage=never

Then regenerate grub and reboot:

grub2-mkconfig -o /boot/grub2/grub.cfg
shutdown -r now

Step 4: Check Swap

Swap should be at least equal to RAM for RAC nodes. Run:

swapon --show

If swap is too small, add a swap file. On a 64GB RAM node:

dd if=/dev/zero of=/swapfile bs=1G count=64
mkswap /swapfile
swapon /swapfile
echo '/swapfile none swap sw 0 0' >> /etc/fstab

Step 5: Adjust Memory Limits in Grid Infrastructure

The OCR might have wrong memory limits. Check with:

crsctl stat res -p -init | grep -E 'MEMORY|MEMLIMIT'

If MEMLIMIT is set lower than real RAM, increase it. For a node with 64GB:

crsctl modify resource 'ora.cluster_interconnect.haip' -attr 'MEMLIMIT=64G'

If It Still Fails

Check the OS logs. Run dmesg -T | grep -i oom — if you see Out-Of-Memory killer messages, then the OS is killing processes due to memory pressure. That means you need to reduce memory overcommit. Add this to /etc/sysctl.conf:

vm.overcommit_memory = 2
vm.overcommit_ratio = 50

Also, if this is a VM, check with your hypervisor team that memory ballooning is disabled. On VMware, set sched.mem.pshare.enable = FALSE in the VMX file.

One more thing — check if you're running a memory-hungry process like systemd-journald or rsyslog that could be eating RAM. Kill it if not needed.

If none of this works, you might have a hardware issue — bad RAM DIMMs. Run memtest86+ overnight.