Linux ulimit Hard Cap: When 'Too Many Open Files' Kills Your App

You've got a server that's been running fine for weeks. Then one morning, some app — maybe a web server, a database, or a custom daemon — starts spitting out EMFILE: too many open files or Cannot allocate memory because of file descriptors. The log says Resource temporarily unavailable. If you check ulimit -n as root, it shows 65536 or something high. But the process still crashes. Here's what's really going on.

Root Cause: The Hard Limit Bites Back

Linux has two kinds of limits: soft and hard. Soft limits are what a process sees by default. Hard limits are the ceiling. When the hard limit is too low, no amount of ulimit -n 999999 in a shell script helps — the kernel won't let the process go past the hard cap. Most often, this bites you with nofile (number of open files) or nproc (number of processes/threads).

I had a client last month running a Node.js app that suddenly failed at 4096 open connections. The soft limit was 65536, but the hard limit was stuck at 4096 — set by an old systemd service file. Took me an hour to find the real culprit.

The hard cap is stored in /proc/$PID/limits for each process. If the value under Max open files shows a hard limit of 1024 for your user, you can't exceed it.

Check Your Current Limits (the Right Way)

Don't just run ulimit -n — that gives you the soft limit for your current shell. To see the hard limit, run:

ulimit -Hn

That shows the hard cap for open files. For the running process, check:

cat /proc/$(pgrep -x your_app)/limits | grep 'Max open files'

That tells you exactly what the kernel enforces for that specific process.

Step-by-Step Fix

1. Raise the System-Wide Hard Limit

Edit /etc/security/limits.conf. Add these lines at the bottom (replace youruser with your actual username or * for all users):

youruser soft nofile 65536
youruser hard nofile 65536
youruser soft nproc 65536
youruser hard nproc 65536

This sets both soft and hard limits to 65536. Without the hard line, the soft limit can't go above whatever the default hard cap is.

2. If Your App Runs Under systemd (Most Modern Services)

Systemd ignores limits.conf for services it manages. You need to edit the service unit file. Run systemctl edit your-service and add:

[Service]
LimitNOFILE=65536
LimitNPROC=65536

Then reload and restart:

systemctl daemon-reload
systemctl restart your-service

Check it took with systemctl show your-service | grep LimitNOFILE.

3. If Your Service Runs Inside Docker

Docker containers have their own ulimit settings. Either pass them at runtime:

docker run --ulimit nofile=65536:65536 --ulimit nproc=65536:65536 your-image

Or set them in your docker-compose.yml:

ulimits:
  nofile:
    soft: 65536
    hard: 65536
  nproc:
    soft: 65536
    hard: 65536

4. For PAM-Aware Sessions (SSH, Login Shells)

If you're logging in via SSH and still see low limits, check that pam_limits.so is enabled. Look in /etc/pam.d/common-session (or system-auth depending on distro) for a line like:

session required pam_limits.so

If it's missing, add it. Then log out and back in.

5. Verify the Fix

Start your app again. Check that cat /proc/$PID/limits | grep 'Max open files' shows 65536 for both soft and hard. If the hard limit still shows something lower, something else is overriding it — often a systemd service file, a container runtime, or a PAM configuration.

What to Check If It Still Fails

• Kernel parameter: The system-wide max file descriptors is controlled by fs.file-max. Run sysctl fs.file-max. It should be at least 65536. If not, add fs.file-max=65536 to /etc/sysctl.conf and run sysctl -p.
• App-specific config: Some apps (like Nginx, PostgreSQL, or Elasticsearch) have their own file descriptor limits in their config files. For example, Nginx has worker_rlimit_nofile.
• Docker daemon: The Docker daemon itself has its own ulimit. If you're running Docker containers, check dockerd --default-ulimit or add --default-ulimit nofile=65536:65536 to the daemon startup.
• Inode exhaustion: If you're using all available inodes on a filesystem, raising ulimit won't help. Run df -i to check inode usage.
• Watchdog or monitoring: Some cloud providers (like AWS or Azure) have their own limits that override Linux settings. Check your account limits for EC2 or container instances.

If none of that works, post the output of cat /proc/$PID/limits and cat /etc/security/limits.conf — I've seen cases where a stray old /etc/security/limits.d/*.conf file was silently overriding everything.