HAProxy SSL Termination Error Fix: Backend Handshake Failure

When This Error Hits

You're running HAProxy 2.4 or later on RHEL 8 or Ubuntu 20.04. Your frontend terminates SSL fine. But the backend servers — Apache or Nginx — start returning 503 errors. The HAProxy logs show SSL handshake failure. This usually happens right after you renew the SSL cert on the backend server, or change the CA bundle. Maybe you pushed a new cert via Ansible and forgot to update the chain. The load balancer suddenly can't trust the backend.

Why It Breaks

The culprit here is almost always a mismatch between the backend's SSL certificate and what HAProxy expects. HAProxy by default validates the backend server's certificate when you set ssl verify required. If the cert's issuer isn't in the CA file HAProxy uses, or the chain is incomplete, the handshake dies. Another common cause: the backend server's IP changed but the cert doesn't match the new IP or hostname. I've seen this a dozen times after someone forgot to update server-template configs.

Don't bother checking the frontend SSL settings first. That's not the problem. The error is at the backend layer. Start with the backend server's cert.

The Fix: 4 Steps

Check the backend server's cert chain
Run this on the backend server to see if the cert chain is complete:
```
openssl s_client -connect backend-server-ip:443 -showcerts
```
Look for a full chain from server cert to root CA. If you see just one cert, the chain is broken. You need to concatenate the server cert, intermediate certs, and root CA into one PEM file.
Update the CA bundle HAProxy uses
On the HAProxy server, the default CA bundle path is often /etc/ssl/certs/ca-bundle.crt or /etc/haproxy/ca.crt. Check which one your config references:
```
grep -R 'ca-file' /etc/haproxy/
```
If the backend server's issuer changed, add the new CA to that file. Or point to a custom bundle that includes it.
Verify the backend server's hostname or IP matches the cert
Use openssl again to check the Subject Alternative Names (SANs):
```
openssl x509 -in /path/to/server.crt -text -noout | grep -A1 'Subject Alternative Name'
```
If the backend server's DNS or IP in HAProxy config doesn't match what's in the cert, update the config or the cert. You can't bypass this if you have ssl verify required.
Restart HAProxy gracefully
After fixing the cert or CA bundle, reload HAProxy without dropping connections:
```
haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid)
```
Check the logs immediately: journalctl -u haproxy -n 20. You should see Server is UP lines.

What to Check If It Still Fails

If the error persists after the steps above, here's what I've seen go wrong:

Cipher mismatch — The backend server might use a cipher HAProxy doesn't support. On older backend servers (e.g., Apache 2.4 with OpenSSL 1.0.2), you might need to relax the cipher list in HAProxy with ssl-default-bind-ciphers.
OCSP stapling — If HAProxy is set to enforce OCSP stapling on the backend (rare but possible), the backend server might not support it. Disable it in the backend config with verify none temporarily to test.
Firewall or SNAT — The backend server might see the HAProxy IP as the client IP. If your firewall drops packets from HAProxy to the backend on port 443, you'll get timeouts that look like SSL errors. Check tcpdump on the backend for incoming SYN packets.

One more thing: if you're using self-signed certs on the backend, you must add them to the HAProxy CA bundle. No exceptions. That trips up a lot of people.

Real example: last month a client had a dev environment where the backend cert expired overnight. HAProxy rejected every connection. They spent an hour checking HAProxy config before I told them to check the backend server's cert. Renewed it, fixed the chain, reloaded HAProxy — problem gone.