Fix Container Pod Network Namespace Failure

What's Happening Here

Your pod is stuck in ContainerCreating. You check the logs and see something like network namespace for pod not found or CNI network not ready. This means the container runtime (Docker or containerd) can't find the network namespace file for your pod. It's almost always a race condition or stale namespace file. I've seen this on Kubernetes 1.24+ with containerd, and on older Docker setups too.

Let's fix it. Start with the quick check, then move to the moderate fix. If neither works, the advanced section will sort it out. Don't jump ahead — try each one in order.

Quick Fix (30 seconds)

This is the first thing you check. The culprit here is almost always a leftover network namespace file from a crashed pod or node reboot.

Step 1: Check if the namespace file exists

SSH into the node where the pod is stuck. Run:

ls /var/run/netns/

If that directory is empty or missing, the issue is that the CNI plugin didn't create the namespace. But if you see a file with the pod UID, that's good news — means the namespace exists but something else is wrong.

Step 2: Restart the kubelet

If the directory is missing or you see nothing useful, restart kubelet. It forces a re-sync of network state.

systemctl restart kubelet

Wait 30 seconds. Check if the pod moves to Running. If yes, you're done. If not, move to the moderate fix. This works about 40% of the time — mostly after node reboots when kubelet doesn't clean up old namespaces.

Moderate Fix (5 minutes)

This handles the common case where old network namespaces are stuck or the CNI plugin is misconfigured.

Step 1: Kill stale network namespaces

List all namespaces:

ip netns list

You might see entries like cni-xxxxxxxx or ns-xxxxxxxx. These are leftover. Delete them:

ip netns delete cni-xxxxxxxx

Or if the namespace file is in /var/run/netns/ but the pod is dead, just remove the file:

rm -f /var/run/netns/cni-*

Then restart kubelet again:

systemctl restart kubelet

Step 2: Verify CNI plugin config

If the pod still fails, check the CNI config. On most setups, it's in /etc/cni/net.d/. Make sure the config file matches your plugin. Example for Flannel:

cat /etc/cni/net.d/10-flannel.conflist

Look for the type field. It should say flannel or bridge. If it's empty or wrong, reinstall the plugin. For Calico:

kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml

Wait 2 minutes. Then delete the stuck pod:

kubectl delete pod <pod-name> -n <namespace>

The new pod should create a fresh namespace. If it still fails, you need the advanced fix.

Advanced Fix (15+ minutes)

You're here because the above didn't work. This usually means a deeper issue with the container runtime or kernel networking. I've seen this on nodes with custom kernels or after a kernel upgrade.

Step 1: Check containerd or Docker runtime

First, confirm which runtime you're using:

crictl version

If you see containerd, check its pods:

crictl pods

Find your pod's ID. Then check its network namespace:

crictl inspect <pod-id> | grep namespace

If the namespace path is missing or points to a file that doesn't exist, you need to restart containerd:

systemctl restart containerd

Then restart kubelet. This reloads all pod sandboxes. It's disruptive — expect all pods on that node to restart. But it fixes the namespace issue 90% of the time.

Step 2: Check for overlay filesystem issues

On some kernel versions (especially 5.4.x on Ubuntu 20.04), overlay2 can cause namespace mount failures. Check dmesg:

dmesg | grep -i namespace

If you see failed to create new netns or operation not supported, the issue might be the kernel. Try this workaround — disable the bridge-nf-call-iptables sysctl:

sysctl -w net.bridge.bridge-nf-call-iptables=0

Then restart kubelet. If that fixes it, make it permanent by adding to /etc/sysctl.conf:

net.bridge.bridge-nf-call-iptables=0

Step 3: Rebuild the network namespace manually

If all else fails, you can manually create the missing namespace. This is a last resort. Get the pod UID from kubectl:

kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.metadata.uid}'

Then on the node, create the namespace file:

touch /var/run/netns/cni-<pod-uid>

Then restart kubelet. This forces the CNI plugin to attach to that file. It's hacky, but it works in a pinch. After the pod runs, delete the manual file.

When to Give Up and Rebuild

If you've done all three and the pod still won't start, the node is likely broken. Drain the node and reboot:

kubectl drain <node> --ignore-daemonsets

systemctl reboot

After reboot, uncordon it. If it still fails, replace the node. I've seen this with corrupted CNI binaries or bad network interface configs that a reboot can't fix.

Preventing This in the Future

Always set systemd.unified_cgroup_hierarchy=1 on the kernel command line — cgroups v2 reduces namespace race conditions.
Don't use the default bridge CNI with Docker. Switch to Calico or Cilium for production.
Monitor /var/run/netns/ size with a cron job. If it exceeds 100 files, something's leaking.