A VPN sidecar that cut two nodes off the internet

Summary

While debugging a VPN container that refused to connect, I ran it with hostNetwork: true to rule out Kubernetes pod networking as the cause. The container has a built-in kill switch: firewall rules that block all traffic except the VPN tunnel. With host networking, those rules were applied to the physical node instead of the pod. The VPN then failed to connect, the rules stayed behind, and the node lost all internet access. Every other pod on it went down with it. This happened on two nodes in the same evening. The lesson for me was about blast radius: giving a container access to the host is a design decision, not a config flag.

Context

I wanted my torrent client to route all its traffic through a commercial VPN. The usual way to do this in Kubernetes is a sidecar: a second container in the same pod that runs gluetun, which sets up a WireGuard tunnel. Because containers in a pod share one network namespace, the application's traffic goes through the tunnel without the application knowing anything about it.

Gluetun also comes with a kill switch. On startup it rewrites the iptables rules of the network namespace it lives in. The default policy on outbound traffic becomes DROP, with exceptions only for localhost, established connections, the local subnet and the VPN endpoint itself. That is exactly what you want from a VPN container: if the tunnel drops, the application must not leak traffic out the real interface. Inside a pod's own namespace the only thing those rules can affect is the pod.

The problem was that the tunnel would not come up. Gluetun's logs showed the same error on every attempt:

RTNETLINK answers: No such device

That is a kernel routing error, and it appeared while gluetun tried to add an IPv6 route for the tunnel. I spent most of an evening on it. I checked MTU, IPv6 routing on the node, kernel versus userspace WireGuard, and whether my ISP or router was blocking the endpoint. None of it fixed anything.

Symptoms

The first outage did not come from Kubernetes at all. To rule out pod networking, I brought the same WireGuard config up directly on k3s-worker-01 with wg-quick. It failed with the identical error. That was actually useful, because it proved the problem was not Kubernetes. But the interface it created on the way to failing was never cleaned up. A wg0 interface with routes attached stayed on the node and hijacked its traffic, and the node lost internet.

The second outage came an hour or two later and is the one this writeup is really about. Still trying to find the connection failure, I deployed the gluetun pod with hostNetwork: true on k3s-worker-02, so the pod would use the node's network namespace directly instead of its own. Shortly after, that node stopped answering as well. DNS resolution timed out, then pings to public addresses showed 100% packet loss. Image pulls on the node failed. Every pod scheduled there was effectively offline.

$ ping -c 3 1.1.1.1
3 packets transmitted, 0 received, 100% packet loss

Investigation

The first outage was easy once I looked at the interfaces. ip a showed the leftover wg0, and wg-quick down wg0 brought the node back. Manually created interfaces do not clean themselves up, which I knew in theory and now know in practice.

The second one I initially read as a DNS problem, because DNS was the first thing to fail. systemd-resolved was running and processing requests, it just could not reach its upstream servers. Pinging 1.1.1.1 directly, with no DNS involved, also failed. So the node had lost connectivity below the DNS layer. Same symptom as the wg0 incident, but this time ip a showed no stray interface.

The timing pointed at gluetun. The node had gone dark right after the hostNetwork pod started its connection attempts, and I knew what gluetun does on startup. If its kill-switch rules had been written into the node's iptables instead of a pod's, then a failed VPN connection would leave the node in exactly the state the kill switch is designed to produce: everything blocked except the tunnel, and no tunnel. Checking the firewall confirmed it:

$ sudo iptables -L -n -v

Chain OUTPUT (policy DROP)
 target  prot  in   out   source      destination
 ACCEPT  all   *    lo    0.0.0.0/0   0.0.0.0/0
 ACCEPT  all   *    *     0.0.0.0/0   0.0.0.0/0   state RELATED,ESTABLISHED
 ACCEPT  all   *    eno1  0.0.0.0/0   192.168.x.0/24
 ACCEPT  udp   *    eno1  0.0.0.0/0   <vpn endpoint>   udp dpt:51820

A default DROP on the node's outbound chain, and the only external destination allowed was the VPN server on port 51820, which the node could not establish a tunnel to. These were the kill switch rules, applied to the wrong namespace.

I scaled the deployment to zero and expected the rules to disappear with the container. They did not. Nothing runs gluetun's teardown when a pod is killed, so the rules it had written into the host stayed there after the container that wrote them was gone.

Root cause

hostNetwork: true removes the namespace boundary that normally keeps a container's network changes contained. Gluetun's kill switch locks down whatever namespace it runs in, and with that flag the namespace was the physical node. When the VPN failed to connect, the node was left with the lockdown and without the tunnel. Kubernetes has no concept of undoing what a container did to the host, so deleting the pod restored nothing. The connection failure itself was a separate problem. The outage came from how I chose to debug it.

Fix

The immediate repair was a manual firewall reset on the node: set the default policies back to ACCEPT and flush the rules. That was safe here only because k3s and Flannel regenerate their own iptables rules automatically. Connectivity came back right away. I verified plain reachability and a registry pull before touching anything else.

The structural fix was to stop using hostNetwork for this workload at all. The VPN sidecar now runs in the pod's own network namespace, where a misfiring kill switch can only take down the one pod that wanted the VPN in the first place. That was the original design. I only reached for hostNetwork as a diagnostic step, and the lesson is that a diagnostic step with node-level side effects needs the same care as a production change.

The connection failure that started all of this was eventually solved by switching VPN providers. The same WireGuard credentials that failed inside every container mode worked fine on a bare Linux host and on my laptop. A different provider's config, using gluetun's native provider mode, connected on the first try. I never conclusively found out what in the original provider's config did not work inside the container environment.

What I'd do differently

Never test a privileged network change on a node that is running other workloads. Both outages came from experiments, a manual wg-quick and a hostNetwork pod, done on live worker nodes because that was convenient. Draining the node first, or testing on a throwaway VM, would have cost five minutes and prevented both.

Read what a container does to its host before giving it host access. I knew gluetun had a kill switch and I knew what hostNetwork meant. I did not put the two together until a node was dark. Any container that needs NET_ADMIN or hostNetwork can break the node, and I should treat it that way.

When a node loses connectivity after network experiments, check for leftovers first: ip a for interfaces, iptables -L for rules. In both incidents the cause was state that outlived the process that created it, and both were visible within a minute once I looked in the right place.