š TL;DR
CNI is a specification, not a piece of software: a small contract of ADD, DEL, and CHECK commands that Kubernetes calls on every podās birth and death. Flannel, Calico, and Cilium all implement that same contract, they just move packets between nodes differently: an overlay tunnel, routed BGP, and an eBPF dataplane, in that order. Pick one based on cluster size, policy needs, and team familiarity, not GitHub stars, and know where to look first when the contract fails.
Introduction
The Rules Need Something to Enforce Them
Article one laid out four rules every Kubernetes network must satisfy.
Every pod gets a real IP.
Any pod reaches any other pod, on any node, with no NAT.
A pod sees its own address the way everyone else does.
Node agents can always reach local pods.
Those are guarantees, not machinery. Something still has to create the podās network interface, hand it an address, and wire in the routes that make rule two actually true. That something is CNI.
CNI stands for Container Network Interface, and the acronym is doing real work: it names a specification and a plugin contract, not a product you install. Kubernetes itself ships no networking. It calls out to whatever plugin binary is configured on a node and trusts that binary to satisfy the four rules. Flannel, Calico, and Cilium are three different answers to the same contract, which is why comparing them plugin for plugin means little until the contract itself is clear.
The spec lives under the Cloud Native Computing Foundation, the same home as Kubernetes, and it predates Kubernetes' adoption of it. Mesos and Cloud Foundry orchestrate against the same contract, which is part of why a plugin binary sits in a directory as mundane as any other executable, /opt/cni/bin, with nothing Kubernetes specific baked into the interface itself. Only kubelet's decision to call it is Kubernetes specific.
CNI is an interface Kubernetes calls, not a network Kubernetes runs.
Part One
The CNI Contract, Three Commands
The specification is small on purpose. A CNI plugin is a standalone executable dropped on every node, usually under /opt/cni/bin. The container runtime invokes it directly, passing network configuration as JSON on stdin and pod identifying details as environment variables. Nothing in the spec requires a background daemon, though most real plugins run one anyway to keep routing state in sync across nodes.
A single network is rarely one plugin doing everything. The spec allows chaining: a main plugin such as bridge or ptp creates the actual interface, an IPAM plugin such as host-local or dhcp hands out the address, and optional meta plugins layer on extras like port mapping or bandwidth limits. Each plugin in the chain receives the previous pluginās result as its own input, so by the time the chain finishes, the pod has an interface, an address, and whatever the chain added on top.
{
"cniVersion": "1.0.0",
"name": "cluster-cni",
"plugins": [
{
"type": "bridge",
"bridge": "cni0",
"isGateway": true,
"ipam": {
"type": "host-local",
"ranges": [[{ "subnet": "10.244.1.0/24" }]]
}
},
{ "type": "portmap", "capabilities": { "portMappings": true } }
]
}
ADD, bring a pod onto the network
Called once, when a podās sandbox is created.
Given the container ID, the network namespace path, and the interface name to configure.
Expected to return the IP, routes, and DNS it assigned, as JSON.
DEL, tear it back down
Called when the pod is deleted, releasing its IP back to IPAM.
Expected to be idempotent, since kubelet can call it more than once during cleanup.
CHECK, confirm nothing drifted
A newer, optional addition to the spec.
Verifies the interface, routes, and IP still match what ADD originally set up.
š” Why āinterfaceā is the exact right word
Nothing in the spec says how a plugin must move packets between nodes. It only fixes what happens at the edge of a single pod. Tunneling, routing, eBPF, all of that is the pluginās own design space.
Part Two
The ADD Lifecycle, From Kubelet to a Working IP
Tracing one pod creation makes the contract concrete, and it reuses a piece from article one: the pause container is what actually receives everything the plugin sets up.
1. Kubelet asks the runtime for a sandbox
The runtime creates the pause container first, which does nothing except hold open a network namespace for the pod.
2. The runtime invokes the CNI plugin with ADD
It passes the pause containerās namespace path, a container ID, and the interface name, commonly eth0.
3. The plugin creates a veth pair
It moves one end into the podās namespace and renames it eth0. The other end stays on the host, wired into a bridge or the host routing table.
4. The plugin assigns an IP through IPAM
Usually a small delegated IPAM plugin, tracking which addresses in the pod CIDR are already handed out, often as lease files on disk so two concurrent ADD calls never claim the same address.
5. The plugin writes routes and returns a result
The routes are what satisfy rule two from article one: any pod, any node, no NAT. The result JSON becomes the pod IP kubectl eventually shows you.
# What the runtime effectively runs on ADD
CNI_COMMAND=ADD CNI_CONTAINERID=3f9a21 CNI_NETNS=/var/run/netns/cni-8b21 \
CNI_IFNAME=eth0 CNI_PATH=/opt/cni/bin /opt/cni/bin/bridge < /etc/cni/net.d/10-cluster.conf
# The host side of the resulting veth pair
ip link show type veth
DEL undoes exactly what ADD built, in reverse. The plugin looks up the same container ID and namespace, deletes the veth pair, removes the routes it added, and returns the IP to the IPAM pool so another pod can claim it later. Kubelet expects DEL to be safe to call more than once, since a crashed kubelet or a node reboot can trigger the same cleanup twice.
š What this actually satisfies
This whole sequence only makes rules two and three true. Rule four, node agents reaching local pods, falls out of the host end of the veth pair sitting in the root namespace where kubelet already lives.
Part Three
Three Plugins, Three Strategies for the Same Contract
Every CNI plugin implements ADD, DEL, and CHECK. Where they differ is the dataplane behind those three commands, how a packet actually gets from one node to another, and what that choice costs you.
1. Flannel, an overlay chosen for simplicity
Wraps every pod packet inside a VXLAN tunnel between nodes.
Works over almost any underlying network, since the physical layer never sees a pod IP directly.
Adds roughly 50 bytes of encapsulation overhead per packet, which is why flannel deployments commonly run pod interfaces at an MTU of 1450 instead of the usual 1500.
Carries no network policy engine of its own.
2. Calico, routed and chosen for BGP and policy
In its default mode, skips encapsulation entirely and announces pod subnets between nodes over BGP.
Peers every node with every other node by default, a full mesh that gets expensive to maintain well before three digit node counts, at which point route reflectors take over the peering instead.
Ships a full NetworkPolicy implementation, including cluster wide default deny rules.
Falls back to IPIP encapsulation automatically on networks that cannot route pod subnets directly.
3. Cilium, chosen for eBPF depth and observability
Replaces much of the traditional iptables based dataplane with eBPF programs running in the kernel.
Can replace kube-proxy entirely, handling Service load balancing in eBPF instead of iptables or IPVS.
Exposes per flow visibility through Hubble, down to which pod talked to which pod over which port.
Enforces policy up to layer seven, including HTTP method and path, and needs a reasonably modern kernel to unlock the full feature set.
Part Four
A Decision Framework, Not a Popularity Contest
None of these three is universally correct. The choice comes down to three questions asked in order.
How large is the cluster, and does the network team already speak BGP.
Does the workload need enforced network policy on day one, or is that a later concern.
How comfortable is the team maintaining eBPF based tooling versus a simpler overlay they can reason about with tcpdump alone.
In practice that maps to rough profiles rather than a scorecard. A small team running a handful of services on a handful of nodes rarely needs anything past Flannel, since the overlay cost is invisible at that scale and the operational surface stays small. A platform team enforcing tenant isolation or compliance boundaries between namespaces reaches for Calico's policy engine long before it reaches for BGP peering itself. A team already running eBPF tooling elsewhere, or one that needs layer seven aware policy and flow level tracing for auditing, is the one actually served by what Cilium's added complexity buys.
Pick the plugin for the traffic and the policy you actually have, not the one with the most stars on GitHub.
ā ļø The one way door
Switching CNI plugins on a live cluster is disruptive, since pod IPs, routes, and often the CIDR layout change underneath running workloads. The pod CIDR, the encapsulation choice, and the policy model are all baked into every nodeās routing table and iptables rules, so undoing that means rebuilding it cluster wide, not swapping a config file. Most teams pick once, early, and live with the tradeoff rather than migrate later.
Part Five
Where to Actually Look When a Pod Wonāt Get an IP
Every guarantee above eventually fails at an inconvenient hour, and the failure almost always shows up the same way: a pod stuck in ContainerCreating, with an event citing a failure to set up the pod's network. Kubelet only reports what it was told. The real cause sits with the plugin binary or its daemon, not with kubelet itself.
1. Check kubeletās own account first
journalctl -u kubelet is where the ADD failure actually surfaces, since kubelet is the one invoking the plugin and reporting back what it returned.
2. Check the pluginās own logs next
Calico, Cilium, and Flannel all run as a daemon pod on every node, calico-node, cilium-agent, and kube-flannel respectively, and their logs usually show the exact reason ADD returned an error.
3. Look for exhausted IP pools
host-local IPAM tracks leases as files under /var/lib/cni/networks/<name>/. A crashed kubelet or a plugin that skipped DEL can leave stale leases behind, and a small pod CIDR runs out of addresses faster than anyone expects.
4. Check for an MTU mismatch
An overlay plugin encapsulating packets without lowering the podās MTU to match produces a specific symptom: small packets work, large ones silently vanish, usually showing up as truncated TLS handshakes or stalled large transfers.
# What kubelet actually saw when ADD ran
journalctl -u kubelet --since "10 min ago" | grep -i cni
# Stale IP leases left behind by a crashed pod
ls /var/lib/cni/networks/<network-name>/
# Confirm the pod's actual interface MTU
kubectl exec <pod> -- ip link show eth0
š” Why this list is short
Most CNI incidents reduce to one of three things: the plugin never ran, ran and hit an exhausted address pool, or ran fine and left a packet size mismatch behind. Start with kubeletās own account before touching the plugin at all.
Closing
What CNI Doesnāt Give You
A CNI plugin makes a podās IP real the moment kubelet asks for it. It says nothing about what happens next. Pods die, get rescheduled, and come back with a new IP every time, and the pluginās job ends the instant DEL runs.
Some callers can tolerate that churn by watching the API server directly. Most cannot. And that gap, a stable name for a set of pods whose addresses keep changing, is exactly what Services exist to close.
šŗļø What to read next
āServices Demystifiedā ā how kube-proxy and CoreDNS turn a moving set of pod IPs into one address you can depend on.





A good read. Keep em coming!
I didn't get a thing akiššš