Nothing But
Processes
Here is the claim this whole doc defends: there is no such thing as a container.
There is no struct container in the Linux kernel, no container syscall, nothing
to point at. What exists is an ordinary process that has been lied to about the world it lives in.
Kubernetes is a control loop over those lies. GKE is Google running the control loop so you don't have to.
We build the whole stack from the bottom: what a process is, what the kernel will and won't hide from it,
how a filesystem gets swapped underneath it, who counts its memory — and only then, many modules later,
what kubectl apply actually sets in motion inside a Google datacenter.
No layer stays a black box.
And it comes with something to build: Scarred Canvas, a generative artwork that cannot be made by one machine and whose subject is the cluster's own mortality. One step per module, starting below.
Reading eighteen modules about infrastructure and building nothing would be a waste. So this doc has a companion: one generative artwork, built one step per module, which cannot be made by a single machine and whose subject is the cluster's own mortality.
The canvas is a Gray-Scott reaction-diffusion field — the chemistry that makes coral, leopard spots and fingerprints — split into a grid of tiles. One pod owns one tile. Here is the property that makes the whole thing work: computing the next state of a cell requires its four neighbours. So cells on a tile's edge need values that live on the adjacent tile. Every step. Every tile. Forever.
Most "art on Kubernetes" projects are render farms, where the distribution is incidental — one big machine with more patience would produce the same picture. The art and the infrastructure never actually need each other, so the infrastructure is decoration.
A neighbour-coupled simulation is different. The tiles have to find each other (DNS, headless Services), keep stable identity across restarts (StatefulSets), survive eviction with their history intact (PVCs, checkpoints), and exchange state continuously or the picture is measurably wrong. Every one of those is a module in this doc, and none of them is optional for the piece to exist.
Then it runs on Spot nodes, so Google reclaims machines out from under it. A reclaimed tile restarts from its last checkpoint while its neighbours kept evolving. The finished piece is a record of that.
That was the pitch. Before designing anything further it needed testing, because two things could have
killed it — and one of them nearly did. The experiment (cmd/scarproof) runs two
identical universes of two adjacent tiles with a real halo exchange, rewinds one tile in one universe,
and diffs the outcomes. It found a bug and then it found a constraint.
While growth is still advancing, though, the mark is plain — and the reason is nice: in a growing system the geometry of where two fronts collide is set by their relative timing, and a rewound tile arrives late forever. So a preempted tile is visibly younger.
That is a better piece than the one originally pitched, and it is true rather than asserted. It also hands the artwork a deadline: it has to be captured while growth is still advancing, so sizing the canvas so that it never finishes becomes a compositional decision rather than a performance one. Art with a shutter speed, set by a reaction rate.
A confession about the two findings above. They were run for you rather than by you,
which is a real loss — a result you produce yourself is worth more than a fact you are handed, and
discovering that the scar heals is the best moment this project had to offer. Treat them as claims to
check rather than settled ground. The questions they opened are in
EXPERIMENTS.md and nobody has answered them: does damage ever cross the
seam, or was 4,000 steps just too short? What checkpoint interval makes a scar invisible — a number you
need in module 14, where good engineering and the artwork want opposite things? Is there a parameter
regime with grain boundaries, where scars survive even a filled canvas? I guessed at that last one and
never tested it.
Nothing gets built before the module that explains it. Each module ends with one step, so the piece and your understanding grow together.
| Module | What you build |
|---|---|
| 01 · processes | Run two tiles as bare processes. Collide their ports, paths and memory on purpose. ← you are here |
| 04 · images | A scratch image around the static binary — a container with no distro in it at all |
| 05 · runtimes | Start one tile under unshare, then runc, then containerd. Same process, three altitudes |
| 10 · pods | The tile Pod: renderer + halo sidecar sharing one network namespace |
| 11 · scheduler | Spread tiles across nodes deliberately; watch placement decisions happen |
| 13 · networking | Neighbour discovery over headless-Service DNS — the canvas becomes connected |
| 14 · storage | Checkpoints to PVCs. This is what makes a scar partial instead of fatal |
| 15–17 · GKE | Spot node pools, Workload Identity to GCS, and then the first real preemption |
| 18 · capstone | Trace one halo packet end to end. Then print the thing |
Repo: ~/scarred-canvas · Go, because a static binary makes the module-4
scratch image the sharpest possible callback to "the three things nobody wrote down".
Before any of this vocabulary earns its keep, you need to feel the actual problem. It is not "deployment is hard" in the abstract. It is a specific, boring, mechanical problem, and it has exactly one sentence:
A program is not self-contained. The thing you built is a binary plus an enormous set of assumptions about the machine it will land on — and nothing in the file records those assumptions.
Say you compiled a service on your laptop. It links against libssl.so.3.
It shells out to ffmpeg. It reads a config from /etc/myapp/.
It expects Python 3.11 on PATH, a TZ of UTC, a writable
/var/cache, and a kernel new enough to have io_uring.
You copy the binary to a server. The server has libssl.so.1.1, no ffmpeg,
Python 3.9, and a read-only /var. The binary does not run. Nothing was wrong
with your code.
- 1The filesystem it expects. Every shared library, interpreter, CA bundle, locale file, and config path. Dozens to thousands of files that must exist, at the right paths, at the right versions.
- 2The resources it expects. "Enough" memory, "enough" CPU, a port it can bind. All unstated, all shared with whatever else is on the box.
- 3The neighbours it tolerates. Which other processes may exist, what they may do to it, and what it may do to them. Also unstated — and by default, the answer is "anything".
Every technology in this doc — images, namespaces, cgroups, pods, requests, node pools — is an answer to one of these three. Keep the numbering; we come back to it constantly.
For about twenty years the industry's answer to all three at once was: ship a whole computer. Put the app in a virtual machine, with its own kernel and its own copy of an operating system, and the assumptions travel with it. This genuinely works. It is also enormous — you pay for a full kernel, a full init system, and a full userland per application. We will price that out precisely in section 04, once we have the vocabulary to say what we are actually paying for.
We are going to spend eighteen modules on things that turn out not to exist. So let's be precise about
the one thing that does. A process is a kernel data structure — in Linux, a
task_struct — plus the memory it owns. When you ask the kernel to run a program,
it allocates one of these and fills in fields. Understanding which fields exist is genuinely most of
the battle, because a container is nothing more than a process whose fields were filled in unusually.
A PID, a parent PID, a process group, a session. This is how anything refers to it — how you kill it, how ps lists it.
Page tables mapping virtual addresses to physical memory. This is the one thing that has always been isolated: process A genuinely cannot read process B's memory. Everything else on this list is shared by default.
Small integers pointing at open files, sockets, pipes. 0, 1, 2 are stdin/stdout/stderr; the rest is whatever it opened.
A uid, a gid, supplementary groups — and (module 6) a set of capabilities. This is what permission checks consult.
A current working directory and a root directory, plus a pointer to a set of mounts. Note that "what / means" is a per-process field. Remember that; module 4 lives here.
Seven or eight pointers, one per namespace type, each naming which "view" of some kernel subsystem this process gets. Module 2 is entirely about these six words.
Now the load-bearing observation. A process, running, is just a CPU executing instructions against its own memory. Adding two numbers requires nothing from the kernel. But the moment a process wants to touch anything outside its own address space, it has to ask — open a file, send a packet, start another process, allocate more memory, read the clock. Each of those is a syscall: a controlled trap into the kernel. On x86-64 Linux there are roughly 350 of them, and they are the complete list of things a process can do to the world.
If every interaction with the outside world goes through a chokepoint the kernel controls, then
the kernel can lie at that chokepoint. It can answer openat("/etc/hosts")
with a different file than the one the host means. It can make getpid() return
1 to a process the host calls 48213. It can let readdir("/proc")
list four processes on a machine running nine hundred. It can make socket() hand back
a socket in a network stack with its own routing table and its own idea of what eth0 is.
None of that is emulation. There is no interception layer, no translation, no performance penalty worth mentioning. The kernel simply consults a different set of pointers when servicing that process's syscalls. The process runs at full native speed and is completely wrong about what machine it is on.
That is the entire trick. A container is a process for which the kernel has been configured to answer a specific set of syscalls differently. Everything else — images, registries, YAML, GKE — is logistics built on top of that one sentence.
One honest caveat, because we said no black boxes: a handful of "syscalls" are serviced without a real trap
into the kernel. gettimeofday() and friends are handled by the vDSO,
a page of kernel-provided code mapped into every process, precisely because trapping is too expensive for
something called that often. It is an optimisation, not an exception to the model — and it is the reason the
time namespace (2020) was awkward to add and is still barely used.
Put two services on one Linux box as ordinary processes and three separate things go wrong. They are worth separating carefully, because each one is fixed by a different kernel mechanism, and those three mechanisms are modules 2, 3 and 4. Almost everyone learns them as one blurry idea called "containers". Learning them as three distinct failures is what makes the rest of this doc easy.
Our machine: one host, 64 GiB of RAM, 16 cores. Two services, both running as uid 1000 because someone was in a hurry: web-app (a model server, PID 812) and billing (a nightly report job, PID 3391). Walk the three tabs.
:8080.billing may do to web-app — today, with no exploit- ✓Read its secrets.
cat /proc/812/environprints web-app's entire environment, API key included. Same uid, so the permission check passes. - ✓Kill it.
kill -9 812. Same uid is the whole authorisation story for signals. - ✓Read and rewrite its files. There is one filesystem.
/srv/web-app/config.yamlis as reachable to billing as to web-app. - ✓Enumerate it.
ls /proclists every process on the machine, with command lines. Deployment topology, arguments, and often credentials-in-argv, free. - ✓Steal its port. There is one network stack, so one
:8080. Restart web-app while billing happens to hold that port and web-app fails to bind. - ✓Watch its traffic. One
lo, one set of interfaces. Given the right capability, onetcpdump.
Note what is not on this list: reading web-app's memory directly. Address spaces have been isolated since the 1960s. That one field was done right, and it is the reason the others feel so startling — we assume "process" means "isolated" because the memory part is.
Those three failures — and their three fixes — are the whole reason a container has three ingredients.
Any time something later in this doc confuses you, ask which of the three it is serving. A Kubernetes
resources.limits block is failure 2. A NetworkPolicy is
failure 1. An image tag is failure 3.
Virtual machines solve all three failures completely and they solve them better than containers do. A VM gets its own kernel, so its process table, its network stack, its filesystem and its memory accounting are separate by construction — not by the kernel agreeing to answer differently. There is no shared kernel to escape through. If isolation strength were the only axis, this doc would end here.
So the question is only ever what a VM costs. And the honest answer is more interesting than the marketing one. Drag the slider: how big is the workload you want to pack onto one 64 GiB host?
At small workload sizes the per-instance tax dominates and containers win big on density.
Model: 64 GiB host, 4 GiB reserved for the host itself, 180 MiB tax per VM, 3 MiB per container. Black cells are tax; coloured cells are your app.
Slide it to the right and watch the ratio collapse toward 1.0×. That collapse is the honest part. If your workload is a 2 GiB JVM, containers buy you almost nothing on memory density. Anyone who tells you containers are "100× lighter than VMs" is quoting a hello-world benchmark. The durable wins are the other three rows:
| Virtual machine | Container | |
|---|---|---|
| Start to serving | ~5–30 s (firmware → kernel → init → app) | ~20–80 ms (clone, mount, exec) |
| Kernels running on host | N + 1 — each patched, booted and monitored separately | Exactly 1 |
| Identical files across instances | Duplicated in every guest's page cache | One copy in the host page cache, shared |
| Memory commitment | Pre-committed per VM, mostly unreclaimable | Allocated on demand; the host reclaims freely |
| Isolation boundary | A separate kernel — strong, well understood | A shared kernel, configured to lie. Weaker (module 6) |
Read the startup row again, because it is the one that changed how software is deployed. If starting an instance costs 20 ms instead of 20 seconds, then starting instances becomes something a program can do in a loop, thousands of times a minute, in response to load or failure. That is the precondition for everything Kubernetes does. Autoscaling, rolling updates, restart-on-crash, and rescheduling a dead node's workload are all just "start a lot of instances quickly" — and they are only practical because the unit is a process rather than a computer.
And the shared-file row is the one people underrate. Run two hundred instances of the same image and the read-only layers exist once in the host's page cache. Two hundred VMs each hold their own copy of glibc, their own copy of Python, their own copy of everything. Module 4 shows exactly how overlayfs makes that sharing structural rather than lucky.
The tradeoff is real and it is a spectrum, not a winner. Because the isolation row genuinely favours VMs, the industry has spent a decade building things in the middle: Firecracker and Cloud Hypervisor (microVMs that boot in ~125 ms and drop most device emulation), Kata Containers (a real VM per pod, wearing a container interface), and gVisor — which is worth remembering, because it is Google's, and it implements the Linux syscall interface in userspace rather than trusting the host kernel to lie safely. When we reach GKE Sandbox in module 17, that is what is running underneath.
We can now say the central claim exactly. Ask the kernel to create a process and it will. Ask it to create a namespace, a cgroup, a mount — it will. Ask it to create a container and it will not know what you mean. There is no such object, no such syscall, no such field.
What you call a container is a recipe: start a process, and while starting it, do five things. Every container runtime ever written — Docker, containerd, runc, CRI-O, podman, and the code inside GKE — is an implementation of this list.
- 1Unpack a root filesystem. Stack the image's read-only layers with overlayfs and add a writable layer on top. You now have a directory that looks like a whole Linux install. → failure 3 · module 4
- 2Create a cgroup and set the limits. Write to
memory.max,cpu.max,pids.max. Do this before the process can allocate. → failure 2 · module 3 - 3Clone the process with new namespaces. One
clone()call withCLONE_NEWPID | CLONE_NEWNS | CLONE_NEWNET | CLONE_NEWUTS | CLONE_NEWIPCand friends. The child is born into a world of its own. → failure 1 · module 2 - 4
pivot_rootinto the new tree, then mount a fresh/proc,/sysand/devinside it. This is the step where the process stops being able to see the host's files at all. - 5Drop privileges and install filters. Shed Linux capabilities, load a seccomp filter that blocks most of those ~350 syscalls, apply AppArmor or SELinux, set
no_new_privs. → module 6 - 6
execve()the program. The process replaces itself with your binary. From here on it is an entirely ordinary process, running at native speed, with no idea any of the above happened.
Six steps, all of them plain Linux syscalls available since roughly 2013. You can do this by hand with
unshare and chroot in about ten lines of shell — and in
module 5 you will.
Here is the consequence that catches people out. Because a container is just a process, it is
still visible from the host. Run ps aux on a Kubernetes node and
your containers' processes are right there in the list, with host PIDs, alongside kubelet and sshd. Run
cat /proc/<pid>/cgroup and you can read which pod it belongs to. Nothing is
hidden from the host — the hiding is strictly one-directional, inward.
Guess each one before you click. Getting this taxonomy straight now will save you a lot of confusion in modules 8 through 12, where the fictions start stacking on top of each other.
task_struct in kernel memory with a PID. The only thing on this board the kernel would recognise as an entity in its own right./proc/<pid>/ns/. It lives as long as something references it, and processes can join existing ones with setns()./sys/fs/cgroup/, holding the limits and the live accounting. You can cat its files right now.
Why bother being this pedantic? Because every hard debugging session in this ecosystem
is a moment where a fiction leaks. A pod is "running" but the process died and PID 1 didn't reap it. A
service has an IP that answers nothing because no endpoint matched the selector. A container is
"using 8 GiB" but that is page cache the kernel would happily drop. A memory limit is enforced but
the JVM inside read /proc/meminfo — which is the host's — and sized its
heap for a machine it isn't on. Every one of those becomes obvious the moment you know which layer is
real and which is a story someone is telling.
The repo at ~/scarred-canvas is stubs and a test suite. The tests
are the specification — twenty of them, pinning the behaviour down closely enough that if they pass, the
canvas works. You write every line of the simulation.
Work down grayscott.go in the order its package comment suggests: New, then
indexing and at, then the update rule, then edges, then checkpointing, then
rendering. cmd/tile is given to you as plumbing — flags, an HTTP server, PNG
encoding — because typing that teaches nothing. The one exception is loadParams,
which encodes a decision rather than boilerplate, so it is a stub with its own tests.
The test that matters is TestSeamHasNoDiscontinuity. Two tiles
coupled across a shared edge, stepping in lockstep, should behave as one continuous surface — you should
not be able to tell where one ends. Every other test is scaffolding to get you to that one. If it fails
while the rest pass, the bug is in what you chose to send across the halo, and working out why
from the symptom is the most valuable ten minutes in this module.
When the sim works you get a picture, which is the reward:
Run two tiles and make all three of module 1's failures happen to you personally. There is deliberately no script for this — the exercise is doing it by hand and noticing what you reach for.
- 1Visibility. Two tiles cannot share a port. Then, from one tile's shell, read the other's environment — on Linux that is
/proc/<pid>/environ. macOS has no/proc; noticing why is worth a minute of its own. → failure 1 - 2Consumption.
-hogholds memory resident on purpose. Find the number that makes your machine unhappy — and if you are brave, the number that makes the OOM killer choose, then note which process it picks. → failure 2 - 3Dependencies. The defaults are absolute paths, like real production software. Get two tiles to fight over
/etc/scarred-canvas/. Then get them to silently corrupt each other by sharing one-statedirectory. → failure 3 - 4And notice they can't share a canvas. Each tile reflects its own edges, so each is a closed universe. Two processes on one machine, no way to be adjacent. That gap is what modules 2–13 close.
Write down every flag you reach for. You will fix all of this with different ports,
different state dirs, a --config override. That list is the container
recipe from section 05, done by hand and badly: you are manually partitioning a namespace (ports), a
filesystem (state dirs), and a dependency set (config). Module 5 replaces each flag with a kernel
primitive that does it properly, and the contrast only lands if you kept the list.
Answer out loud first, in your own words, then click to compare. Module 2 assumes all four of these, so if one feels shaky it is worth re-reading that section rather than pushing on.
/proc/<pid>/environ exposes its environment, kill works on it, its files are your files, its ports come from the same single network stack. This asymmetry is why "process" misleadingly feels like it means "isolated" — one dimension of it always was./proc, which is itself serviced by kernel code. Namespaces are just an extra pointer on the process: when servicing that syscall the kernel consults the process's PID namespace instead of the global one. Same code path, one different lookup. No interception layer exists, so there is nothing to slow down — the process runs at full native speed while being entirely wrong about the machine.oom_score, adjustable via oom_score_adj) and kills the largest — so a 38 GiB model server dies for a 24 GiB report job that merely tipped the machine over. The deeper point: the kernel has no concept of "this batch job's budget", so it cannot contain the damage to the offender. That concept is exactly what a cgroup adds, and cgroup v2 can run the OOM kill inside the offending group.
Module 1 gave us the three failures and the six-step recipe. Modules 2–6 build each ingredient of that
recipe properly, at the level of actual syscalls and actual files under /sys.
Modules 7–14 build Kubernetes as the answer to "now do that across a thousand machines that keep dying".
Modules 15–18 are GKE: what Google operates on your behalf, and where its version differs from vanilla
in ways that will bite you.
a private view
of the world
Eight namespace types, one clone() flag each. We will look at what every single
one virtualises, why PID 1 inside a namespace inherits strange and consequential duties, how a
veth pair between two network namespaces is the seed of all container networking,
and why the user namespace — the newest and most awkward — is the one that decides whether "root in a
container" should scare you. Then we build a working namespace jail by hand.
Written — keep scrolling.
actually is
Module 1 ended on a sentence: the kernel can lie at the syscall chokepoint. A namespace is the mechanism for that lie, and it is much less magical than the word suggests. It is a kernel object — allocated, reference-counted, freed — that represents one instance of a global resource. The process table used to be global. With PID namespaces, there can be many, and each process holds a pointer to the one it belongs to.
That's it. Eight kinds of pointer, hanging off task_struct. When the kernel services
a syscall that touches a namespaced resource, it follows the calling process's pointer instead of reaching
for a global. Two processes with the same pointer see the same world; two with different pointers don't
know the other exists.
Namespaces are exposed as magic symlinks under /proc/<pid>/ns/. Each one
"points at" a fake path containing an inode number — and that number is the namespace's identity:
Compare those numbers between two processes and you know exactly what they share. Same
net inode, same network stack. Different pid inode,
different process table. This is the single most useful debugging trick in the whole ecosystem, and it
works on any Linux box without any container tooling installed at all.
The 4026531xxx range is the initial namespaces — the ones the kernel creates at boot. Anything you create later gets a different number.
- 1
clone()— create a new process, and while doing so, put it in new namespaces. PassCLONE_NEWPID,CLONE_NEWNETand so on. This is how containers start: the namespaces come into existence with the process. - 2
unshare()— move yourself into new namespaces, right now, without forking. Theunsharecommand-line tool is a thin wrapper. (One wrinkle:CLONE_NEWPIDhere affects your children, not you — a process cannot change its own PID.) - 3
setns()— join a namespace that already exists, given a file descriptor to one of those magic symlinks. This is whatnsenter,docker execandkubectl execall do: open/proc/<pid>/ns/net, callsetns(), and now you are inside.
A namespace lives as long as something references it — a member process, an open file descriptor, or a bind mount. When the last reference goes, the kernel frees it. This is why a container's network configuration vanishes the instant its last process exits, and it is also why Kubernetes needs a trick to keep a pod's network alive while its containers restart. We'll meet that trick — the pause container — in module 10.
They arrived over eighteen years, each solving a specific complaint, which is why they are not a tidy set. Toggle them below and watch what the process can see collapse. The command underneath is the real one — if you have a Linux box, you can paste it and get exactly the view described.
/ containsThe first one, which is why its flag is just "NEWNS" with no name in it — nobody knew there would be others. It privatises the mount table, so a process can have a completely different idea of what exists at every path. Module 4's whole job happens in here. Its subtlety is mount propagation: mounts are shared by default on systemd systems, so a naive new mount namespace still sees the host's later mounts unless you explicitly mark it private.
Hostname and NIS domain name. Named for the UNIX Time-sharing System struct it came from, which is a terrible name for "the thing that lets your container be called tile-0 instead of gke-prod-pool-a-9f3x". Trivial in implementation, and the reason logs from inside a pod are legible.
System V IPC objects, POSIX message queues, and /dev/shm. Rarely thought about until it bites: two Postgres containers on one host would otherwise fight over shared memory segments. This is also the namespace that pod containers deliberately share, so sidecars can talk cheaply.
A private process-ID number space. The first process inside becomes PID 1 and inherits duties it almost certainly is not prepared for — which is section 03, because it is where real production bugs live. Also: PID namespaces nest, so a process has a different PID in each ancestor namespace. Your container's PID 1 is PID 48213 on the node.
Interfaces, routing tables, iptables/nftables rules, socket port space, /proc/net. A fresh one contains exactly one interface — lo — and it is DOWN. This is the namespace that makes "every pod gets its own IP" possible, and section 04 is entirely about how you connect one to anything.
uid and gid mapping, plus capabilities. The newest of the load-bearing ones and by far the most consequential for security, because it is the only namespace an unprivileged process may create — which is what makes rootless containers possible and what makes kernel security people nervous. Section 05.
Hides your position in the cgroup tree, so /proc/self/cgroup shows / rather than /kubepods/burstable/pod3f2a…. Purely about not leaking the host's layout — and about letting software inside a container reason about its own limits without needing to know it is in one.
Offsets for CLOCK_MONOTONIC and CLOCK_BOOTTIME only — deliberately not CLOCK_REALTIME, because wall-clock time is genuinely global and pretending otherwise breaks everything that talks to anything. Added for checkpoint/restore, barely used, and awkward precisely because of the vDSO problem from module 1: those clocks are read without entering the kernel at all.
Notice what is missing from this list. There is no namespace for the page cache, no
namespace for kernel modules, no namespace for the system clock's wall time, no namespace for
/proc/cpuinfo or /proc/meminfo. That last one has real
consequences: a JVM inside a container with a 2 GiB limit reads /proc/meminfo,
sees the host's 64 GiB, and sizes its heap for a machine it is not on. Container-aware runtimes had to
be taught to read the cgroup instead. Namespaces virtualise what someone bothered to virtualise, and
the gaps are where the abstraction leaks.
strange duties
The PID namespace deserves its own section, because it is the one that changes how your program
behaves rather than just what it can see. Enter a new PID namespace and the first process becomes
PID 1 — and on Linux, PID 1 is not an ordinary process. It inherits the duties of
init, whether or not it has any idea what those are.
There are three rules, and every one of them causes production incidents:
- 1Signals with default actions are discarded. For an ordinary process,
SIGTERMwith no handler means "die". For PID 1, the kernel skips the default action entirely — the signal is dropped unless the process explicitly installed a handler. This is a safety feature so a stray signal can't kill the system's init. In a container it means your shutdown request may do precisely nothing. - 2Orphans get reparented to it. When any process's parent dies, its children are re-adopted by PID 1 of their namespace. PID 1 is then responsible for calling
wait()on them when they exit. If it doesn't, they stay in the process table forever as zombies — and enough zombies exhausts the PID limit. - 3If PID 1 dies, everything dies. When a PID namespace's init exits, the kernel sends
SIGKILLto every remaining process in that namespace and tears it down. There is no "the main process crashed but the sidecar kept going".
Rule 1 is the famous one, and it has a shape you will recognise the moment you see it. Step through what happens when something asks your tile to stop — and flip between the two ways of writing the same Dockerfile line:
Both forms are the same binary. The only difference is whether a shell sits between the kernel and
your program — and that shell, as PID 1, silently absorbs the signal your program needed to receive.
Kubernetes does exactly what docker stop does here: SIGTERM,
wait terminationGracePeriodSeconds (default 30), then SIGKILL.
A pod that takes the full grace period to die every single time is almost always this bug.
The general fixes, and when each is right. Use exec form so your process is
genuinely PID 1, and install a SIGTERM handler — correct for a single-process
container, which is most of them. If your container really does spawn children, put a tiny init in
front (tini, dumb-init, or
docker run --init): it forwards signals and reaps zombies, which is all PID 1
was ever supposed to do. What you should not do is leave a shell there and shorten the grace
period, which is the common non-fix — it converts a slow ungraceful death into a fast one.
a network namespace
Create a network namespace and you get the most isolated thing in this entire doc: a network stack with
one interface, lo, and it is DOWN. No routes. No addresses. Nothing
can reach in and nothing can get out. It is not "restricted", it is unplugged.
This is genuinely useful — it means two tiles can both bind :8431 with no conflict
whatsoever, because there are two entirely separate port spaces. It also means neither is reachable, which
is not what you wanted. So you need a cable.
A veth pair is exactly that: a virtual Ethernet cable with two
ends. Whatever goes in one end comes out the other. Put one end in the namespace and leave the other on
the host, and you have a link. Connect many host-side ends to a bridge — a virtual switch
— and every namespace can reach every other.
That diagram is not a simplification of Docker's default network. It is Docker's default
network — docker0 is the bridge, and every container gets a veth pair into it.
Kubernetes replaces the addressing scheme and the NAT rules, but the plumbing is the same.
Do that twice, attach both host-side ends to a bridge, and you have built the substrate that every
container network on earth runs on. Module 13 replaces the manual steps with a CNI plugin
— a binary Kubernetes invokes with the pod's namespace path, which does precisely this and prints JSON
describing what it did. Knowing that CNI is "a program that runs ip link for you"
removes most of the mystery from Kubernetes networking before we ever get there.
in here
You now know enough for the question everyone asks badly. Is being root inside a container dangerous? The honest answer is: that depends entirely on whether a user namespace is in play, and by default in Docker and in Kubernetes, it is not.
Recall from earlier: a uid is just an integer the kernel compares. A user namespace adds
a translation table between the integer a process believes it has and the integer the kernel uses for
permission checks against everything outside. The mapping lives in
/proc/<pid>/uid_map, and it is three numbers: inside-uid, outside-uid,
count.
| No user namespace (the default) | With a user namespace | |
|---|---|---|
| uid inside | 0 — root | 0 — root |
| uid the kernel sees | 0 — actually root | 100000 — a nobody |
| Escapes the mount namespace? | You are root on the host | You are uid 100000 on the host |
| Reads a host file owned by root | Blocked only by not being able to see it | Blocked by permissions, properly |
| What actually protects you | Dropped capabilities, seccomp, AppArmor — module 6 | All of that, plus the uid means nothing outside |
Read the second row twice. Without a user namespace, "root in the container" and "root on the host" are the same uid 0. What stops that being catastrophic is not identity — it is that the process has had most of its capabilities stripped and its syscalls filtered. Take those away and there is nothing left.
So why isn't this on by default everywhere? Two honest reasons. First, it breaks things: files on shared volumes end up owned by uids nobody expected, and the remapping has to be consistent across every host that might mount the same storage. Second — and this is the interesting one — the user namespace is the only namespace an unprivileged process is allowed to create. That is exactly what makes rootless containers possible, and it is also what makes kernel maintainers uneasy: it hands unprivileged users a legitimate path into kernel code that previously only root could reach. A long line of privilege escalation CVEs has come through that door, which is why several distributions ship switches to turn it off entirely.
Kubernetes has been catching up here slowly; user namespaces for pods only became broadly usable in recent releases, and we'll come back to what GKE does about it in module 17. For now the takeaway is diagnostic: when someone says "the container runs as root", the useful follow-up is "root mapped to what?"
its own world
Build step 1 ended with two tiles fighting over one port and one filesystem, and a list of flags you invented to keep them apart. Now you delete that list, because the kernel does it properly.
First: you need a Linux box, and that fact is the lesson. None of
unshare, ip netns or /proc/<pid>/ns/
exists on macOS, because namespaces are a Linux kernel feature and nothing else implements them.
This is also the real answer to "how does Docker run on a Mac": it doesn't. Docker Desktop boots a Linux
VM and runs your containers inside that. Every container you have ever run on a laptop has been running
on a hidden Linux kernel. Use lima or colima for a local
VM, or spin a small GCE instance — which has the side benefit of being the same environment modules
15–18 end up in.
Inside, ps aux shows two processes instead of hundreds. Now send it
SIGTERM and watch nothing happen — the tile has no handler installed, and PID 1
discards signals whose only action is the default. Then send SIGKILL and watch it
die instantly, because that one cannot be refused by anybody.
There is a stub and a failing test waiting for you: shutdown() in
cmd/tile, which must write a final checkpoint before the process exits. Install it
as a SIGTERM handler, then repeat task 1 and confirm the signal is now honoured.
This is not busywork, and it is worth seeing why now rather than in module 14. The gap between a tile's
last checkpoint and the moment it dies is the size of its scar. A tile that handles
SIGTERM loses nothing when Kubernetes evicts it politely. A tile that ignores it
loses everything since its last periodic save — and gets SIGKILLed
terminationGracePeriodSeconds later anyway. Your signal handling is an
artistic parameter.
The headline. Give each tile its own network namespace and the port collision from build step 1 simply stops existing — not worked around, gone, because there are now two independent port spaces.
Then discover the cost: you cannot reach either of them. Fix that with the veth recipe from section 04 —
a pair per tile, both host-side ends on a bridge — until curl 10.200.0.2:8431/frame.png
returns a picture. When it does, you have hand-built the thing a CNI plugin will do for you in module 13.
Take the list of flags you wrote down in build step 1 and mark which ones a namespace has just made unnecessary. Ports and state directories should both be gone. Note what is not gone: nothing here stopped a tile from eating all the memory, and nothing gave the two tiles a shared canvas. Those are modules 3 and 13.
Say each answer out loud first. Module 3 leans on the first and last of these especially.
ps shows 3 processes. The host shows 900. Same kernel — what is actually different?ps doesn't ask the kernel directly — it reads /proc, so you also need a mount namespace with a fresh /proc mounted inside it. That's why unshare --pid alone gives you the confusing result of "I am PID 1 but I can still see everything", and why the flag you actually want is --pid --fork --mount-proc. Those 900 processes never went anywhere; they are still there, still running, and still visible from the host — the hiding is one-directional.ENTRYPOINT is in shell form, so PID 1 is /bin/sh, not your program. SIGTERM arrives at PID 1; the shell installed no handler; the kernel refuses to apply a default action to PID 1; the signal is discarded. Ten seconds later the runtime gives up and sends SIGKILL, which nobody can refuse. Your program never heard about any of it and never ran its shutdown path. Shortening the grace period just makes the ungraceful death happen sooner — the fix is exec form (so your process is genuinely PID 1 and can install a handler), or a real init like tini in front if you actually have child processes to reap.:8431 and neither is wrong. The cost is that a namespace is useless until you build it a path, which is what veth pairs and bridges are for, and what a CNI plugin automates.cannot do
Two tiles now have private process tables, private mounts, private network stacks and private hostnames. They cannot see each other, cannot signal each other, cannot collide on a port. Failure 1 from module 1 is comprehensively solved.
And yet: either one can still eat all 64 GiB. Nothing in this module said a single word about how much of anything a process may use. Namespaces answer "what can you see" — they have no opinion whatsoever about "how much can you take". A perfectly isolated tile can still starve every other tile on the node, and the OOM killer will still pick its victim by size rather than by blame.
accounting
& limits
The cgroup2 unified hierarchy as actual files you can cat; why
cpu.weight and cpu.max are completely different promises;
what memory pressure looks like before the kill; and why Kubernetes requests and
limits are not two settings for one thing but two settings for two different subsystems —
which is the single most misunderstood pair of numbers in the ecosystem. Then we cap a tile and make
the OOM killer choose correctly for once.
Not written yet. Say the word and it gets built.