Last time: three stateless services, one request path, and a security model made mostly of things the system doesn’t have. All of that ran on a throwaway k3d cluster. This chapter puts it on machines I own.
TL;DR
- The cluster is three machines: an Arch laptop as the control plane (amd64), a Raspberry Pi (arm64) and a WSL2 desktop (amd64). It runs kubeadm, Kubernetes v1.36 and Calico in IPIP (IP-in-IP tunnel) mode.
- A
kubeadmoverlay and one script (make up-kubeadm) build the images on the control plane, import them into its containerd and deploy. - App pods are pinned to the control plane, so any request that enters through Traefik on another node crosses a tunnel. That decision comes back to bite later.
- Two incidents from this stretch have their own posts: an nginx permission error, and cross-node traffic that died while every health signal stayed green.
The machines
| Role | Hardware | Architecture |
|---|---|---|
| Control plane | Arch Linux laptop | amd64 |
| Worker | Raspberry Pi | arm64 |
| Worker | WSL2 desktop | amd64 |
Traefik comes from its Helm chart in its own namespace, exposed on NodePorts 30080 and 30443 with no LoadBalancer, since there is nothing at home to provide one. Calico is the CNI because the NetworkPolicies from the last chapter only mean something if the CNI enforces them.
A mixed-architecture cluster has one practical consequence. Images are built for amd64, so they can only run on amd64 nodes.
The kubeadm overlay
The base manifests don’t know about any particular cluster. A kustomize overlay adds what this one needs:
- Image tag
kubeadm, pull policyNever. There is no registry in this setup. The script builds the images on the control plane and imports them straight into its containerd (thek8s.ionamespace, the one the kubelet reads). - A node selector and a toleration. Because the images exist only on the control plane, every app pod is pinned there and tolerates its taint.
- Traefik’s namespace. The NetworkPolicy that admits ingress traffic is patched to match the
traefiknamespace. - The ingress class and the public URL. The Ingress gets
ingressClassName: traefik, and the OAuth redirect URL points atlocalhost:30080.
The script checks before it acts: the right kubectl context, the right machine, an amd64 host, a Ready node, the IngressClass present, and a .env file. A deploy script that refuses to run in the wrong place is cheaper than a postmortem.
Pinning has a cost, and it is easy to miss on the day. Traefik can run on any node. If it runs on the Pi, every request travels Pi → tunnel → laptop. A request that fails between nodes looks like an app failure and isn’t one.
A fixed tag is a trap
The images always carry the tag kubeadm. kubectl apply sees an unchanged manifest and does nothing, so a freshly imported image is never picked up.
The fix is in the script: after applying, it annotates each Deployment with a change cause and runs kubectl rollout restart. The annotation is copied onto the new ReplicaSet, so kubectl rollout history shows what each revision was, by default the git commit, marked (dirty) if the tree has uncommitted changes. Before that, the history showed <none> for every revision.
Do we need two replicas?
I asked Claude Code whether the app needed two replicas. The answer separates two ideas:
- Statelessness makes replicas safe. Two copies of drive-sync can serve any request, because the state is in an encrypted cookie. The dashboard is static files.
- Availability is what makes them useful. Two replicas on the same node protect against a crashed process, not a dead node.
The manifests say replicas: 2 for drive-sync and the dashboard, and 1 for analytics, where one replica keeps the in-memory cache warm. There is no PodDisruptionBudget and no topology spread constraint, and with every pod pinned to one node there isn’t much to spread. So the replicas are honest about what they do. They cover rolling updates and process crashes, not hardware.
Two outages
The first deploy on real hardware had two failures, and each is a post of its own:
- The dashboard pod crashed with
Permission deniedon its own nginx config. The file was readable by me and by root, and the container runs as a different user. The first fix did not work, which is the interesting part. - Pod traffic between nodes died while every Calico health check stayed green. BGP said Established. The packets disagreed. Moving Traefik to another node changed nothing. It took three separate days to put together.
I’ll tell both with their evidence, one hop at a time. They are posts, not footnotes, because each has a mechanism worth knowing about when it isn’t your cluster.
Takeaways
- Pinning is a decision with a network consequence. If a workload can only run on one node, say so, and check what has to cross a node boundary to reach it.
- Constant image tags need a restart step. Either use a unique tag per build, or make the script restart the Deployment and record why.
- Replicas answer “what dies?” Name the failure you’re covering before counting copies.
- Make the deploy script say no. Guard clauses cost five lines each and rule out a class of mistakes.
What’s next
Next, the first real backup, and what happens when real data meets assumptions made from a synthetic one.
The story continues as the work does.
Source: github.com/eyosias-k-negash/ivk8. Commit: a194cca.