Performance-Fresser ■ Episode 12
“There’s a lot of extra concepts, config files, and infrastructure you have to manage to do something basic.”
Kelsey Hightower, the man who literally wrote Kubernetes Up & Running
Kubernetes was built at Google, for Google: an organisation that runs millions of containers across global data centres, serving billions of requests per second, operated by thousands of engineers who cannot remember the last time they deployed fewer than fifty services simultaneously. Google open-sourced the concept. The CNCF adopted it. And 82 per cent of container users now run it in production.
The question nobody asks at the architecture meeting is whether any of those 82 per cent actually have Google’s problems. The answer, for the overwhelming majority, is no. They have a web application, a database, and a deployment that worked perfectly well before someone added “Kubernetes migration” to the Q3 roadmap.
Let us examine what that migration actually costs.
The Control Plane Tax
Before your application serves its first request, before a single byte of business logic executes, Kubernetes demands its tribute. The control plane is not optional. It is the nervous system of the cluster, and it is remarkably hungry.
The minimum viable control plane: etcd (the distributed key-value store that remembers everything, consuming 2-8 GB of RAM for the privilege), kube-apiserver (the front door through which every request must pass), kube-scheduler (which decides where your pods live), kube-controller-manager (which ensures reality matches the YAML), kubelet on every node (reserving up to 25 per cent of each node’s memory), CoreDNS (because services need names), and kube-proxy plus a CNI plugin (because packets need routes).
A production-grade, highly available deployment of this apparatus, before you have deployed a single line of your own code, requires 6-12 CPU cores and 12-24 GB of RAM. That is the cover charge. The club is empty. The music has not started. But the bill is already running.
“But K3s is lightweight,” someone offers helpfully. K3s, the stripped-down Kubernetes distribution marketed as the sensible alternative, idles at 1.6 GB of RAM. Doing nothing. Serving no requests. Simply existing, in the quiet, expensive way that Kubernetes does.
The YAML Labyrinth
Deploying a single service on Kubernetes requires a Deployment manifest (20 lines), a Service definition (19 lines), and an Ingress resource (27 lines). That is 66 lines of YAML for one service to accept traffic. In production, with ConfigMaps, Secrets, health checks, resource limits, pod disruption budgets, and the other liturgical requirements, the count exceeds 200 lines per service.
The same deployment on FreeBSD: one entry in rc.conf.
The same deployment on Linux: one systemd unit file, ten lines.
The YAML is not merely verbose. It is a maintenance surface. Every manifest is a contract with the cluster, and contracts drift. The YAML committed to Git and the state actually running in the cluster diverge silently, in the way that only distributed systems can. Configuration management becomes configuration archaeology.
The Utilisation Scandal
Here is the number that should appear on every quarterly board report and never does.
According to Cast.ai’s 2024 Kubernetes Cost Benchmark, surveying over 2,100 organisations: average CPU utilisation is 10 per cent. Memory utilisation is 23 per cent. That means 87 per cent of provisioned CPU sits idle. Nine out of ten cycles purchased and never used. The cloud provider sends an invoice for every one of them.
Let that number settle into the chair. For every unit of CPU your application actually consumes, you are paying for ten. Not because the workload requires it. Because Kubernetes requires headroom, redundancy, node overhead, system daemons, and the comforting illusion that autoscaling will sort it out eventually. The cloud provider, naturally, does not object to billing you for nine idle cores.
The Operational Sinkhole
If the waste were quiet, one might tolerate it. It is not.
The Komodor 2025 Enterprise Kubernetes Report surveyed enterprise teams and found numbers that ought to make any CTO reach for the antacids. 38 per cent of teams experience high-impact outages weekly. Not monthly. Not quarterly. Every week, more than a third of Kubernetes deployments suffer incidents severe enough to impact users.
Of those incidents, 79 per cent are caused by recent changes. The very mechanism that Kubernetes is supposed to manage, deployment, is the primary source of its failures. The orchestrator is orchestrating its own outages.
The time cost: 34 workdays per year spent troubleshooting Kubernetes issues. Not building features. Not shipping products. Not serving customers. Thirty-four days, nearly seven working weeks, maintaining the system that maintains the system. It is the infrastructure equivalent of a car that spends more time in the garage than on the road.
The Exits
The companies that have left Kubernetes do not whisper about it. They publish the numbers.
Amazon Prime Video moved their video monitoring pipeline from a microservices architecture on Kubernetes back to a monolith. Cost reduction: 90 per cent. Not 9. Ninety. The same workload, the same throughput, one-tenth of the infrastructure cost. Amazon’s own team decided that Amazon’s own orchestration platform was too expensive for Amazon’s own service. The irony writes itself.
37signals, the makers of Basecamp and HEY, left the cloud entirely. DHH published the savings with characteristic understatement: over $10 million saved across five years. They bought servers. They hired two infrastructure engineers. The maths was not close.
GEICO (yes, the insurance company) reported 2.5 times higher costs after their Kubernetes adoption. The platform that was supposed to reduce operational overhead increased it by 150 per cent.
The staffing story is perhaps the most revealing. One organisation reduced their DevOps team from 12 engineers to 3 after leaving Kubernetes. Nine engineers, nine full-time salaries, had been maintaining the platform, not the product. Three-quarters of the operations team existed solely to keep the orchestration layer alive. When the orchestration layer departed, so did the need for nine people whose job was feeding it.
The Certification Tax
Kubernetes is sufficiently complex that operating it requires formal certification. The Certified Kubernetes Administrator exam costs $445 per engineer. With the recommended training course, the bill climbs to $1,950 per head. A team of five (the minimum for credible on-call rotation) costs roughly $10,000 in certification alone. Not in tooling. Not in cloud spend. In the privilege of being allowed to understand the system you have already paid for.
FreeBSD requires no certification. Neither does systemd. Neither does
rc.conf. The documentation is the certification. It is freely
available, thoroughly written, and has not changed its fundamental model in
twenty-five years. There is something to be said for technology that does not
require a training budget.
The 5 Per Cent Rule
Kubernetes makes sense. It genuinely does, under a specific set of conditions that the conference talks neglect to mention.
You need Kubernetes if you are running 50 or more services across multiple geographic regions, with a dedicated platform engineering team whose sole occupation is maintaining the cluster, and your organisation has the budget, the headcount, and the operational maturity to absorb the overhead without flinching.
That describes approximately 5 per cent of the companies currently running Kubernetes. The other 95 per cent adopted it because a conference talk made it sound inevitable, a job posting made it sound mandatory, and nobody in the room had the institutional courage to ask: “But do we actually need this?”
The Verdict
Ten per cent CPU utilisation. Twelve to twenty-four gigabytes of RAM consumed
before your first deployment. Two hundred lines of YAML to do what one
rc.conf entry accomplishes. Thirty-four workdays per year
lost to troubleshooting. Thirty-eight per cent of teams suffering weekly
outages. Nine out of twelve DevOps engineers maintaining the platform rather
than the product. Ninety per cent cost reductions when teams leave.
And beneath all of it, Kelsey Hightower, the man whose name is synonymous with Kubernetes advocacy, quietly noting that there is rather a lot of infrastructure required to do something basic.
Kubernetes was built for Google’s problems at Google’s scale with Google’s budget. You are not Google. You almost certainly never will be. And the invoice for pretending otherwise is 34 workdays, 90 per cent of your CPU, and nine engineers who could have been building the product.
The failure stories are curated and catalogued. The exit stories are published with numbers. The performance data is surveyed across thousands of organisations. The evidence is not ambiguous. It is merely inconvenient.
The question was never whether Kubernetes works. It works beautifully at Google’s scale, for Google’s problems. The question is whether your three services, your single region, and your team of eight need a platform that was designed to orchestrate the infrastructure of one of the largest distributed systems operations on Earth.
The answer, for 95 per cent of you, is a polite but firm no. And the 34 workdays you save can be spent building the thing your customers actually pay for.