On 9 November 2023, Becky Pauley and I walked onto a stage at KubeCon + CloudNativeCon North America in Chicago to deliver Kubernetes Confessions: Tales of Overspending and Redemption.
The title was playful, but the problem was not. Kubernetes gives teams a powerful way to turn demand into infrastructure. Without good feedback, it also gives small assumptions a very efficient path to becoming a large bill.
Our aim was not to promote a particular vendor or promise a magic percentage saving. It was to give engineers a practical way to understand where money was going, why apparently sensible choices accumulated waste and how to improve the system without creating fear around every deployment.
More than 360 people registered for the session. The scale of that interest confirmed something we had already seen in customer conversations: cloud cost was no longer a finance-only concern. It had become a platform design and developer-experience problem.
A confession needs psychological safety
The word confession matters because overspending is often treated as a failure of individual discipline.
A team asks for too much memory. A namespace grows quietly. A non-production environment stays online. Somebody buys a commitment before understanding the workload. Each decision is easy to criticise after the invoice arrives.
Blame is a poor optimisation strategy.
Most engineers cannot see the full cost of a change when they make it. They see a deployment manifest, an autoscaler or a request for capacity. The eventual bill is separated by accounts, discounts, shared services and allocation rules. If feedback arrives weeks later as a spreadsheet, the system has made the wrong behaviour easier than the right one.
The first act of redemption is therefore visibility—not merely another dashboard, but cost information close enough to the engineering decision to be useful.
Visibility before optimisation
A total cloud bill tells you that money was spent. It does not tell you what to change.
Useful visibility connects spend to the dimensions teams recognise:
- cluster, namespace, workload and environment;
- requested capacity versus observed use;
- shared platform services and an explicit allocation method;
- changes over time, including deployments and scaling events;
- ownership, so the people with context can investigate;
- unit economics where they are meaningful, such as cost per tenant, job or transaction.
The goal is not perfect allocation before any action is taken. It is enough shared evidence for a platform team and a workload team to have the same conversation.
That distinction prevents cost tooling becoming surveillance. The information should help a team improve its system, not rank engineers by infrastructure spend.
Right-sizing is a feedback loop
Requests and limits are one of the first places people look for savings. They are also where blunt optimisation can damage reliability.
A request is a scheduling and capacity decision. A limit can change runtime behaviour. Reducing either because a seven-day graph looks quiet is not the same as understanding the workload.
A better loop is:
- observe representative demand over a meaningful period;
- identify seasonality, batch behaviour and failure scenarios;
- compare requested, limited and actually used resources;
- propose a change with clear headroom;
- test it safely;
- watch reliability and performance as closely as cost;
- repeat.
Automation can recommend or implement some of this, but it needs guardrails. The cheapest pod is not the one that is repeatedly evicted, throttled or unable to absorb a normal peak.
Discounts cannot repair the wrong architecture
Committed-use discounts, savings plans and spot capacity can all be valuable. They solve different problems.
A commitment improves the price of predictable baseline use. It does not make idle capacity useful. Spot instances can reduce the cost of interruptible work, but only when the workload and platform have been designed for interruption.
The order matters:
- understand the workload;
- remove avoidable waste;
- establish the real baseline;
- apply the appropriate pricing model.
Buying a discount too early can turn a temporary assumption into a long-term constraint. A lower unit price is still waste if the unit should not exist.
Platform teams shape the default bill
Cost optimisation becomes sustainable when it is built into the paved road.
A platform can provide sensible initial requests, scheduled shutdown for ephemeral environments, workload ownership metadata, cost-aware scaling patterns and feedback in the tools engineers already use. It can make the efficient path easier while preserving an escape hatch for workloads with genuine exceptions.
This is why Kubernetes cost belongs in platform product thinking. The user is not asking for a cost report. They are trying to ship a reliable service within reasonable constraints.
A useful platform translates those constraints into defaults and timely feedback.
Redemption is continuous
There is no final state called optimised. Workloads change. Traffic moves. Prices evolve. A successful efficiency change can create enough headroom for new demand.
The durable practice is a review loop with shared ownership:
- finance explains the commercial model;
- platform teams improve defaults and allocation;
- workload teams bring application context;
- leaders protect reliability and sustainability from false trade-offs;
- everyone agrees which outcome is being improved.
The best result is not a dramatic one-off reduction followed by another year of drift. It is a system in which a surprising cost can be explained quickly and the next engineering decision is better informed.
From Chicago back to practice
Speaking with Becky at KubeCon made the work public, but the ideas came from messy, ordinary engineering situations: incomplete labels, uncertain ownership, over-sized requests, appealing discounts and teams doing their best without a useful feedback loop.
That is why the talk was framed around confession and redemption. Everybody has a cost story. The useful question is whether the organisation can learn from it without hiding the evidence or blaming the person who found it.
You can view the original KubeCon session listing or watch the recorded talk.
The original LinkedIn trail is also preserved through the pre-event announcement, conference update and post-event reflection.