Token Cost Management Will Be the New Cloud Cost Management

Cloud cost management has been one of the big operational challenges of the past decade. Entire practices and disciplines have formed around taming runaway cloud bills, aligning spend to value, and squeezing efficiency out of infrastructure. But as AI models become core to business workflows, a new cost class is developing fast: token cost management.

In the same way that cloud costs once blindsided organisations with their scale and complexity, token costs are now sneaking up on businesses too. As I talk to customers about AI use cases, almost no one is considering the impact of token volumes and costs. And just like with cloud in the past, those that get their heads around it faster will have a serious advantage.

Why tokens are the new compute

In AI, the meter isn’t CPU cycles or storage space, it’s tokens. Every time you send text to an LLM, you pay for each input token. Every time it responds, you pay for output tokens. Multiply that across summarisation, Q&A, agents chaining calls together, and suddenly your budget is being eaten line by line, word by word.

As a result, this cost profile is fundamentally different from traditional infrastructure. It’s fine-grained, volatile, and in many cases very unpredictable. And because LLMs are now embedded in production systems, the spend quickly becomes material. It's not monopoly money.

The opacity problem

If cloud costs were tricky to predict, token costs raise the bar. The length of a response, the way a prompt is engineered, the decision to include historical context or not, all of these can double or halve your bill. Different models and vendors price tokens differently. Input and output rates vary. Even subtle choices, like formatting or over-engineering a system prompt, can lead to spiralling consumption. I tell customers, don't think of it as tasks, think of it as actions. It might sound like nuance but there can be many actions involved in the execution of one task. And even within actions there are multiple tiers, because it depends on the volume of information that you are passing to the LLM, or requesting back.

That opacity is what makes this a governance issue as much as a technical one. Finance, product, and engineering leaders will want visibility and control. And rightly so.

Borrowing lessons from cloud cost management

The good news is we don’t have to start from scratch. The discipline of cloud cost management has already given us a good starting point:

• Rightsizing becomes prompt pruning: trimming unnecessary instructions, reusing existing solutions / formats that you know work.

• Autoscaling becomes model routing: cascading from cheaper, faster models to larger ones only when absolutely necessary.

• Reserved instances become caching: storing results or partial computations to avoid paying for the same answer twice.

• Tagging and chargeback become attribution: logging token use by feature, by team, by customer segment.

• FinOps culture becomes TokenOps: cross-functional ownership of usage, spend, and optimisation.

These patterns can be applied almost like-for-like, they just need to be slightly adapted to the AI world.

Competitive advantage lives in efficiency

There’s another reason token cost management matters: efficiency is a differentiator. If your competitor delivers the same value while using half the tokens, they can offer better margins, more attractive pricing, or reinvest savings elsewhere. In practice, this comes down to prompt design, model choice, caching strategies, and routing logic.

The winners won’t just build powerful AI products, they’ll also be build efficient ones.

Where to start

For most organisations, the first step is simple: measure it. If you don’t already capture token usage metadata (feature, user, prompt version, model type), start now. Without observability, there’s no optimisation.

Next, experiment: prune prompts, test model substitutions, introduce caching. Small tweaks often deliver disproportionate results. From there, build governance. Introduce budgets, alerts, quotas, so spend is no longer completely opaque but a managed variable.

The next FinOps

Cloud cost management quickly became a board-level conversation. The same will happen with token cost management. As AI spend grows, organisations will demand accountability, forecasting, and optimisation. That's only natural.

Those that treat it seriously will avoid the same mistakes of the cloud era. Those that don’t will find themselves with AI projects that work technically but collapse financially.

I can't stress enough, token cost management isn’t just a side concern that you deal with in the future. It will be as central to enterprise AI as cloud cost management was to the last decade of digital transformation.