The LLM Cost Optimization Guide: The Ranked Stack That Actually Moves the Bill

Most teams trying to reduce LLM costs start in the wrong place. They negotiate rates, swap providers, or buy a tool, when the bill is actually set by five levers with wildly different paybacks. This guide ranks them by typical savings, gives you the math to size each one against your own numbers, and says plainly when optimization is not worth the engineering time. It is written for CTOs and FinOps leads who own a production AI bill, especially in regulated industries where the cost lever and the compliance lever are often the same piece of infrastructure.

The context: 98% of organizations now manage AI spend as a FinOps discipline, up from 31% two years ago [7]. The bill is real, it is growing, and most of the advice on cutting it is vendor marketing. Here is the version with numbers.

First, the formula every lever plugs into

Your monthly inference cost is:

cost = sum over models of (input tokens x input price) + (output tokens x output price)

That is it. Every optimization below works by shrinking one of those four terms: fewer tokens in, fewer tokens out, a cheaper price per token, or cheaper infrastructure under the tokens. If you cannot decompose your bill this way today, by model and by feature, stop reading and fix that first. Cost per request telemetry is the prerequisite for everything else, and we wrote a separate piece on instrumenting it.

One asymmetry matters throughout: output tokens typically cost 3-5x input tokens. Levers that cut output tokens pay disproportionately.

The ranked stack: five levers, honest ranges

Ranges are synthesized from the sources cited below and from what we see in client assessments. Your distribution of traffic decides where you land in each range. The math for each lever tells you how to find out.

1. Model routing: the biggest lever for most teams

Most production traffic does not need a frontier model. Classification, extraction, summarization, and FAQ-style answers run fine on models that cost 10-15x less per token. Routing sends each request to the cheapest model that clears your quality bar for that task type.

The academic result that made this concrete: RouteLLM showed over 85% cost reduction on MT-Bench while retaining 95% of GPT-4 performance, using a trained router and cascade fallback [1]. Production numbers are lower because real traffic is messier. Treat 30-60% of model spend as the honest range.

Sizing math:

savings = premium tokens x premium price - (routed share x cheap price + unrouted share x premium price)

Worked example. You send 100M output tokens a month to a model at $15/M. All-premium cost: $1,500. You find 70% of traffic clears eval on a $1.5/M model. New cost: 30M x $15 + 70M x $1.5 = $450 + $105 = $555. That is 63% off the output side, before touching inputs.

The catch is the prerequisite column. Routing without a per-task eval gate is how you learn about quality regressions from customer tickets two weeks later. We go deeper on the mechanics in model tier routing, and the infrastructure it lives on is the AI gateway.

2. Caching: near-free money if your traffic has repetition

Two flavors. Provider prompt caching reuses the KV state of a repeated prompt prefix: Anthropic charges cache reads at roughly 10% of the base input price (with a 25% write premium and a default 5-minute TTL) [3], and OpenAI applies automatic discounts of 50-90% on cached input tokens depending on model [4]. Semantic caching goes further: if a new request is close enough to a previous one, you return the stored answer and skip the model entirely.

Sizing math:

savings = input tokens x cacheable share x hit rate x input price x discount

Example. 500M input tokens a month at $3/M. 40% of your prefix is a stable system prompt plus retrieved context. At a 60% hit rate and a 90% discount: 500M x 0.4 x 0.6 x $3/M x 0.9 = $324 saved monthly. The hit rate is the variable you measure, not the discount. OpenAI's caching is automatic and best-effort, so reported hit rates in the wild vary; instrument before you forecast.

The cheapest design change: put the stable content (system prompt, tool definitions, few-shot examples) at the front of the prompt and the variable content at the end. Caches match prefixes. A volatile timestamp at the top of your system prompt silently zeroes this lever.

3. Prompt discipline: the lever nobody wants to do

Field audits of production LLM applications claim 40-60% of token budgets are waste: redundant calls, full chat histories resent every turn, unbounded outputs, verbose system prompts [10]. That figure is a vendor estimate, so discount it. But every assessment we run finds some version of it.

The fixes are unglamorous: cap max_tokens, trim history to the turns that matter, stop re-sending resolved context, compress system prompts, and cache reference documents instead of inlining them. Because output tokens cost 3-5x input, an output cap is often worth more than any input optimization.

Sizing math is a straight subtraction: tokens removed x price. Measure your p50 and p95 output length per feature before and after. Typical outcome is 10-30% of total spend for a week of work, which makes this the best effort-adjusted lever on the list.

4. Quantization: half the GPUs, if you self-host

If you serve your own models, quantization is the largest infra lever. The most thorough public study to date, over 500,000 evaluations across the Llama-3.1 family, found FP8 effectively lossless at all model scales, well-tuned INT8 within 1-3% accuracy of full precision, and INT4 weight-only competitive for many serving setups [2]. Cutting precision halves or quarters memory, which means the same model on half the GPUs or on a cheaper GPU tier.

Sizing math: compare cost per million tokens before and after, on your hardware, at your concurrency. A 70B model that needs two 80GB GPUs at BF16 fits on one at INT4. If your utilization supports it, that is close to 50% off serving infrastructure for that model.

The gate: run your own eval suite on the quantized checkpoint before you ship it. The 1-3% average degradation hides variance on hard, domain-specific tasks, and in regulated workflows the hard tasks are the ones that matter.

5. Spot capacity: brutal discounts, batch only

AWS advertises up to 90% off on-demand for EC2 Spot [5]. GPU spot capacity is thinner and interruption rates vary by chip: one vendor's measured data puts H100 spot interruptions under 5% and A100 at 15-20% [6]. Treat those as vendor-measured figures, not guarantees.

This lever applies only to interruptible work: eval sweeps, embedding backfills, document reprocessing, fine-tuning runs with checkpoints. Never put latency-sensitive production serving on spot.

Sizing math:

savings = interruptible GPU hours x (on-demand price - spot price) - rework cost from interruptions

If you do not self-host, skip levers 4 and 5 entirely and spend the time on 1-3. The decision of when self-hosting itself beats API pricing has its own math, covered in our self-hosted LLM breakeven analysis.

The case against: maybe just wait

The strongest argument against all of this is that the market optimizes faster than you can.

Stanford's AI Index documented a roughly 280-fold drop in inference cost for GPT-3.5-level performance between late 2022 and late 2024 [8]. If prices fall an order of magnitude every year or two, an engineering quarter spent on routing tiers and cache layouts may be obsolete before it pays back. The rational move, this argument goes, is to write simple code against one good model, ride the price curve down, and spend scarce engineering time on use cases instead.

There is a sharper version for routing specifically. Research on LLM routers shows they can converge toward degenerate behavior, over-routing to cheap models as cost pressure rises [9], and production teams report the failure mode that actually bites is silent quality regression: the bill drops, quality drops with it, and nobody notices until tickets arrive. An optimization program without a standing eval budget is a slow-motion incident.

And the honest FinOps version: if your total AI bill is $5K a month, a heroic 40% reduction saves $2K. That does not cover the engineering. Below some threshold, the correct optimization is doing nothing.

Our read

The wait-it-out argument is right for small bills and wrong for the readers of this guide, for two reasons.

First, percentages become absolute dollars at scale. At $100K a month, the same 40% is a hire. The breakeven is a calculation, not a vibe: compare one engineer-month against your measured savings from the sizing math above.

Second, in regulated industries the cost infrastructure is the compliance infrastructure. The gateway that routes requests is also the chokepoint where you log them, redact them, and keep client data away from the wrong provider. The eval gate that protects routing quality is the same gate that catches the hallucination before it reaches a client. You were going to build these controls anyway. The cost savings come along for free, which flips the payback math decisively.

The one number that matters: cost per successful request, by feature, measured. Everything above is a way to move it.

What this means for you

• Measure before you optimize. If you cannot state cost per request by model and feature today, that is your only project this month. Every other number in this guide depends on it.
• Sequence by effort-adjusted savings. Prompt discipline and caching this quarter, routing next quarter with an eval gate, quantization and spot only after your bill or your sovereignty requirements justify self-hosting.
• Tier by data sensitivity, not just task difficulty. In finance, healthcare, legal, and advisory work, the routing table is a data-governance artifact. Which model may see which client data is a policy decision; encode it in the router, not in a wiki page.
• Budget eval spend as part of the savings. A routing program without continuous evals trades a visible cost problem for an invisible quality problem. Regulators and clients forgive the first, not the second.
• Re-run the math quarterly. Prices, cache discounts, and model tiers move fast enough that last quarter's optimal stack is a guess today.

Book a free assessment: systemics.ai.

References

1. "RouteLLM: Learning to Route LLMs with Preference Data." Ong et al., arXiv:2406.18665, June 2024.
2. "Give Me BF16 or Give Me Death? Accuracy-Performance Trade-Offs in LLM Quantization." Kurtic, Marques, Pandit, Kurtz, Alistarh. Red Hat AI / Institute of Science and Technology Austria, ACL 2025 (arXiv:2411.02355).
3. "Prompt caching." Anthropic Documentation, accessed August 2026. https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching
4. "Prompt caching." OpenAI Platform Documentation, accessed August 2026. https://platform.openai.com/docs/guides/prompt-caching
5. "Amazon EC2 Spot Instances." Amazon Web Services, accessed August 2026. https://aws.amazon.com/ec2/spot/
6. "Cloud GPU Spot Instances: Availability, Interruption Rates, and When to Use Them." Thunder Compute, July 2026.
7. "State of FinOps 2026." FinOps Foundation, 2026. https://data.finops.org/
8. "AI Index Report 2025." Stanford Institute for Human-Centered AI, April 2025.
9. "On the Degenerate Convergence of LLM Routers." arXiv:2602.03478, 2026.
10. "LLM Cost Optimization: Why an AI Gateway Is the Missing Layer." TrueFoundry, June 2026.