LLM Cost Levers, Ranked by Typical Impact
The five levers that move an LLM bill, honest savings ranges, a routing table template, and the breakeven math for when fine-tuning beats prompting.
The formula
cost = sum over models of (input tokens x input price) + (output tokens x output price)
Every lever shrinks one of those four terms. Output tokens typically cost 3-5x input tokens, so output-side cuts pay disproportionately.
The levers, ranked
| Rank | Lever | Typical savings | Applies to | Prerequisite | Main risk |
|---|---|---|---|---|---|
| 1 | Model routing | 30-60% of model spend | Anyone on premium models | Eval gate per task type | Silent quality regression |
| 2 | Prompt and semantic caching | 20-50% of input spend | Repeated prefixes, RAG, multi-turn | Cache-aware prompt layout | Stale or leaked cached context |
| 3 | Prompt discipline | 10-30% of total spend | Everyone | Nothing | Output truncation bugs |
| 4 | Quantization | 40-60% of serving infra | Self-hosted only | Quality eval on your tasks | Accuracy loss on hard tasks |
| 5 | Spot capacity | 60-90% on eligible batch | Self-hosted batch only | Checkpointing and retries | Mid-job interruptions |
Ranges are synthesized from published sources and assessment work; your traffic distribution decides where you land. Order by effort-adjusted payback: 3 and 2 this quarter, 1 next quarter, 4 and 5 when self-hosting is already justified.
Prompt discipline (do this first)
- Cap
max_tokensper feature. An output cap is often worth more than any input optimization. - Trim chat history to the turns that matter; stop resending resolved context.
- Compress system prompts. Every line costs on every call.
- Cache reference documents instead of inlining them per request.
- Measure p50 and p95 output length per feature before and after. Typical outcome: 10-30% of total spend for a week of work.
Caching facts (as of mid-2026; verify against current provider docs)
| Provider | Mechanism | Pricing shape |
|---|---|---|
| Anthropic | Explicit cache control on prompt blocks | Cache reads around 10% of base input price; writes carry a premium; default TTL 5 minutes |
| OpenAI | Automatic prefix caching | Cached input tokens discounted roughly 50-90% depending on model; best-effort, so measure hit rate before forecasting |
Layout rule: stable content first (system prompt, tool definitions, few-shot examples), variable content last. Caches match prefixes; a volatile timestamp at the top of the prompt zeroes the lever.
Routing table template
| Task type | Quality bar | Tier | Why |
|---|---|---|---|
| Classification, extraction | Deterministic eval | Small/cheap | 10-15x cheaper per token, clears eval |
| Summarization, FAQ | Eval plus spot checks | Mid | Quality insensitive at this length |
| Client-facing drafting, analysis | Human rubric, sampled | Frontier | Quality is the product |
| Regulated data (any task) | Policy | Approved tiers only | Routing is a data-governance artifact |
Rules: routing lives in the gateway, not in app code; every tier change passes an eval gate; the fallback chain is configured and has carried traffic at least once.
When fine-tuning beats prompting on cost
Breakeven: fine-tune when monthly prompt-overhead cost exceeds monthly fine-tune cost.
monthly prompt overhead = extra few-shot tokens per call x calls per month x token price
monthly fine-tune cost = training amortized over model lifetime + any serving premium
Fine-tuning tends to win when all of these hold:
- One stable task shape (classification, extraction to a fixed schema, a house style).
- High, steady call volume on that shape.
- A few-shot prompt long enough that its tokens dominate the call.
- A maintainable training set in the hundreds to low thousands of good examples.
Prompting wins when the task changes weekly, volume is low, or the dataset cannot be maintained. The end state is a fine-tuned small model behind the same gateway interface: the routing rule changes, the app code does not.
Quick checks
- [ ] Cost per successful request by feature: measured, not estimated.
- [ ] p50 and p95 output length per feature; caps set.
- [ ] Cache hit rate measured before savings are claimed.
- [ ] Eval gate runs before any routing change ships.
- [ ] The bill reconciles with attribution data weekly.
Book a free assessment: systemics.ai.
Start with a free architecture assessment.
Thirty minutes, no slides. Tell us where your AI posture stands and we will tell you honestly whether we can help.