LLM cost optimization comes down to five moves, in this order: measure what each feature spends, trim the tokens you send, cache what repeats, route each task to the cheapest model that passes your quality bar, and test every change against a fixed dataset before it ships. Skip the measuring and the testing, and you are guessing. This guide walks through each move with a worked cost example, the tradeoffs, and the places where savings quietly turn into quality problems.
It is written for owners, operations leaders, and IT decision-makers who already have an LLM feature in production, or are about to, and want the bill to stay predictable without building a large internal AI team.
TL;DR
- Output tokens usually cost several times more than input tokens, so shortening answers often saves more than shortening prompts.
- Tag every API call by feature and key first. You cannot cut a cost you cannot attribute.
- Caching stable context, capping output, batching non-urgent work, and routing simple tasks to smaller models are the four levers with the best risk-to-reward ratio.
- Every downgrade or routing rule needs a before-and-after test on the same frozen dataset, with quality, latency, and error rates reported next to cost.
- Headline claims of 70% or 90% savings describe specific workloads. Expect your own number to come from your own traffic.
Table of Contents
- Where LLM spend actually comes from
- Step 1: Measure before you manage
- Step 2: Pull the five cost levers
- Step 3: Prove the savings did not cost you quality
- What the big savings claims leave out
- Costs that hide outside the token line
- Our perspective
- Getting a second set of eyes
- A 30-day plan you can start this week
- FAQ
- Sources
Where LLM spend actually comes from
Most API-based LLM bills are token bills. Providers quote prices per million tokens, with separate rates for input, output, and, where offered, cached input. As one example, OpenAI's pricing page lists a standard short-context tier at $10 per million input tokens, $50 per million output tokens, and $1 per million cached input tokens, with a 30-minute cache write at $12.50 per million. Rates differ by model and by context length, so always read the current page for the model you actually run.
Two details from that pricing structure drive everything else:
- Input and output are priced differently. TokenRate makes the point that tracking has to count them separately, because a blended token number hides where the money goes.
- Repeated context is the cheapest thing to fix. If every request begins with the same instructions, policy text, or reference material, you are paying full price for words the model has effectively already seen.
There are also costs that never show up on the token line: engineering time spent babysitting prompts, retries after failures, and the cost of a wrong answer reaching a customer. Optimization that ignores those is just cutting quality.
A worked example
Suppose a support-triage feature handles 10,000 requests a day. Each request sends about 2,000 input tokens (1,500 of them a fixed instruction and policy block) and gets back about 500 output tokens. Using the rates above purely as an illustration:
| Scenario | Daily input cost | Daily output cost | Daily total |
|---|---|---|---|
| Baseline, no caching | $200 (20M tokens at $10/M) | $250 (5M tokens at $50/M) | $450 |
| Fixed block cached at $1/M | $65 ($15 cached + $50 uncached) | $250 | $315 |
| Cached, and output trimmed from 500 to 300 tokens | $65 | $150 | $215 |
The arithmetic is simple, but the lesson is not. Output was 20% of the tokens in the baseline and 56% of the cost. And the caching row assumes every request hits the cache, which real traffic will not do, so treat it as a ceiling, not a forecast. Cache writes also carry their own price, so a prefix that changes often can cost more than it saves.
Step 1: Measure before you manage
Most teams reach for a cheaper model first. Start with attribution instead.
The FinOps Foundation's guide to a generative AI cost and usage tracker recommends recording the actual input and generated-token counts for every API call and tying them to the request key and use case. Separate keys per application, or one common interface that tags each call, make that attribution possible.
In practice, log at least these fields on every call:
- Feature or workflow name (for example, "invoice-extraction" or "intake-summary")
- Model and provider
- Input tokens, cached input tokens, and output tokens
- Cost per request, calculated from the current rate card
- Latency, and whether the call errored or fell back to another model
One vendor playbook from INFRO describes grouping cost by model, project, key, provider, or route, with feature tags or user IDs as filterable metadata. You can build the same view with a thin wrapper in your own code. Databricks lists tail latency, time to first byte, error and fallback rates, and tokens and cost per request as the operational metrics worth watching alongside quality.
The result is usually humbling. One or two workflows account for most of the spend, and one of them is often a feature nobody remembers shipping. If you want the tracing layer to go deeper than cost, our guide to LLM observability with session tracing and OpenTelemetry covers how to instrument it.
Step 2: Pull the five cost levers
Once you can see spend by workflow, work through the levers below in order of risk, lowest first.
Trim the context you send
The cheapest token is the one you never send. Common culprits are full conversation histories resent on every turn, whole documents pasted when two paragraphs matter, and long few-shot example lists. The FrugalGPT study describes two prompt adaptation techniques: prompt selection, which uses fewer examples, and query concatenation, which batches several queries into one prompt. Both cut prompt token costs roughly in proportion to what they remove.
For retrieval-augmented generation (RAG), the same logic applies: retrieve fewer, more relevant chunks rather than everything that loosely matches. For long conversations, summarize older turns instead of replaying them. Amnic's FinOps playbook lists these same tactics: caching repeated context, retrieving fewer relevant chunks, summarizing older turns, and constraining retries and output.
The risk: trimming can remove the one sentence the model needed. Test it (see Step 3).
Cache what repeats
Prompt caching rewards a stable prefix. If your system prompt, policy text, or tool definitions come first and stay identical, the provider can bill that portion at the cached rate. Runtype notes that some providers charge a premium for cache writes, which is why the prefix has to be reused enough to pay that back.
Caching matters most for agents. An academic evaluation of major providers, "Don't Break the Cache", reports API cost reductions of 45 to 80% for multi-turn agentic tasks that store recurring context. AWS similarly points to agent loops, conversations with history, and batch runs that resend an identical opening block as the natural fits.
Two practical rules follow. Put stable content first and variable content last, because a changed token early in the prompt can break the cached prefix. And do not put timestamps, user names, or request IDs at the top of the prompt.
You can also cache at a higher level than the prompt. In the SemanticALLI research, caching intermediate checkpoints of an analytics pipeline produced an 83.10% hit rate on one stage and bypassed 4,023 LLM calls. The authors' argument is that agent pipelines often rebuild identical intermediate reasoning, so caching at logical milestones beats caching only whole prompt-and-answer pairs.

Cap and shape the output
Because output tokens cost more, they deserve their own attention. Practical controls include:
- A maximum output length on every call
- Structured output with a JSON schema, so the model returns fields instead of paragraphs
- Stop sequences that end generation once the answer is complete
- Instructions that skip preambles and pleasantries, which one practitioner write-up counts as wasted tokens
- Hard limits on retries and agent loop iterations, so a stuck agent cannot spend all afternoon
Loop caps are the most overlooked. An agent that retries a failing tool call twenty times is a billing event, not just a bug. Our piece on AI agent monitoring in production covers alerting on exactly this kind of runaway behavior.
Batch the work that can wait
Not every call needs an answer in two seconds. Overnight document classification, weekly report generation, and backfills can often run asynchronously. OpenAI's Batch API processes requests within 24 hours at 50% savings on both input and output. If a quarter of your volume is non-urgent, moving it to batch is one of the safest savings available, since the model and the prompt do not change.
Route each task to the right-sized model
Model routing sends simple tasks to smaller, cheaper models and reserves the large model for hard ones. Runtype recommends exactly this split, with the caveat that you evaluate results for each workload. The FrugalGPT paper explores the idea of cascading across models for the same reason: many queries do not need the most capable system.
Routing is where savings are largest and where quality risk is highest. Start with a task you can grade objectively, such as classification, extraction, or formatting, before routing anything customer-facing or judgment-heavy. Microsoft's guidance on evaluating a model router notes the tradeoffs are workload-specific: high-volume tasks may tolerate small quality regressions, while high-impact ones may not.
Step 3: Prove the savings did not cost you quality
A cost cut that makes answers worse is not a saving. It is a deferred complaint. Use a simple protocol:
- Freeze a dataset. Pull a few hundred real, representative inputs from production (with sensitive data handled according to your policies) and keep them fixed.
- Score the current setup. Run the existing model and prompt with a defined rubric, whether automated scorers, human review, or both. Databricks advises evaluating the candidate on the same frozen dataset with the same scorers as the current model.
- Run the candidate. Apply the change (shorter prompt, smaller model, new routing rule) and score it identically. Arize recommends comparing baseline and candidate workflows on the same dataset against the same quality criteria.
- Compare side by side. Put pass rate, cost per request, latency, and error rate in one table. Runtype suggests running both variants on production-derived inputs with a fixed rubric and human review before any downgrade.
- Ship gradually and watch. Roll out to a slice of traffic, with a one-step rollback, and monitor the same metrics live.
Decide your acceptable regression in advance, by workflow. Losing two points of accuracy on tagging internal tickets may be fine. Losing two points on a legal intake summary may not be. That is a business decision, and it belongs with the person who owns the outcome, not only with the engineer tuning the prompt. For higher-stakes outputs, keep a person in the loop; our practical guide to human-in-the-loop AI explains where to place checkpoints.
What the big savings claims leave out
Pages ranking for this topic promise reductions anywhere from 40% to 90%. Those numbers are not fabricated so much as unportable. They tend to describe a particular workload with heavy repetition, a long stable prefix, and tasks easy enough for a small model. A few conditions usually sit behind a large number:
- A high share of traffic shares a stable prefix, so caching applies to most calls.
- A large fraction of requests are simple enough for a cheaper model.
- The starting point was wasteful, such as unbounded output or full history resent each turn.
- The savings were measured against list price at one moment, before rate cards changed.
If your workload has none of those traits, expect less. The sensible way to forecast is to take your tagged usage from Step 1, apply each lever only to the traffic it plausibly touches, and sum the result. The worked example above does this in miniature: caching touched only the fixed block, and output trimming touched only output.
Also weigh savings against the cost of the work. A two-week engineering project to save $300 a month is not optimization. Our 90-day AI ROI framework and one-page scorecard is built for that kind of tradeoff conversation with finance.
Costs that hide outside the token line
A few expenses sit beside the API bill and deserve the same scrutiny:
- Security incidents. Cutting corners on prompt handling can expose data. See our checklists on LLM data leakage attack surfaces and prompt injection defense before you add a cheaper pathway that bypasses your existing controls.
- Governance gaps. Unowned agents with open-ended budgets tend to multiply. A spending cap per agent and a named owner are cheap governance. The broader baseline is in our AI agent governance guide.
- Scope creep. Pilots that never reach production still bill. If your back-office pilots are stalling, the pilot-to-scale guide for SMB back-office AI is a better place to start than a pricing spreadsheet.
Our perspective
We see LLM cost as a design problem that shows up as a billing problem. When a small company's AI bill surprises it, the cause is rarely the model's list price. It is an agent with no loop limit, a prompt that grew by accretion, or a feature nobody measured. Our order of operations reflects that: attribute spend, fix the structural waste, then change models, and only with a test that the business owner has signed off on.
We also push back on optimizing too early. A pilot that has not proven value does not need a routing layer. A workflow that has proven value deserves monitoring, rollback, and human-in-the-loop checkpoints from day one, and cost controls are part of that same production discipline rather than a separate project. Rank opportunities by ROI and risk, not by which technique sounds cleverest. If you are still deciding where agents belong at all, the agentic AI roadmap for small and midsize businesses lays out how we sequence it.
— jaras
Getting a second set of eyes
If your team has an LLM feature in production and no clear picture of what each workflow costs, that is the right moment for an outside review. We start with a free technology assessment, turn the findings into a prioritized, plain-language plan that you own, and then deliver and run it if you want us to. For AI work, that means finding where agents create leverage, ranking it by ROI and risk, and shipping with monitoring, rollback, and human checkpoints. You can see how we work on our AI governance page, or book a free technology assessment with Mindpod Technologies.

A 30-day plan you can start this week
Week 1: Instrument. Tag every call by feature and key. Record input, cached, and output tokens and cost per request. Rank workflows by spend.
Week 2: Take the safe wins. On the top two workflows, add output caps and loop limits, move stable instructions to the front of the prompt, and shift non-urgent work to batch processing.
Week 3: Build the test set. Freeze a representative dataset and rubric for the top workflow. Score the current setup.
Week 4: Try one routing or trimming change. Run the candidate against the frozen set, compare cost, quality, latency, and errors, and ship to a small slice with rollback ready. Then repeat on the next workflow.
By the end, you will know your cost per workflow, your first measured savings, and whether a bigger architectural change is worth the effort.

FAQ
How much does one million tokens cost?
It depends on the model, the direction, and the context length. One current example: OpenAI's pricing page lists a standard short-context tier at $10 per million input tokens and $50 per million output tokens. Smaller models cost less and premium models more, so check the rate card for the exact model you use.
Which LLM is the most cost-effective?
The cheapest model per token is not automatically the cheapest per correct answer. A small model that fails a task and needs retries or human cleanup can cost more than a larger one that passes first time. Compare cost per successful outcome on your own frozen dataset, workflow by workflow.
What should I optimize first?
Start with measurement, then the lowest-risk levers: output caps, loop limits, prompt caching, and batch processing for non-urgent work. Model downgrades and routing come after, because they carry the most quality risk and need the most testing.
Can I cut LLM costs without hurting quality?
Often yes, but only if you verify it. Compare the old and new setups on the same dataset with the same scoring, and include latency and error rates, not just accuracy. Tolerance for regression should be set per workflow: a high-volume internal task can often absorb more than a customer-facing or high-impact one.
Sources
- OpenAI API pricing
- OpenAI Batch API pricing
- TokenRate: Tokens to Dollars
- FinOps Foundation: How to Build a Generative AI Cost and Usage Tracker
- INFRO: The production LLM cost optimization playbook
- Databricks: Metrics to evaluate new AI models
- FrugalGPT (arXiv)
- Amnic: How to Optimize LLM Cost
- Runtype: LLM cost optimization without quality loss
- Don't Break the Cache (arXiv)
- AWS: Amazon Bedrock Intelligent Prompt Routing and prompt caching
- SemanticALLI (arXiv)
- aiexpert.ee: Cost-optimizing inference
- Arize: How to reduce LLM costs without sacrificing quality
- Microsoft Foundry: Evaluate model router for your workload
