LLM cost optimization is the work of lowering what one production request costs without lowering what it is worth. FrugalGPT (Chen, Zaharia and Zou, 2023) found popular LLM APIs "with fees that can differ by two orders of magnitude." Risk differs as well. Azure OpenAI documentation states that "Prompt caching has no impact on the output content returned in the model response beyond a reduction in latency and cost," while a semantic cache, according to Azure API Management, "can surface responses that are incorrect, outdated, or unsafe for the current request." So price one request first, map each technique to the term it lowers and the risk it adds, and apply them from the safest up. LLM cost optimization is done when every technique you kept has a measured saving and an evaluation run that says quality held.
What LLM Cost Optimization Changes, Lever by Lever
As a definition, LLM cost optimization lowers the cost of a production request by attacking one of its terms (uncached input, cached input, output, retries or routing overhead) with a technique whose quality risk you have measured. FrugalGPT named the strategy types in 2023: "1) prompt adaptation, 2) LLM approximation, and 3) LLM cascade." Today they map to token budgets, to caches and smaller models, and to routing.
| Lever | What it lowers | When it works | Quality risk |
|---|---|---|---|
| Prompt (prefix) caching | Input price of a repeated prefix | Long, static prefix reused within minutes | None to the answer; misses when the prefix changes |
| Semantic caching | The whole call | Repeated questions with user-independent answers | A wrong, stale or unsafe answer served from cache |
| Model routing or cascade | Average price per request | Mixed easy and hard traffic you can evaluate | Hard requests answered by the weaker model |
| Smaller or distilled model | Price per token | Narrow, stable task with volume | Capability lost outside the task |
| Quantization (self-hosted) | GPU memory and hardware per token | You run the weights | Accuracy loss at low bit widths |
| Batch API or server batching | Price or GPU time per request | Work that can wait, or high concurrency | Latency, not quality |
| Token budget | Tokens per request | Long outputs, oversized prompts, unused tool definitions | Truncated answers or missing context |
Only prefix caching and batching leave the answer untouched; every other lever needs an evaluation run before and after.
Which team owns these levers, and when an LLM workload needs its own operations practice at all, is covered in where cost engineering sits in an LLMOps practice; this article goes one level down, to how each lever works and what it can break.
How to Price One Request Before You Optimize It
Every LLM cost optimization decision starts from the price of one request. Our formula adds uncached input, cache writes, cache reads and output tokens, each at its own price, multiplies the sum by one plus the retry rate, and adds helper calls such as a router or the embedding lookup of a semantic cache.
The inputs come from the usage fields of each response. OpenAI reports cached_tokens, and Google Cloud returns cachedContentTokenCount in the response metadata. Without these fields, a cache hit rate is a guess.
On the OpenAI and Anthropic price lists published in November 2026, output costs five times input for every model in the main table, and input prices within one provider span 80 times at Anthropic and 100 times at OpenAI. Answer length and model tier outweigh most prompt micro-tuning.
Report the number per feature and per tenant, since a monthly total hides which endpoint is expensive. Databricks documents the mechanism: "Identify LLM spend drivers by attributing cost to users, teams, or projects with service tags and request tags." A model gateway does the same through token accounting and cost attribution per feature and tenant. Cost per request is an engineering number. How it rolls up into the budget of a use case or a whole program is covered in how per-request cost rolls up into an AI program budget.
Prompt Caching and Semantic Caching: Two Caches, Two Risks
The first LLM cost optimization move is usually prefix caching, because it cannot change the answer. In OpenAI's words, "Prompt caching preserves that state for a reusable prefix: the unchanged tokens at the beginning of a prompt." vLLM offers the same for self-hosted models as automatic prefix caching.
The price has two parts. Anthropic bills cache writes at 1.25 times the base input price and reads at 0.1 times on most models, so its 5-minute cache pays back after one read. OpenAI shows the arithmetic: "Across ten requests, one write and nine full reads cost 2.15× at that rate, compared with 10× without caching." Google gives a 90% discount on implicitly cached tokens for Gemini 2.5 and later.
A hit needs a long enough prefix (Amazon Bedrock minimums run from 512 to 4,096 tokens), a repeat within the 5-minute default lifetime at Anthropic, and identical content, so OpenAI advises placing timestamps and user-specific content "at the end rather than the beginning." Cacheability is a prompt layout decision.
Prefix caching also stops at prefill, as the vLLM documentation puts it: "APC only reduces the time of processing the queries (the prefilling phase) and does not reduce the time of generating new tokens (the decoding phase)." Output stays at full price.
Semantic caching returns a stored answer for a similar question and skips the call. The GPT Semantic Cache preprint (Regmi and Pun, 2024) reports "cache hit rates ranging from 61.6% to 68.8%" and "positive hit rates exceeding 97%" on customer-service queries. The price of that saving is the Azure warning quoted above. We use a semantic cache only where the answer depends on neither the user nor the date, partitioned per tenant, with the similarity threshold set on a test set.
Model Routing and Cascades: Cheap by Default, Expensive on Evidence
Routing is the LLM cost optimization lever with the largest reported savings and the most exposure to quality loss. FrugalGPT's cascade "can match the performance of the best individual LLM (e.g. GPT-4) with up to 98% cost reduction," on the authors' datasets and 2023 models.
RouteLLM (Ong et al., 2024) trains routers "that dynamically select between a stronger and a weaker LLM during inference." On public benchmarks they cut cost by more than half in some settings without lowering response quality, and kept working when the model pair changed. Hybrid LLM (Ding et al., 2024) reports "up to 40% fewer calls to the large model, with no drop in response quality."
Amazon Bedrock intelligent prompt routing is a managed option in which "The fallback model acts as an anchor." AWS lists its limits: "Intelligent prompt routing is only optimized for English prompts," and it "can’t adjust routing decisions or responses based on application-specific performance data."
Our condition for any router is an evaluation set drawn from your own traffic. Without it, routing moves cost from the invoice to quality. We score each path separately and count the router's own calls in the per-request formula.
Right-Sizing: A Smaller Model, a Distilled One, or Quantized Weights
Right-sizing covers three moves with different arithmetic. A smaller model from the price list lowers the price per token. Distillation, in the Amazon Bedrock definition, transfers knowledge "from a larger more intelligent model (known as teacher) to a smaller, faster, cost-efficient model (known as student)." It pays off on a narrow, stable task with volume, and you own the training job. Quantization applies only when you host the weights.
GPTQ (Frantar et al., 2022) quantizes models with 175 billion parameters to "3 or 4 bits per weight, with negligible accuracy degradation relative to the uncompressed baseline." AWQ (Lin et al., 2023) found that "Protecting only 1% salient weights can greatly reduce quantization error." LLM.int8() (Dettmers et al., 2022) cut "the memory needed for inference by half while retaining full precision performance." The saving is hardware per token, while the API price stays put, and "negligible" is a benchmark average, so we rerun the evaluation on the client's task.
Batching and Token Budgets: Pay Less for Work That Can Wait or Shrink
Batching is the second LLM cost optimization lever that leaves the answer untouched. The OpenAI Batch API gives a "50% cost discount compared to synchronous APIs," and "Each batch completes within 24 hours (and often more quickly)." The Anthropic Batch API cuts input and output prices by 50 percent, and Anthropic's cache multipliers "stack with other pricing modifiers, including the Batch API discount."
Self-hosted servers batch internally; the PagedAttention paper behind vLLM (Kwon et al., 2023) reports two to four times higher throughput at the same latency.
A token budget is the price side of prompt design. Cap output first, since it costs five times input. Watch thresholds: Anthropic's November 2026 price list charges Claude Haiku 5.5 more for prompts above 100,000 tokens. Load tool definitions on demand; Anthropic described an agent whose usage fell from 150,000 to 2,000 tokens that way. Which tokens deserve a place in the context is a design question, covered in deciding which tokens earn a place in the context; the budget here is about what the remaining tokens cost and where the price changes.
The Order We Apply Them, and How We Measure the Result
The order we use for LLM cost optimization follows risk, not the size of the promised saving.
- Measure cost per request for each endpoint from the usage fields.
- Turn on prefix caching and move static content to the front of the prompt.
- Send everything that can wait to a batch API.
- Set token budgets: output caps, prompt length thresholds, tool definitions.
- Add routing or a smaller model, each behind an evaluation set.
- Quantize or distill once volume is stable.
- Add a semantic cache last, only for answers that do not depend on the user.
At DS Stream we start with one endpoint and its usage logs, price a request term by term, and introduce techniques one at a time, measuring cost and quality before and after each. To start, bring us one endpoint's usage logs and last month's bill.
FAQ
How do you calculate LLM cost per request?
LLM cost optimization needs a per-request number built from the usage fields of each response: uncached input tokens, cache writes and cache reads at their own multipliers, and output tokens, which cost five times input on the OpenAI and Anthropic price lists published in November 2026. Multiply by one plus the retry rate and add any router or embedding calls.
Does prompt caching lower the cost of output tokens?
No. A prefix cache skips recomputing the shared beginning of a prompt, which lowers the price of those input tokens; vLLM's documentation states that it does not reduce the time spent generating new tokens. Long answers stay expensive even at a high cache hit rate.
When is model routing worth the extra complexity?
When traffic mixes easy and hard requests and you have an evaluation set drawn from that traffic. Research routers such as RouteLLM and Hybrid LLM sent a large share of queries to the smaller model without measurable quality loss on their benchmarks, but a router tuned on someone else's data can misjudge yours.
Should you quantize a model or switch to a smaller one?
It depends on who runs the weights. If you call an API, quantization is not your lever; a smaller or distilled model changes the price per token. If you host the model, 8-bit and 4-bit methods such as LLM.int8(), GPTQ and AWQ cut the memory each replica needs, so the same traffic runs on fewer or cheaper GPUs. In both cases the decision rests on your own evaluation set, not on benchmark averages.
