How to reduce LLM token spend without reducing quality

Measure cost per useful outcome

A low price per token does not guarantee an efficient system. One workflow may make several model calls, retrieve excessive context, retry failed tools, repeat stable instructions, or produce outputs nobody uses. Start by attributing requests, input tokens, output tokens, cached tokens, model choice, tool calls, latency, retries, and failures to a specific workflow.

Then connect technical usage to an outcome such as a completed intake, a reviewed document, a resolved question, or a prepared proposal. Cost per completed workflow is more actionable than one blended monthly invoice.

A useful calculation is: model usage, tools, hosting and review cost divided by the number of accepted results. Include failed attempts in the costs, not in the successful-result count. A faster or cheaper model can still be the more expensive choice if people must repair its output.

Send the evidence the task actually needs

Large prompts often contain entire documents, long conversation histories, duplicated system instructions, and retrieval results that do not affect the answer. Improve chunking, metadata, filtering, ranking, and query construction so the model receives the smallest evidence set that can support the result.

Reducing irrelevant text can help, but shorter is not always better. Removing a necessary exception or source can make the answer less reliable. Compare retrieval and prompt changes against the same accepted-answer examples.

Check when prompt caching will help

Many providers can reuse cached prompt or context prefixes under specific conditions. Place stable instructions and repeated reference material in a consistent structure, separate them from request-specific content, and measure actual cache hits. Caching value depends on provider rules, traffic shape, timing, and prompt consistency.

For example, Anthropic documents cache-prefix matching, cache lifetimes and usage fields in its prompt caching guide. Measure cache writes as well as reads. A workload that rarely reuses its context may not benefit. Other providers and hosting platforms have different conditions.

Route work by difficulty

Not every step requires the most capable model. Classification, extraction, formatting, and straightforward retrieval may meet their quality threshold with a smaller or less expensive model. Complex synthesis or ambiguous exceptions can escalate to a stronger model or a person.

Routing should be based on evaluations, not assumptions. Build representative test cases and compare quality, latency, failure patterns, and total workflow cost across candidate models.

Put limits on agent loops

Agent loops can create unexpected spend when tools fail, instructions are ambiguous, or completion conditions are weak. Set maximum turns, tool budgets, retry policies, timeouts, output limits, and explicit stop conditions. Log each step so unusual cost can be traced to a cause.

Batch and defer non-urgent work

Recurring research, document enrichment, classification, and large offline processing jobs may not need an immediate response. Where a provider and workflow support it, batch or asynchronous processing can lower cost and smooth demand. This should be balanced against freshness, operational deadlines, and failure recovery.

Anthropic's batch processing documentation describes one provider's asynchronous option. Check completion windows, unsupported features, data retention and failed-request handling before using it. A batch discount does not guarantee that the whole workflow will cost less.

Keep a prioritized improvement backlog

Prioritize changes using expected savings, implementation effort, quality risk, and workflow volume. Review the largest cost centres first. Recheck after model changes, source growth, workflow expansion, or major shifts in usage. AI economics is an ongoing operating discipline, not a one-time prompt cleanup.