Most AI features are paying premium-model prices for work a cheaper model does identically. The fix is engineering, not negotiation.
LLM APIs price per token — as of mid-2026, roughly $2–3 per million input tokens for flagship workhorses like Claude Sonnet, GPT-5.4, and Gemini 3.1 Pro (with output tokens 5–6× dearer), down to $0.10–0.30 for small models. Teams overspend not because prices are high but because every request ships to the biggest model with an uncached prompt and an unbounded context. Four techniques — routing, caching, batching, and context discipline — stack multiplicatively, and together they routinely cut bills by 60–90% with no visible quality change.
Model routing is the biggest single lever. Classification, extraction, formatting, and simple Q&A run identically on models costing 10–20× less than a flagship — a small model at $0.10–1 per million input tokens against a frontier model at $2.50–5. Route by task type (or by a cheap classifier’s confidence), reserve the big model for genuinely hard reasoning, and measure with evals so quality is a fact rather than a fear.
Prompt caching attacks the other constant: repeated context. System prompts, tool definitions, and shared documents resent on every call can be cached by the provider — Anthropic discounts cached input by 90%, so a 4,000-token system prompt bills like 400 tokens on every call after the first. For chat products and agents, where the same prefix rides along with every message, caching alone often halves the bill.
Anything that doesn’t need an instant answer — nightly enrichment, bulk classification, report generation, evals — belongs on the batch API, where OpenAI and Anthropic both take a flat 50% off. Batch discounts stack with caching: async work with a cached prefix can land near a quarter of standard rates.
Context discipline is the quiet one: output tokens cost 5–6× input, so cap response lengths and prefer structured outputs over prose. Trim conversation history instead of resending everything; summarize long threads; in RAG, send the five chunks that matter rather than fifty that might. Instrument cost per request from day one — the teams with the lowest bills are the ones that can see them.
Route classification and extraction to a small model — 10–20× cheaper on those calls.
Turn on prompt caching for system prompts and tool definitions — up to 90% off repeated input.
Move non-urgent jobs to the batch API for a flat 50% discount.
Cap output lengths and switch prose responses to structured output.
Add per-request cost telemetry and per-feature budgets so regressions surface immediately.
Build a small eval set so model downgrades are verified, not vibes-based.
For frontier-hard reasoning, yes. For the classification, extraction, and templated generation that dominate most real workloads, small models score identically on evals — which is why routing is the first lever, and why you build the eval set before you downgrade.
Anthropic bills cached input at 10% of the normal rate; OpenAI and Google offer similar discounts. For apps where a large system prompt or document rides on every request, that's typically a 40–70% reduction on input spend.
Only at sustained high volume with an ops team to run GPUs well. Below that, per-token APIs — especially with routing, caching, and batching applied — are almost always cheaper in fully-loaded terms.
Only for non-interactive workloads where it's applied — batching is for background jobs and bulk processing, not user-facing chat responses, which stay on the fast path.
Prompt caching, by a wide margin, for any workload with repeated context — system prompts, retrieved documents, and conversation history that don't need to be re-processed every call.
A practical comparison of retrieval-augmented generation and fine-tuning for AI agents — cost, latency, data freshness, and when each approach actually makes sense.
Read the guideSupport volume was outpacing the team — an agent now handles the routine work, with clean human escalation for the rest.
Read the case study