LEARN / PROMPT CACHING

Prompt caching: the 90% discount most AI teams never claim

Every major LLM provider will sell you the same tokens at a fraction of the price if you just re-send them in the right order. It's called prompt caching, it's sitting in the pricing page, and most teams are paying full fare anyway.

What it is

When your requests share a common prefix — a long system prompt, tool definitions, a document — the provider can process that prefix once, cache the result, and serve it back on subsequent requests. Cached input typically bills at around a tenth of the normal input price, in exchange for a small write premium the first time (commonly ~1.25x for short-lived caches). The exact rates and time-to-live vary by provider — check their pricing page — but the shape is the same everywhere.

The math that should get your attention

Say your product sends a 2,000-token system prompt with every request, 1,000 times a day, on a $3/MTok-input model. Uncached, that prefix alone costs ~$6.00/day — $180/month for the same bytes, re-read daily like a goldfish. Cached at ~0.1x, the same traffic costs ~$0.66/day plus a few cents of cache writes. Same model, same prompts, ~90% off. The bigger your shared context (few-shot examples, RAG documents, codebases), the more absurd the uncached number gets.

The fine print

• — It's a prefix match. Caching works on the exact bytes from the start of the request. Any change early in the prompt invalidates everything after it.
• — Caches expire. Typical TTLs are minutes (with paid options for longer). Bursty traffic keeps the cache warm; sparse traffic re-pays the write.
• — There's a minimum size. Very short prefixes often aren't cached at all — silently.

The silent cache-breakers

These are the classic ways teams pay full price while believing they cache:

• — A timestamp (“Current date: …”) interpolated at the top of the system prompt. One byte changes, the whole cache dies, every request.
• — Non-deterministic serialization: JSON keys in random order, tool lists that shuffle between requests.
• — Per-user IDs early in the prompt, guaranteeing zero cross-user reuse.
• — Swapping models or tool sets mid-conversation (caches are scoped per model).

The fix is boring and effective: keep stable content first, volatile content last, and check your provider's cache_read usage fields. If they read zero on repeated traffic, you're donating margin.

When it matters

Caching is the highest-leverage optimization for anything that re-sends context: chatbots with long system prompts, agents re-reading the same repository, multi-step agent loops, RAG over a stable corpus. For one-off prompts it does nothing — there's no prefix to reuse. Know which kind of traffic you have: measure a representative prompt and multiply by your real volume.