Two Kinds of LLM Caching, and When Each One Actually Fires

There are two things people call "LLM caching" and they solve different problems. One is a knob the vendor exposes on the model API — a hint that says "reuse the attention state for this prefix." The other is a Redis you put in front of the LLM so identical requests never hit the model at all.

They look interchangeable in a slide. They aren't. I've measured both. This writeup collects what I learned from three passes: OpenAI's automatic prefix cache on a job-matching agent, AWS Bedrock's cachePoint marker across Mistral and Nova, and the response-level cache I explicitly chose not to build for a batch document-evaluation service.

The pattern that keeps showing up: the mechanism is real, but the conditions that make it fire are narrower than the marketing suggests, and the only way to know whether it's working is to log the usage counters and stare at them.

Before the war stories, a page of theory. The whole reason prompt caching exists, and every one of its quirks, comes from a single implementation detail inside every transformer LLM: the KV cache. Skip Part 0 if you already know how attention prefill works; the rest of the article won't teach it.

---

Part 0 — Why prefix caching works: KV cache in one page

Every transformer attention layer computes three projections of each token, called Q (query), K (key), and V (value). Attention pairs the current position's Q against every earlier position's K to decide "how much should I attend to that past token", then blends the corresponding V vectors with those weights into the output. Q is what a position asks for; K is what each past position advertises; V is what each past position hands back.

That role split is why only two of the three get cached. When you generate token n+1:

• You need K and V for every earlier token (positions 1..n). These are reused every step, unchanged, forever — the K/V for position 42 is the same whether you're generating token 100, 500, or 5000. Cacheable.
• You need Q only for the current position (n+1). It's a one-shot computation that dot-products against the cached K/V, then gets discarded. Position n+2 will need its own new Q against the same K/V. Nothing to cache — Q is never reused.

So the KV cache is exactly the two projections that get read repeatedly. Q is recomputed fresh each step, but it's tiny (one token's worth) and never appears twice.

Naïvely, generating a 500-token response over a 5000-token prompt would recompute K and V for every prior token at every step: 500 × 5000 = 2.5M redundant projections. Transformers avoid this with the KV cache: after computing K and V for a token, keep them in GPU memory. When the next token needs them, read from cache instead of recomputing. Every production LLM inference server does this — vLLM, TGI, TensorRT-LLM, Bedrock's own backend, all of them.

The inference call splits cleanly into two phases:

1. Prefill: run the whole prompt through the model in parallel to compute K and V (and Q, momentarily) for every prompt token. K and V get stored; Q is used to produce the first output token, then discarded. This phase is compute-bound — a matmul over the full prompt length — and it's where TTFT (time to first token) is spent.
2. Decode: generate output tokens one at a time. Each step computes only one new Q (for the new position), one new K, one new V; appends the K/V to the growing cache; reads the whole cache to compute attention. This is memory-bandwidth bound and much cheaper per token than prefill.

Now the trick that makes prefix caching possible: the K and V of a token depend only on the tokens before it, never on tokens after. Autoregressive attention is causal by construction. So if two different requests start with the same 2000-token prefix and diverge at token 2001, the K and V for the first 2000 tokens are byte-identical in both requests. (The Q vectors would also be identical, but nobody bothers caching Q — each is used once and thrown away, whether the prefix is cached or not.)

If the inference server keeps those 2000 tokens' K/V around after request A finishes and request B arrives with the same prefix, it can skip the prefill for those 2000 tokens entirely — load the cached K/V, run prefill only over the new suffix (which does need to compute fresh Q vectors, one per new position, to attend against the cached K/V), then decode as usual.

That's it. That's the whole mechanism. "Prompt caching" is nothing more than "don't evict the KV cache between requests, and reuse it when the next request shares a prefix." Every characteristic you'll see below drops out of this:

Two implications worth internalising before you build anything:

• The cacheable region always starts at token 0. You can't cache the middle of a prompt while leaving the start dynamic. This is why every provider makes you put dynamic content (user question, request timestamp, batch item) at the end of the prompt.
• The cache is a physical thing living in specific GPUs. It's not a logical service. Anything that changes which GPU handles your request — autoscaling, deployment, a routing shuffle — can cold-cache you. TTL isn't the only source of misses.

Everything in Part 1 is either using this mechanism (cachePoint says "please keep the KV blocks for the prefix up to here") or working around one of its constraints (prompt_cache_key says "please route me to the same GPU as last time").

---

Part 1 — Prompt cache in practice

Every vendor's prompt-caching API is a thin wrapper over the same underlying KV-cache mechanism from Part 0. What differs is how you ask the server to keep the blocks around, and how the pricing surfaces:

Two very different vendors, one shared reality: you have to make sure the prefix is long enough, stable enough, and hit often enough — or the marker does nothing and no one tells you.

Primary sources for the numbers in this table: OpenAI prompt-caching guide, AWS Bedrock prompt-caching guide.

The rest of Part 1 is two war stories that make those three words concrete.

1.1 — OpenAI: the test that reported 0% whether or not the cache worked

The workload was a job-matching agent. It pulls postings, scores each against a stored resume, and surfaces the good ones. Scoring runs in batches of 8, and each LLM call is shaped like this:

SystemMessage : <scoring rubric>                    ← static
HumanMessage  : CANDIDATE: <candidate brief>        ← static, from stored resume
                === JOBS TO SCORE (8) ===
                ### JOB id=0 ...                    ← dynamic, new every batch

The first two blocks never change between batches. About 1024 tokens of static prefix out of ~5500 total per call — a natural cache target.

Making the number visible first. Before optimising anything I wanted the hit rate in logs. LangChain surfaces it a couple of levels down on the response:

def score_batch(llm, system_prompt, candidate_brief, jobs):
    job_blocks = [f"### JOB id={i}\n{_job_brief(job)}" for i, job in enumerate(jobs)]
    human = (
        f"CANDIDATE:\n{candidate_brief}\n\n"
        f"=== JOBS TO SCORE ({len(jobs)}) ===\n" + "\n\n".join(job_blocks)
    )
    resp = llm.invoke([SystemMessage(content=system_prompt), HumanMessage(content=human)])

    meta = getattr(resp, "usage_metadata", None) or {}
    cached = int((meta.get("input_token_details") or {}).get("cache_read") or 0)
    usage = {
        "input":      int(meta.get("input_tokens") or 0),
        "cache_read": cached,
        "output":     int(meta.get("output_tokens") or 0),
    }
    return _parse_results(resp), usage

Aggregated across batches, priced at list ($0.40/M input, $0.10/M cached, $1.60/M output), logged per batch. This turned out to be the most valuable part of the whole exercise, though not for the reason I expected.

The cache is real, but erratic. Two things showed up immediately:

• Block size is 128 tokens. cache_read never comes back as an arbitrary number, always a multiple of 128. On a ~1024-token prefix it topped out at exactly 1024.
• Hit rate was a coin flip. Identical prefixes seconds apart would report 91% then 0%. Four consecutive batches all came back cache_read=0.

The flapping had a cause. The gpt-4.1-mini fleet is many machines, each with its own local cache. Without a routing hint, identical requests scatter across shards and a fresh shard sees a cold cache. OpenAI exposes a prompt_cache_key top-level parameter that the load balancer combines with the prefix hash to keep the same (prefix, key) pair sticky to one shard. Documented ceiling ~15 req/min per key before it spills.

def get_llm(temperature=None, cache_key=None):
    """cache_key → OpenAI's prompt_cache_key: a sticky-routing hint so repeated
    requests with the same shared prefix land on the same shard. Pick a stable
    per-workload string. Ignored for Anthropic, which uses cache_control markers.
    """
    model_kwargs = {}
    if cache_key:
        model_kwargs["prompt_cache_key"] = cache_key
    return ChatOpenAI(
        model=settings.openai_model,
        api_key=settings.openai_api_key,
        temperature=settings.llm_temperature if temperature is None else temperature,
        model_kwargs=model_kwargs,
    )

# At the call site — one stable key per worker.
llm = get_llm(cache_key="jobagent:matcher")

One key per workload, never a global one. Different workers have different prefixes; sharing a key makes them evict each other. Never bake in timestamps or per-request entropy or you lose the stickiness that's the entire point.

The A/B test that fooled me for months. To verify the key helped I sent the same message list twice through the same client, differing only in prompt_cache_key:

PRICE_IN     = 0.40 / 1_000_000
PRICE_CACHED = 0.10 / 1_000_000

fake_job = {
    "title": "AI Engineer", "company": "Test Co",
    "description_text": "Build LLM features in Python.",
    "skills": ["Python", "LLM"], "min_years_exp": 2,
}
human = (f"CANDIDATE:\n{candidate_brief}\n\n"
         f"=== JOBS TO SCORE (1) ===\n### JOB id=0\n{_job_brief(fake_job)}")
msgs = [SystemMessage(content=system_prompt), HumanMessage(content=human)]

def call(llm, label):
    resp = llm.invoke(msgs)
    usage = getattr(resp, "usage_metadata", None) or {}
    cached = int((usage.get("input_token_details") or {}).get("cache_read") or 0)
    in_total  = int(usage.get("input_tokens") or 0)
    out_total = int(usage.get("output_tokens") or 0)
    cost = ((in_total - cached) * PRICE_IN + cached * PRICE_CACHED
            + out_total * (1.60 / 1_000_000))
    print(f"[{label}] input={in_total} (cache_read={cached}, "
          f"fresh={in_total-cached})  output={out_total}  cost=${cost:.6f}")

print("--- arm A: NO prompt_cache_key")
no_key = get_llm(temperature=0.0)
call(no_key, "A.1"); call(no_key, "A.2"); call(no_key, "A.3")

print("\n--- arm B: prompt_cache_key='jobagent:matcher'")
keyed = get_llm(temperature=0.0, cache_key="jobagent:matcher")
call(keyed, "B.1"); call(keyed, "B.2"); call(keyed, "B.3")

First run gave me what I wanted: 47.7% warm hit rate without the key, 95.4% with it. Clean gap, obvious win. I wrote it down and moved on.

Running the same script months later:

--- arm A: NO prompt_cache_key (baseline)
[A.1 cold] input=1073 (cache_read=0, fresh=1073)  output=56  cost=$0.000519
[A.2 warm] input=1073 (cache_read=0, fresh=1073)  output=53  cost=$0.000514
[A.3 warm] input=1073 (cache_read=0, fresh=1073)  output=53  cost=$0.000514

--- arm B: prompt_cache_key='jobagent:matcher'
[B.1 cold] input=1073 (cache_read=0, fresh=1073)  output=56  cost=$0.000519
[B.2 warm] input=1073 (cache_read=0, fresh=1073)  output=54  cost=$0.000516
[B.3 warm] input=1073 (cache_read=0, fresh=1073)  output=53  cost=$0.000514

Warm hit rate — A: 0.0%   B: 0.0%

Zero hits on both arms. Including the keyed one that had just measured 95.4%.

The answer is in the output: input=1073.

OpenAI only caches prompts of 1024 tokens or more, in 128-token blocks. The script's comment proudly explains that it uses "one tiny fake job… so the cacheable prefix dominates" — and in doing so it shrank total input to 1073 tokens, 49 over the minimum. With a 128-token block size there was no room to store anything above the threshold. The optimisation that was supposed to make the effect obvious is what suppressed it.

The test wasn't measuring the cache. It was measuring its own undersizing, and would have reported 0% whether the cache was on or off.

The number at realistic size. Same test, 8 jobs and full-length descriptions — matching production:

--- arm A: no key
[A.1 cold] input=3608 cache_read=0    hit=0.0%
[A.2 warm] input=3608 cache_read=3456 hit=95.8%
[A.3 warm] input=3608 cache_read=3456 hit=95.8%
--- arm B: keyed
[B.1 cold] input=3608 cache_read=0    hit=0.0%
[B.2 warm] input=3608 cache_read=3456 hit=95.8%
[B.3 warm] input=3608 cache_read=3456 hit=95.8%

Caching behaves as documented once the prompt clears 1024: cold miss, then a stable 95.8% warm, cache_read=3456 a clean multiple of 128.

But note what didn't happen. Both arms hit 95.8%. The prompt_cache_key made no measurable difference on this run. I'm not going to claim it's useless — shard drift is intermittent by nature, and a single clean run can't refute the earlier 47.7%/95.4% observation any more than that one run established it. I have one measurement pointing each way, which means I don't actually know the effect size. Honest state: unresolved.

What the savings really are. The 95.8% figure is same-prompt repetition. Real batches score different jobs, so cache only covers the static prefix:

• Cacheable prefix: ~1024 tokens
• Total input per batch: ~5500 tokens
• Steady-state hit rate: ~18% per warm batch
• First batch: always cold
• Estimated saving: ~$0.15/month

The ~1024-token prefix sits exactly at the minimum. It works because the surrounding batch pushes total input well past the threshold. A workload with a prefix near the boundary and a small dynamic suffix can silently get nothing.

1.2 — AWS Bedrock: the model you picked determines whether the marker exists

Second workload, different vendor. A batch document-evaluation service — the kind of thing that takes a set of documents, evaluates each along several criteria, and returns a structured verdict per document per criterion. Each criterion runs the same prompt template with the same system section of roughly 1.5k tokens (evaluator role, output JSON schema, a couple of few-shot examples), so a single submission fires a burst of calls that all share that prefix. Textbook cache target.

The model family in play was Mistral on Bedrock. Before wiring cachePoint into the code I wanted to know it would work at all.

Bedrock's shape is different from OpenAI's. No automatic caching, no routing key. You explicitly insert a cachePoint block into system[], messages[], or toolConfig.tools[], and Bedrock caches everything from the start of that array up to the marker:

resp = client.converse(
    modelId="us.amazon.nova-pro-v1:0",
    system=[
        {"text": VERY_LONG_SYSTEM_PROMPT},
        {"cachePoint": {"type": "default"}},   # ← everything above is cacheable
    ],
    messages=[
        {"role": "user", "content": [{"text": user_input}]},
    ],
    inferenceConfig={"maxTokens": 500},
)

Up to 4 markers per request (across system + messages + tools), type always "default" (reserved for future tiers), no configuration required beyond that.

The test. I set up a matrix: two Mistral models with and without the marker, plus Nova Lite as a positive control (Nova is on the documented support list, so if it works it proves my payload shape is right and any failure above is model-specific):

"""
Verify whether Bedrock supports prompt caching (cachePoint) for Mistral models.

Test matrix (all in us-west-2):
  1. mistral.ministral-3-8b-instruct           + cachePoint
  2. mistral.mistral-large-3-675b-instruct     + cachePoint
  3. Baseline (no cachePoint) to prove the payload otherwise works
  4. us.amazon.nova-lite-v1:0 as positive control
"""
import json, boto3
from botocore.exceptions import ClientError

client = boto3.client("bedrock-runtime", region_name="us-west-2")

# ~2700 tokens — well above every documented minimum
LONG_SYSTEM = (
    "You are a senior document evaluator. Given a document and a criterion, "
    "you must produce a structured verdict. Output strictly-valid JSON with "
    "keys: rating, reasoning, suggested_actions. Rating is one of "
    "excellent|acceptable|weak|reject. reasoning is a short paragraph. "
    "suggested_actions is a list of strings, most important first, or null "
    "if no action is required. "
) * 30

def try_call(model_id: str, use_cache: bool):
    system_blocks = [{"text": LONG_SYSTEM}]
    if use_cache:
        system_blocks.append({"cachePoint": {"type": "default"}})

    print(f"\n=== {model_id}  cache={use_cache} ===")
    try:
        resp = client.converse(
            modelId=model_id,
            system=system_blocks,
            messages=[{"role": "user",
                       "content": [{"text": "Summarise this document in one line."}]}],
            inferenceConfig={"maxTokens": 100, "temperature": 0.1},
        )
    except ClientError as e:
        err = e.response["Error"]
        print(f"  ❌ {err['Code']}: {err['Message'][:200]}")
        return

    usage = resp.get("usage", {})
    print(f"  ✅ usage: {json.dumps(usage)}")

for m in ["mistral.ministral-3-8b-instruct",
          "mistral.mistral-large-3-675b-instruct"]:
    try_call(m, use_cache=False)
    try_call(m, use_cache=True)

# Positive control — Nova is on Bedrock's documented cachePoint support list
try_call("us.amazon.nova-lite-v1:0", use_cache=False)
try_call("us.amazon.nova-lite-v1:0", use_cache=True)

Actual output:

=== mistral.ministral-3-8b-instruct  cache=False ===
  ✅ usage: {"inputTokens": 2748, "outputTokens": 100, "totalTokens": 2848}

=== mistral.ministral-3-8b-instruct  cache=True ===
  ❌ AccessDeniedException: You invoked an unsupported model or your request
     did not allow prompt caching. See the documentation for more information.

=== mistral.mistral-large-3-675b-instruct  cache=False ===
  ✅ usage: {"inputTokens": 2748, "outputTokens": 100, "totalTokens": 2848}

=== mistral.mistral-large-3-675b-instruct  cache=True ===
  ❌ AccessDeniedException: You invoked an unsupported model or your request
     did not allow prompt caching. See the documentation for more information.

=== us.amazon.nova-lite-v1:0  cache=False ===
  ✅ usage: {"inputTokens": 2773, "outputTokens": 0, "totalTokens": 2773}

=== us.amazon.nova-lite-v1:0  cache=True ===
  ✅ usage: {"inputTokens": 11, "outputTokens": 0, "totalTokens": 2773,
             "cacheReadInputTokens": 0, "cacheWriteInputTokens": 2762}

Three things worth staring at:

1. Mistral rejects the marker outright. AccessDenied, not silently ignored. That's actually the friendlier failure — you find out at deploy time, not in a slow trickle of "wait, why isn't caching helping".
2. Nova's first call writes the cache, doesn't read it. cacheWriteInputTokens=2762 (2762 tokens went into cache), cacheReadInputTokens=0 (nothing was cached yet). inputTokens drops from 2773 to 11 because the 2762 that went to cache are billed separately at the write rate. Second call within 5 min would flip: cacheReadInputTokens≈2762, cacheWriteInputTokens=0.
3. The billed inputTokens is only the fresh portion. Not the total. Cost accounting has to sum three numbers (inputTokens + cacheReadInputTokens * 0.1 + cacheWriteInputTokens * 1.25), not one. Easy to under-report or over-report if you assume it's one line item.

Consequence. A workload sitting on Mistral can't use Bedrock prompt caching without switching model families, and that switch isn't free: Mistral Large → Nova Pro or Claude Sonnet costs a re-evaluation on the actual task, and the caching savings only pay off if the new model scores well enough to keep. When a model was chosen for cost and latency in the first place, switching to unlock caching can erase the very win it's supposed to provide. Worth knowing the optimisation exists and where it fits; not worth taking without a benchmark first.

1.3 — Is cachePoint a guarantee? No.

A question that came up while I was writing this: if I add the marker and the prefix is above the minimum, is the cache hit rate 100%?

No. cachePoint (and OpenAI's automatic caching) both mean "eligible to cache", not "will cache". The observable miss modes:

• Prefix below the vendor minimum. Silently ignored, no error. Only cacheWriteInputTokens=0 tells you.

• TTL expired. 5-minute sliding window on Bedrock, roughly the same on OpenAI. Low-frequency workloads never hit.

• Prefix isn't byte-identical. A timestamp, a request_id, a whitespace difference — full miss, cache written from scratch. Templates with dynamic fields in the middle of "static" text are the classic case.

• Shard / region routing. Multiple backend replicas, each with its own KV cache in its own GPU memory (drops directly out of Part 0). OpenAI documents this and offers prompt_cache_key to pin. Bedrock only acknowledges the cross-region variant — "At times of high demand, these optimizations may lead to increased cache writes." Intra-region shard routing is not documented, so I measured it. Two runs of the same setup: Nova Lite in us-west-2, prefix 2312 tokens (well above the 1024 minimum), everything byte-identical, all inside the 30-min TTL.

  Run 1 — 20 back-to-back calls (~17 s total):

  call  cache_read  cache_write  status
  ────  ──────────  ───────────  ──────
    0            0         2312  cold (expected)
    1         2312            0  HIT
    2            0         2312  MISS ← re-wrote cache
    3            0         2312  MISS ← re-wrote again
    4-19      2312            0  HIT × 16
  

  17 hits, 2 misses on 19 warm attempts → 89.5%. Every miss was cache_write=2312, cache_read=0 — the signature of landing on a replica whose cache is cold, not of TTL eviction (which would just fail to return cached data, not conjure fresh writes on an in-window repeat).

  Run 2 — 10 calls at each of five inter-call intervals (0s, 2s, 10s, 30s, 60s):

  interval  attempts  hits  misses  hit%
  ────────  ────────  ────  ──────  ─────
      0s         9      9       0   100.0%
      2s         9      9       0   100.0%
     10s         9      9       0   100.0%
     30s         9      9       0   100.0%
     60s         9      8       1    88.9%
  

  Zero misses across the first four intervals. One miss at the 60s interval, mid-run (not related to time-since-start). This is the honest picture: the same test procedure gave 89.5% on one execution and 100% on four out of five the next day. Miss rate isn't a stable single number; it's low but intermittent, and I can't produce a clean "N% miss rate" from these sample sizes.

  What the two runs together do establish: misses do happen under otherwise- ideal conditions, all evidence points at replica routing (the cache_write signature, the fact that longer intervals within TTL don't monotonically degrade), and Bedrock exposes no pinning parameter for Claude or Nova to drive it to zero. Practical read: plan for a warm hit rate under 100%, probably 85-95% on Nova/Claude under ideal conditions, worse under autoscaling or cross-region routing. Don't build a system whose economics assume perfect caching.

• > 4 cachePoint markers. Excess ones are ignored, and Bedrock doesn't tell you which.

• Model doesn't support it. Some models AccessDeny (Mistral on Bedrock), some silently ignore. Always confirm with a positive control that returns non-zero cacheWrite/cacheRead.

• Cascade invalidation across sections. Bedrock's docs are explicit: the section order for cache lookups is tools → system → messages, and "changing content in an earlier section invalidates the cache for later sections". Rotate a tool definition and your system-prompt cache is gone too, even if the system prompt is byte-identical.

• TTL ordering rule (Bedrock, when mixing 1h and 5min blocks). Longer TTL must appear before shorter TTL in the array. Get this backwards and Bedrock rejects the request; get it "almost right" and the 1h block silently expires in 5 min.

• Batch inference API doesn't cache at all on Bedrock. Caching is on-demand endpoints only. If you route the same workload through the batch API for cost, you lose caching — sometimes a wash, sometimes not.

• Simplified Anthropic mode looks back only ~20 blocks. If your messages array is longer than that between your last user turn and the checkpoint, the older content is invisible for caching purposes.

Two operational rules I take from this:

1. The only way to know it's working is cacheReadInputTokens / (cacheRead + cacheWrite + input) as a per-call metric, logged to whatever your observability stack is. A healthy warm workload should sit above 60%.
2. Any dynamic content — timestamps, IDs, user names — belongs after the cachePoint, never before. Sounds obvious, catches everyone at least once.

---

Part 2 — Response cache: skipping the model entirely

Response caching is a completely different thing. Instead of asking the model provider to reuse internal state, you keep a Redis (or in-memory dict, or DynamoDB, whatever) in front of the LLM. Key = a hash of the request. Value = the response you got last time.

def call_llm(prompt: str, model: str, temperature: float) -> str:
    cache_key = hashlib.sha256(f"{model}|{temperature}|{prompt}".encode()).hexdigest()

    cached = redis.get(f"llm:response:{cache_key}")
    if cached:
        return json.loads(cached)   # zero model calls, zero tokens billed

    resp = bedrock.converse(modelId=model, messages=[{"role": "user",
                            "content": [{"text": prompt}]}])
    redis.setex(f"llm:response:{cache_key}", ttl=3600, value=json.dumps(resp))
    return resp

On a hit, the LLM is not invoked at all. Latency drops from ~1s to ~1ms, cost drops to zero. On a miss, you paid the full LLM call plus a Redis round-trip. This is the cheapest possible caching when it fires and the most useless when it doesn't.

2.1 — Two flavours

Exact match: key is a hash of the full request. Reliable, trivial to implement, high precision. Miss rate is high because "same request" is a strict predicate — a trailing space in the prompt is a full miss.

Semantic match: key is the embedding of the prompt, and you compare similarity against past requests with a vector index. Higher hit rate because paraphrased questions collapse to the same slot. Downside: you're now guessing whether two requests "mean the same thing", and that guess can be wrong. GPTCache, langchain.cache.RedisSemanticCache, llmcache all implement this. Every one of them has a false-positive story if you look hard enough.

2.2 — When it works, when it doesn't

The document-evaluation workload in Part 1.2 falls firmly in the ❌ column. Each submission carries a fresh set of documents, and the LLM call bakes that document text into the prompt. Two submissions almost never share request bytes even if they share the criterion, so response-cache hit rate would round to zero. Wiring it up would add cost (Redis + maintenance) with no offsetting benefit. The right call was to not build it.

That decision took thirty seconds because the criterion is simple: is the same exact request likely to arrive twice? If no, response cache is dead weight. If yes, it's the highest-leverage optimisation available because it bypasses the model entirely.

2.3 — Semantic cache's specific trap

If you go the semantic route, the failure mode that eats teams is false-positive hits. Two examples that share high embedding similarity but need different answers:

• "What time does the sun rise in Beijing" vs "What time does the sun rise in Shanghai" — very close embedding, completely different correct answer.
• "How do I cancel my order" vs "How do I cancel my subscription" — same story.

Tuning the similarity threshold is a knob with no correct setting. Too strict, hit rate collapses. Too loose, wrong answers ship. Auditing what got returned is hard because the user thinks they asked a fresh question. I'd only reach for semantic caching if (a) the underlying answer space is small and stable (FAQ / static docs), and (b) the cost of an occasional wrong answer is low.

---

Part 3 — How to actually pick

The choice between prompt cache and response cache isn't either/or; a mature system layers them. But most projects don't need both, and layering them without a signal to test each independently is a debugging nightmare later.

Decision tree that survives contact with reality:

Is the *exact* same request likely to arrive twice?
├── Yes → Response cache first. Highest leverage, LLM not invoked at all.
│         Use exact-match unless you have a strong reason (FAQ shape) for semantic.
│
└── No → Response cache is dead weight. Skip.
          │
          ├── Does the request share a long (>1024 token) static prefix across calls?
          │   ├── Yes → Prompt cache. Confirm your model supports it FIRST
          │   │         (Mistral on Bedrock does not). Log cacheRead/cacheWrite
          │   │         from day one.
          │   │
          │   └── No → Neither cache helps. Look elsewhere (output-side
          │             reductions, model swap, batching).

Rules I ended up writing on the wall:

1. Check vendor minimums and support list before building a measurement rig. OpenAI's 1024-token floor is documented. Bedrock's supported model list is documented. Both are trivial to look up. I've now wasted time on both by not doing this.

2. A test that reports 0% when things are broken and 0% when things are fine is not a test. Add an assertion the test would fail if the setup itself is wrong. assert input_tokens > VENDOR_MINIMUM in the OpenAI script. assert "cacheWriteInputTokens" in usage on Bedrock. If a measurement can only come back one way, it isn't measuring.

3. Measure at production shape. Every simplification I made to sharpen the signal is what destroyed it. Realistic batch size, realistic payload, or you're testing a different system than the one you ship.

4. Instrument before you optimise. Per-batch hit-rate logging is the one thing I'd keep if I had to throw out everything else. It converts hit rate from a thing you occasionally wonder about into a thing you'd notice breaking.

5. Look at the output side first. Input caching is the interesting problem; output tokens are usually the expensive one, and they're 3-4× the price per token. On the OpenAI job-matching workload I got a 61% cost reduction just from deleting three response fields nobody read. Ten-minute change, beat every caching trick by a wide margin.

6. Be suspicious of the clean result you wanted. The 47.7%-vs-95.4% number was plausible, matched my hypothesis, and I recorded it without re-running. It took months to find out I couldn't reproduce it. A result that confirms what you expect deserves a second run more than a surprising one does.

7. "The model supports caching" is not the same as "your calls will cache". Support means the marker isn't rejected. Actual caching depends on prefix length, prefix stability, TTL, shard routing, marker count. Assume nothing; verify with the usage counters.

---

Closing

None of this saved life-changing money. The OpenAI work chases about $0.15 a month at my scale. The Bedrock optimisation I chose not to do would have saved maybe 30% on input tokens for one service. The point isn't the dollars.

The point is that "we added caching" is a claim people put in slides and architecture diagrams that almost nobody has actually verified. When I started digging, I found:

• One project where the cache had been "on" for months but was hitting 0% because the test that verified it undersized the input.
• Another project where I would've spent a sprint wiring up cachePoint before discovering the model didn't support it.
• A third case where response cache was on the roadmap as an "obvious win" for a workload whose actual hit rate would round to zero.

Every one of these would have shipped and quietly done nothing. The instrumentation — the per-call cacheReadInputTokens / cacheWriteInputTokens / inputTokens triple, logged as a hit-rate metric, alerted on when it drifts — is worth more than any specific optimisation. Once you can see it, the right decision is usually obvious. Without seeing it, you're guessing, and guesses about caching are almost always wrong.

---

Primary sources

• OpenAI: Prompt caching guide — the source for the 1024-token minimum, prompt_cache_key semantics, ~15 req/min per-key routing ceiling, and GPT-5.6+ explicit-mode pricing (1.25× cache write, extended-retention TTLs).
• AWS: Bedrock prompt caching — source for per-model minimum-prefix numbers, the 4-checkpoint limit, the tools → system → messages cascade rule, TTL ordering constraint, and the on-demand-only restriction (batch API doesn't cache).

Every specific number in this article was cross-checked against these two docs at time of writing. If they diverge from what you read here, trust the vendor. Caching parameters change more often than blog posts do.