Two Kinds of LLM Caching, and When Each One Actually Fires

In October 2026 I moved a job-matching agent from gpt-4.1-mini to gpt-6-luna, changed nothing else, and reran my prompt-cache measurements. The A/B test reported a 99.9% warm hit rate. Production got nothing: every batch wrote its whole prompt to the cache at 1.25× the input price, and no later batch ever read it back. The default setup now cost 25% more per batch than having no cache at all, and the agent's cost line did not show it, because it had no field for cache writes.

That is the sharpest version of something I kept running into while measuring LLM caching. The mechanism is real, but it fires under narrower conditions than the marketing suggests, those conditions change with the model generation, and the only way to know whether it is working is to log the usage counters and look at them.

People use "LLM caching" for two different things. One is a setting the vendor exposes on the model API, a hint that says "reuse the attention state for this prefix". The other is a Redis you put in front of the LLM so that identical requests never reach the model at all.

The two are easy to confuse on a slide. I have measured both, and this note collects what I learned from three projects: OpenAI's prefix cache on that job-matching agent, measured on two model generations, AWS Bedrock's cachePoint marker on Mistral and Nova, and a response cache I decided not to build for a batch document-evaluation service.

Part 0 covers the theory first, because every quirk of prompt caching comes from one implementation detail inside transformer LLMs: the KV cache. If you already know how attention prefill works, skip to Part 1. The rest of the note does not explain it again.

---

Part 0: Why prefix caching works

Every transformer attention layer computes three projections of each token, called Q (query), K (key) and V (value). Attention pairs the current position's Q against every earlier position's K to decide how much to attend to each past token, then blends the corresponding V vectors with those weights into the output. Q is what a position asks for, K is what each past position advertises, and V is what each past position hands back.

That split is why only two of the three get cached. When you generate token n+1:

• You need K and V for every earlier token (positions 1..n). They are reused at every step without changing: the K/V for position 42 is the same whether you are generating token 100, 500 or 5000. These can be cached.
• You need Q only for the current position (n+1). It is computed once, dot-producted against the cached K/V, and discarded. Position n+2 needs its own new Q against the same K/V. Q is never reused, so there is nothing to cache.

So the KV cache holds exactly the two projections that get read repeatedly. Q is recomputed at each step, but it is small (one token's worth) and never appears twice.

Naively, generating a 500-token response over a 5000-token prompt would recompute K and V for every earlier token at every step: 500 × 5000 = 2.5M redundant projections. Transformers avoid this with the KV cache. After computing K and V for a token, the server keeps them in GPU memory, and later steps read them from there instead of recomputing them. Every production LLM inference server does this, including vLLM, TGI, TensorRT-LLM and Bedrock's own backend.

The inference call splits cleanly into two phases:

1. Prefill: run the whole prompt through the model in parallel to compute K and V (and, briefly, Q) for every prompt token. K and V are stored; Q is used to produce the first output token and then discarded. This phase is compute-bound, a matmul over the full prompt length, and it is where TTFT (time to first token) goes.
2. Decode: generate output tokens one at a time. Each step computes one new Q, K and V for the new position, appends the K/V to the growing cache, and reads the whole cache to compute attention. This phase is bound by memory bandwidth and costs much less per token than prefill.

Prefix caching is possible because the K and V of a token depend only on the tokens before it, never on the tokens after it. Autoregressive attention is causal by construction. So if two requests start with the same 2000-token prefix and diverge at token 2001, the K and V for the first 2000 tokens are byte-identical in both. (The Q vectors would be identical too, but nobody caches Q. Each one is used once and thrown away whether the prefix is cached or not.)

If the inference server keeps those 2000 tokens' K/V after request A finishes, and request B arrives with the same prefix, the server can skip prefill for those 2000 tokens entirely. It loads the cached K/V, runs prefill only over the new suffix (which does need fresh Q vectors, one per new position, to attend against the cached K/V), and then decodes as usual.

That is the whole mechanism. "Prompt caching" means not evicting the KV cache between requests, and reusing it when the next request shares a prefix. Every behaviour in the rest of this note follows from that:

Three consequences to keep in mind before you build anything:

• The cacheable region always starts at token 0. You cannot cache the middle of a prompt while leaving the start dynamic, which is why every provider tells you to put dynamic content (the user question, a request timestamp, the batch item) at the end of the prompt.
• The cache physically lives in specific GPUs. Anything that changes which GPU handles your request, such as autoscaling, a deployment or a routing change, can leave you with a cold cache. TTL is not the only cause of misses.
• Where the server writes matters as much as what it could reuse. Older OpenAI models matched any shared prefix. On GPT-5.6 and later, the K/V for your static prefix is only reusable if some request stored an entry that ends exactly there.

Everything in Part 1 either uses this mechanism (cachePoint and OpenAI's explicit breakpoints ask the server to keep the KV blocks for the prefix up to that point) or works around one of its limits (prompt_cache_key, on models before GPT-5.6, asks to be routed to the same GPU as last time).

---

Part 1: Prompt caching in practice

Every vendor's prompt-caching API is a thin wrapper over the KV-cache mechanism from Part 0. What differs is how you ask the server to keep the blocks, and how the pricing shows up:

The vendors differ a lot, but the requirement is the same everywhere: the prefix has to be long enough, stable enough, and reused often enough. If it is not, the marker does nothing and nothing tells you.

Primary sources for the numbers in this table: OpenAI prompt-caching guide, AWS Bedrock prompt-caching guide.

The rest of Part 1 is two projects where those three conditions mattered, with the first one measured on two model generations.

1.1 OpenAI: one agent, two model generations

The workload was a job-matching agent. It pulls postings, scores each against a stored resume, and surfaces the good ones. Scoring runs in batches of 8, and each LLM call is shaped like this:

SystemMessage : <scoring rubric>                    ← static
HumanMessage  : CANDIDATE: <candidate brief>        ← static, from stored resume
                === JOBS TO SCORE (8) ===
                ### JOB id=0 ...                    ← dynamic, new every batch

The first two blocks never change between batches. That is about 1024 tokens of static prefix out of roughly 5500 per call, which makes it an obvious cache target.

I measured the cache on this workload twice: first on gpt-4.1-mini, then in October 2026 on gpt-6-luna, after OpenAI changed how caching works from GPT-5.6 on. The same logging and the same A/B test gave me a misleading answer both times, for different reasons.

Before optimising anything, I wanted the hit rate in the logs. LangChain exposes it a couple of levels down on the response:

def score_batch(llm, system_prompt, candidate_brief, jobs):
    job_blocks = [f"### JOB id={i}\n{_job_brief(job)}" for i, job in enumerate(jobs)]
    human = (
        f"CANDIDATE:\n{candidate_brief}\n\n"
        f"=== JOBS TO SCORE ({len(jobs)}) ===\n" + "\n\n".join(job_blocks)
    )
    resp = llm.invoke([SystemMessage(content=system_prompt), HumanMessage(content=human)])

    meta = getattr(resp, "usage_metadata", None) or {}
    cached = int((meta.get("input_token_details") or {}).get("cache_read") or 0)
    usage = {
        "input":      int(meta.get("input_tokens") or 0),
        "cache_read": cached,
        "output":     int(meta.get("output_tokens") or 0),
    }
    return _parse_results(resp), usage

I aggregate it across batches, price it at gpt-4.1-mini list prices ($0.40/M input, $0.10/M cached, $1.60/M output), and log it per batch. This logging turned out to be the most useful part of the whole exercise, and later the part that misled me.

On gpt-4.1-mini the logs showed that the cache worked, but erratically:

• Block size is 128 tokens. cache_read never comes back as an arbitrary number; it is always a multiple of 128. On a prefix of about 1024 tokens it topped out at exactly 1024.
• Hit rate was a coin flip. Identical prefixes a few seconds apart reported 91% and then 0%. Four consecutive batches all came back with cache_read=0.

There was a reason for the flapping. The gpt-4.1-mini fleet is many machines, each with its own local cache. Without a routing hint, identical requests scatter across shards, and a shard that has not seen the prefix has a cold cache. OpenAI exposes a top-level prompt_cache_key parameter, which the load balancer combines with the prefix hash so that the same (prefix, key) pair keeps going to the same shard. The documented ceiling is about 15 requests per minute per key before traffic spills to other shards.

def get_llm(temperature=None, cache_key=None):
    """cache_key → OpenAI's prompt_cache_key: a sticky-routing hint so repeated
    requests with the same shared prefix land on the same shard. Pick a stable
    per-workload string. Ignored for Anthropic, which uses cache_control markers.
    """
    model_kwargs = {}
    if cache_key:
        model_kwargs["prompt_cache_key"] = cache_key
    return ChatOpenAI(
        model=settings.openai_model,
        api_key=settings.openai_api_key,
        temperature=settings.llm_temperature if temperature is None else temperature,
        model_kwargs=model_kwargs,
    )

# At the call site: one stable key per worker.
llm = get_llm(cache_key="jobagent:matcher")

Use one key per workload, not one global key. Different workers have different prefixes, and if they share a key they evict each other. Never put timestamps or other per-request values in the key, or you lose the stickiness the key exists for.

To check that the key helped, I sent the same message list through the same client twice, changing only prompt_cache_key:

PRICE_IN     = 0.40 / 1_000_000
PRICE_CACHED = 0.10 / 1_000_000

fake_job = {
    "title": "AI Engineer", "company": "Test Co",
    "description_text": "Build LLM features in Python.",
    "skills": ["Python", "LLM"], "min_years_exp": 2,
}
human = (f"CANDIDATE:\n{candidate_brief}\n\n"
         f"=== JOBS TO SCORE (1) ===\n### JOB id=0\n{_job_brief(fake_job)}")
msgs = [SystemMessage(content=system_prompt), HumanMessage(content=human)]

def call(llm, label):
    resp = llm.invoke(msgs)
    usage = getattr(resp, "usage_metadata", None) or {}
    cached = int((usage.get("input_token_details") or {}).get("cache_read") or 0)
    in_total  = int(usage.get("input_tokens") or 0)
    out_total = int(usage.get("output_tokens") or 0)
    cost = ((in_total - cached) * PRICE_IN + cached * PRICE_CACHED
            + out_total * (1.60 / 1_000_000))
    print(f"[{label}] input={in_total} (cache_read={cached}, "
          f"fresh={in_total-cached})  output={out_total}  cost=${cost:.6f}")

print("--- arm A: NO prompt_cache_key")
no_key = get_llm(temperature=0.0)
call(no_key, "A.1"); call(no_key, "A.2"); call(no_key, "A.3")

print("\n--- arm B: prompt_cache_key='jobagent:matcher'")
keyed = get_llm(temperature=0.0, cache_key="jobagent:matcher")
call(keyed, "B.1"); call(keyed, "B.2"); call(keyed, "B.3")

The first run gave me what I wanted: 47.7% warm hit rate without the key and 95.4% with it. It was a clean gap and an obvious win, so I wrote it down and moved on.

Running the same script months later:

--- arm A: NO prompt_cache_key (baseline)
[A.1 cold] input=1073 (cache_read=0, fresh=1073)  output=56  cost=$0.000519
[A.2 warm] input=1073 (cache_read=0, fresh=1073)  output=53  cost=$0.000514
[A.3 warm] input=1073 (cache_read=0, fresh=1073)  output=53  cost=$0.000514

--- arm B: prompt_cache_key='jobagent:matcher'
[B.1 cold] input=1073 (cache_read=0, fresh=1073)  output=56  cost=$0.000519
[B.2 warm] input=1073 (cache_read=0, fresh=1073)  output=54  cost=$0.000516
[B.3 warm] input=1073 (cache_read=0, fresh=1073)  output=53  cost=$0.000514

Warm hit rate — A: 0.0%   B: 0.0%

Both arms got zero hits, including the keyed arm that had measured 95.4%.

The reason is in the output: input=1073.

OpenAI only caches prompts of 1024 tokens or more, in 128-token blocks. A comment in the script explains that it uses "one tiny fake job… so the cacheable prefix dominates". Doing that shrank the total input to 1073 tokens, 49 over the minimum, and with a 128-token block size there was no room to store anything above the threshold. The simplification that was meant to make the effect obvious is what suppressed it.

At that size the test would have reported 0% whether the cache was on or off, so it was measuring its own undersizing and nothing else.

Here is the same test with 8 jobs and full-length descriptions, matching production:

--- arm A: no key
[A.1 cold] input=3608 cache_read=0    hit=0.0%
[A.2 warm] input=3608 cache_read=3456 hit=95.8%
[A.3 warm] input=3608 cache_read=3456 hit=95.8%
--- arm B: keyed
[B.1 cold] input=3608 cache_read=0    hit=0.0%
[B.2 warm] input=3608 cache_read=3456 hit=95.8%
[B.3 warm] input=3608 cache_read=3456 hit=95.8%

Once the prompt clears 1024 tokens, caching behaves as documented: a cold miss, then a stable 95.8% warm hit rate, with cache_read=3456, a clean multiple of 128.

Both arms hit 95.8%, though. The prompt_cache_key made no measurable difference on this run. I am not going to claim the key is useless. Shard drift is intermittent, and a single clean run cannot refute the earlier 47.7%/95.4% result any more than that one run established it. I have one measurement pointing each way, which means I don't actually know the effect size. The question is still open.

The 95.8% figure comes from sending the same prompt repeatedly. Real batches score different jobs, so the cache only covers the static prefix:

• Cacheable prefix: ~1024 tokens
• Total input per batch: ~5500 tokens
• Steady-state hit rate: ~18% per warm batch
• First batch: always cold
• Estimated saving: ~$0.15/month

The prefix of about 1024 tokens sits exactly at the minimum. It gets cached because the rest of the batch pushes total input well past the threshold. A workload with a prefix near the boundary and a small dynamic suffix can silently get nothing.

Then the model changed. gpt-4.1-mini retires on 2027-01-14, so in October 2026 I moved the agent to gpt-6-luna with reasoning effort none, changed nothing else, and reran the same measurements. The lessons so far still held. Most of the mechanism did not.

The routing key no longer affects hit rate. From GPT-5.6 on, OpenAI routes requests to the right cache itself. Same A/B as above, 8 real jobs:

--- arm A: no key
[A.1] input=5696 cached=0    write=5693
[A.2] input=5696 cached=5693 write=0     hit=99.9%
[A.3] input=5696 cached=5693 write=0     hit=99.9%
--- arm B: keyed
[B.1] input=5696 cached=0    write=5693
[B.2] input=5696 cached=5693 write=0     hit=99.9%
[B.3] input=5696 cached=5693 write=0     hit=99.9%

The two arms are identical, so the open question about the key does not carry over to new models. One detail: B.1 was cold, even though arm A had cached the exact same prompt seconds earlier. The docs now describe the key as a way to keep cache accounting separate per customer or user, so on new models a key splits the cache instead of steering it. With one key per worker that costs nothing, but it does not help either.

The output shows two smaller changes. cached is now exact (5693) rather than rounded down to a multiple of 128, and there is a new field, cache_write_tokens.

Cache writes now cost money. On GPT-5.6 and later, tokens written to the cache bill at 1.25× the input rate. For luna that is $0.125/M written, against $0.10/M uncached and $0.01/M read from cache. On older models writes were free, so caching was a free bonus. Now you pay 25% extra up front, and you only come out ahead if the entry is read back.

On the real workload it never is. The A/B sends the same prompt three times, while production sends different jobs in every batch:

batch 1: input=5721 cached=0 write=5718
batch 2: input=5745 cached=0 write=5742
batch 3: input=5519 cached=0 write=5516
batch 4: input=5967 cached=0 write=5964

No batch reuses anything, and every one writes its entire prompt at the premium rate. The old cache matched any shared prefix. The new one only matches at boundaries. In the default implicit mode it writes one entry, ending at the last user message, and a later request can only hit a prefix that ends exactly where an earlier entry ended. The matcher puts the static candidate brief and the dynamic job list in the same user message, so the only entry ever written includes that batch's jobs, and no other batch can match it.

Moving the brief into a user message of its own does not help. I tried it and got the same four cold batches. The end of an earlier message is a place the cache looks, but implicit mode only writes at the last message.

What works is an explicit breakpoint right after the static part. Set prompt_cache_options: {"mode": "explicit"} on the request, split the user message into two content blocks, and put prompt_cache_breakpoint: {"mode": "explicit"} on the brief's block. The docs only show this on the Responses API, but Chat Completions accepts it too. So does LangChain, which passes the extra key through in the content block and the options through extra_body. A Responses API run:

batch 1: input=5483 cached=0    write=1200
batch 2: input=5709 cached=1200 write=0
batch 3: input=5325 cached=1200 write=0
batch 4: input=5622 cached=1200 write=0

The prefix is written once and read by every later batch. That is about 22% of input per warm batch, close to the static-prefix hit rate I estimated for gpt-4.1-mini, except that now it is deterministic.

Input cost per batch at luna prices (about 5,600 input tokens), compared with no caching:

Explicit mode with no breakpoint at all gives write=0, cached=0 on every batch. That is the closest thing to turning the cache off.

Which one to choose depends on how many batches share the prefix within the cache's 30-minute lifetime. The breakpoint costs 1,200 × 0.025 extra on the first batch and saves 1,200 × 0.09 on each batch after it, so it loses on a one-batch run and wins from two batches up. The nightly runs are 24 hours apart, so the first batch of each is always cold, but the batches within a run are seconds apart. Over the last 30 days (17 runs, 4.2 batches per run on average) at luna prices, total matcher cost including output would have been:

In dollars this is still a fraction of a cent per run. What changed is the sign: the default setup now costs more than having no cache at all.

The logs got it wrong again. The agent's cost estimate counts fresh and cached tokens but not writes, and LangChain's usage_metadata does not report cache_write_tokens at all; it only maps cached_tokens to cache_read. So on luna the cost line understates every batch's input cost by about 25%. And hit_pct=0 reads like "the cache missed", when what really happened is "paid extra to write an entry nobody will read". It is the undersized test from the first round in a new form: the instrumentation had no field to tell the two cases apart.

The agent's code has not caught up yet. It still sets the key, has no breakpoint after the brief, and leaves writes out of the cost line.

Put the two rounds side by side and the A/B test failed the same way both times. It sends one prompt three times, so a warm call reuses the whole prompt, and it cannot tell that apart from reusing only the static prefix. On gpt-4.1-mini that gap was harmless: the test said 95.8%, production got about 18%, and both numbers came from the same cache behaviour. On gpt-6-luna the test said 99.9% and production got nothing, at a 25% premium. The blind spot did not change. The model generation decided how much it cost.

1.2 AWS Bedrock: whether the marker exists depends on the model

The second workload was on a different vendor: a batch document-evaluation service, which takes a set of documents, evaluates each one against several criteria, and returns a structured verdict per document per criterion. Every criterion uses the same prompt template with the same system section of roughly 1.5k tokens (evaluator role, output JSON schema, a couple of few-shot examples), so one submission fires a burst of calls that all share that prefix. It is a textbook cache target.

The model family in use was Mistral on Bedrock. Before wiring cachePoint into the code, I wanted to know whether it would work at all.

Bedrock works differently from OpenAI. There is no automatic caching and no routing key. You insert a cachePoint block into system[], messages[] or toolConfig.tools[], and Bedrock caches everything from the start of that array up to the marker:

resp = client.converse(
    modelId="us.amazon.nova-pro-v1:0",
    system=[
        {"text": VERY_LONG_SYSTEM_PROMPT},
        {"cachePoint": {"type": "default"}},   # ← everything above is cacheable
    ],
    messages=[
        {"role": "user", "content": [{"text": user_input}]},
    ],
    inferenceConfig={"maxTokens": 500},
)

You can use up to 4 markers per request across system, messages and tools. type is always "default" (reserved for future tiers), and there is nothing else to configure.

I set up a test matrix: two Mistral models with and without the marker, plus Nova Lite as a positive control. Nova is on the documented support list, so if it works, my payload shape is right and any failure on the Mistral models is specific to them:

"""
Verify whether Bedrock supports prompt caching (cachePoint) for Mistral models.

Test matrix (all in us-west-2):
  1. mistral.ministral-3-8b-instruct           + cachePoint
  2. mistral.mistral-large-3-675b-instruct     + cachePoint
  3. Baseline (no cachePoint) to prove the payload otherwise works
  4. us.amazon.nova-lite-v1:0 as positive control
"""
import json, boto3
from botocore.exceptions import ClientError

client = boto3.client("bedrock-runtime", region_name="us-west-2")

# ~2700 tokens, well above every documented minimum
LONG_SYSTEM = (
    "You are a senior document evaluator. Given a document and a criterion, "
    "you must produce a structured verdict. Output strictly-valid JSON with "
    "keys: rating, reasoning, suggested_actions. Rating is one of "
    "excellent|acceptable|weak|reject. reasoning is a short paragraph. "
    "suggested_actions is a list of strings, most important first, or null "
    "if no action is required. "
) * 30

def try_call(model_id: str, use_cache: bool):
    system_blocks = [{"text": LONG_SYSTEM}]
    if use_cache:
        system_blocks.append({"cachePoint": {"type": "default"}})

    print(f"\n=== {model_id}  cache={use_cache} ===")
    try:
        resp = client.converse(
            modelId=model_id,
            system=system_blocks,
            messages=[{"role": "user",
                       "content": [{"text": "Summarise this document in one line."}]}],
            inferenceConfig={"maxTokens": 100, "temperature": 0.1},
        )
    except ClientError as e:
        err = e.response["Error"]
        print(f"  ❌ {err['Code']}: {err['Message'][:200]}")
        return

    usage = resp.get("usage", {})
    print(f"  ✅ usage: {json.dumps(usage)}")

for m in ["mistral.ministral-3-8b-instruct",
          "mistral.mistral-large-3-675b-instruct"]:
    try_call(m, use_cache=False)
    try_call(m, use_cache=True)

# Positive control: Nova is on Bedrock's documented cachePoint support list
try_call("us.amazon.nova-lite-v1:0", use_cache=False)
try_call("us.amazon.nova-lite-v1:0", use_cache=True)

Actual output:

=== mistral.ministral-3-8b-instruct  cache=False ===
  ✅ usage: {"inputTokens": 2748, "outputTokens": 100, "totalTokens": 2848}

=== mistral.ministral-3-8b-instruct  cache=True ===
  ❌ AccessDeniedException: You invoked an unsupported model or your request
     did not allow prompt caching. See the documentation for more information.

=== mistral.mistral-large-3-675b-instruct  cache=False ===
  ✅ usage: {"inputTokens": 2748, "outputTokens": 100, "totalTokens": 2848}

=== mistral.mistral-large-3-675b-instruct  cache=True ===
  ❌ AccessDeniedException: You invoked an unsupported model or your request
     did not allow prompt caching. See the documentation for more information.

=== us.amazon.nova-lite-v1:0  cache=False ===
  ✅ usage: {"inputTokens": 2773, "outputTokens": 0, "totalTokens": 2773}

=== us.amazon.nova-lite-v1:0  cache=True ===
  ✅ usage: {"inputTokens": 11, "outputTokens": 0, "totalTokens": 2773,
             "cacheReadInputTokens": 0, "cacheWriteInputTokens": 2762}

Three things stand out:

1. Mistral rejects the marker outright with AccessDenied instead of silently ignoring it. That is the friendlier failure, because you find out at deploy time instead of slowly wondering why caching is not helping.
2. Nova's first call writes the cache and does not read it. cacheWriteInputTokens=2762 means 2762 tokens went into the cache, and cacheReadInputTokens=0 means nothing was cached yet. inputTokens drops from 2773 to 11 because the 2762 cached tokens are billed separately at the write rate. A second call within 5 minutes should flip this to cacheReadInputTokens≈2762 and cacheWriteInputTokens=0.
3. The billed inputTokens is only the fresh portion, not the total. Cost accounting has to combine three numbers (inputTokens + cacheReadInputTokens * 0.1 + cacheWriteInputTokens * 1.25). If you assume it is one line item, you will under-report or over-report.

So a workload on Mistral cannot use Bedrock prompt caching without switching model families, and switching is not free. Moving from Mistral Large to Nova Pro or Claude Sonnet means re-evaluating on the actual task, and the caching savings only pay off if the new model scores well enough to keep. When a model was chosen for cost and latency in the first place, switching to unlock caching can wipe out the saving it was supposed to bring. It is worth knowing the option exists, but not worth taking without a benchmark first.

1.3 Is cachePoint a guarantee?

A question came up while I was writing this: if I add the marker and the prefix is above the minimum, is the hit rate 100%?

No. cachePoint, like OpenAI's automatic caching, makes a prefix eligible to be cached. It does not promise that it will be. These are the miss modes you can observe:

• Prefix below the vendor minimum. It is silently ignored, with no error. Only cacheWriteInputTokens=0 tells you.

• TTL expired. Bedrock uses a 5-minute sliding window for Claude and Nova. OpenAI keeps the cache in memory for about 5-10 minutes on models before GPT-5.6, and for at least 30 minutes on GPT-5.6 and later. Low-frequency workloads never hit.

• Prefix not byte-identical. A timestamp, a request_id or a whitespace difference causes a full miss, and the cache is written from scratch. Templates with dynamic fields in the middle of "static" text are the classic case.

• Write boundary in the wrong place (OpenAI, GPT-5.6 and later). A request only hits a prefix that ends exactly where an earlier write ended, and implicit mode only writes at the end of the last message. A static prefix followed by dynamic content in the same message is never cached on its own, as 1.1 shows.

• Shard or region routing. There are multiple backend replicas, each with its own KV cache in its own GPU memory, as Part 0 predicts. OpenAI documents this and, on models before GPT-5.6, offers prompt_cache_key to pin requests. Bedrock only acknowledges the cross-region version: "At times of high demand, these optimizations may lead to increased cache writes." Routing between shards inside a region is not documented, so I measured it. I ran the same setup twice: Nova Lite in us-west-2, a 2312-token prefix (well above the 1024 minimum), everything byte-identical, all well inside the TTL (the longest gap between calls was 60 s).

  Run 1, 20 back-to-back calls (about 17 s in total):

  call  cache_read  cache_write  status
  ────  ──────────  ───────────  ──────
    0            0         2312  cold (expected)
    1         2312            0  HIT
    2            0         2312  MISS ← re-wrote cache
    3            0         2312  MISS ← re-wrote again
    4-19      2312            0  HIT × 16
  

  17 hits and 2 misses on 19 warm attempts, which is 89.5%. Every miss showed cache_write=2312, cache_read=0, which is what landing on a replica with a cold cache looks like. It cannot be TTL eviction, because every call was a repeat well inside the TTL window.

  Run 2, 10 calls at each of five intervals between calls (0 s, 2 s, 10 s, 30 s, 60 s):

  interval  attempts  hits  misses  hit%
  ────────  ────────  ────  ──────  ─────
      0s         9      9       0   100.0%
      2s         9      9       0   100.0%
     10s         9      9       0   100.0%
     30s         9      9       0   100.0%
     60s         9      8       1    88.9%
  

  No misses at the first four intervals, and one at the 60 s interval, in the middle of the run (so not related to time since the start). The same test procedure gave 89.5% on one day and 100% at four of the five intervals the next day. The miss rate is not a stable number. It is low but intermittent, and these sample sizes cannot give a clean "N% miss rate".

  What the two runs do establish: misses happen even under otherwise ideal conditions, the evidence points at replica routing (the cache_write pattern, and the fact that longer intervals inside the TTL do not steadily make it worse), and Bedrock has no pinning parameter for Claude or Nova to drive misses to zero. In practice, plan for a warm hit rate below 100%, probably 85-95% on Nova or Claude under ideal conditions and lower with autoscaling or cross-region routing. Do not build a system whose economics assume perfect caching.

• More than 4 cachePoint markers. The extra ones are ignored, and Bedrock does not tell you which.

• Model does not support it. Some models return AccessDenied (Mistral on Bedrock) and some silently ignore the marker. Always confirm with a positive control that returns non-zero cacheWrite/cacheRead.

• Cascade invalidation across sections. Bedrock's docs say the lookup order is tools → system → messages, and "changing content in an earlier section invalidates the cache for later sections". Change a tool definition and your system-prompt cache is gone too, even if the system prompt is byte-identical.

• TTL ordering (Bedrock, when mixing 1 h and 5 min blocks). Longer TTLs must come before shorter ones in the array. Get the order backwards and Bedrock rejects the request. Get it almost right and the 1 h block silently expires after 5 minutes.

• Batch inference does not cache at all on Bedrock. Caching only works on on-demand endpoints, so if you move the same workload to the batch API to save money, you lose caching. Sometimes that evens out and sometimes it does not.

• Simplified Anthropic mode only looks back about 20 blocks. If your messages array has more than that between your last user turn and the checkpoint, the older content is invisible for caching.

Two operational rules I take from this:

1. The only way to know caching is working is to log cacheReadInputTokens / (cacheRead + cacheWrite + input) as a per-call metric in whatever observability stack you use. A healthy warm workload should sit above 60%.
2. Dynamic content such as timestamps, IDs and user names goes after the cachePoint, never before it. It sounds obvious, and it catches most people at least once.

---

Part 2: Response caching, which skips the model entirely

Response caching is a different thing altogether. Instead of asking the model provider to reuse internal state, you put a Redis (or an in-memory dict, or DynamoDB) in front of the LLM. The key is a hash of the request, and the value is the response you got last time.

def call_llm(prompt: str, model: str, temperature: float) -> str:
    cache_key = hashlib.sha256(f"{model}|{temperature}|{prompt}".encode()).hexdigest()

    cached = redis.get(f"llm:response:{cache_key}")
    if cached:
        return json.loads(cached)   # zero model calls, zero tokens billed

    resp = bedrock.converse(modelId=model, messages=[{"role": "user",
                            "content": [{"text": prompt}]}])
    redis.setex(f"llm:response:{cache_key}", ttl=3600, value=json.dumps(resp))
    return resp

On a hit, the LLM is not called at all. Latency drops from about 1 s to about 1 ms, and the cost is zero. On a miss, you pay for the full LLM call plus a Redis round trip. It is the cheapest caching there is when it fires, and pure overhead when it does not.

2.1 Two flavours

Exact match: the key is a hash of the full request. It is reliable, trivial to implement and precise. The miss rate is high because "same request" is a strict test: a trailing space in the prompt is a full miss.

Semantic match: the key is an embedding of the prompt, and you compare it against past requests with a vector index. The hit rate is higher because paraphrased questions land on the same entry. The downside is that you are now guessing whether two requests mean the same thing, and the guess can be wrong. GPTCache, langchain.cache.RedisSemanticCache and llmcache all implement this, and each of them has a false-positive story if you look for one.

2.2 When it works and when it does not

The document-evaluation workload from 1.2 is a bad fit. Each submission carries a fresh set of documents, and the LLM call includes that document text in the prompt. Two submissions almost never share request bytes even when they share the criterion, so the response-cache hit rate would round to zero. Wiring it up would add cost (Redis plus maintenance) and save nothing, so I did not build it.

That decision took thirty seconds, because the test is simple: is the same exact request likely to arrive twice? If not, a response cache is dead weight. If it is, a response cache is the highest-leverage optimisation available, because it bypasses the model entirely.

2.3 The trap in semantic caching

If you go the semantic route, the failure that hurts teams is false-positive hits: two requests with high embedding similarity that need different answers.

• "What time does the sun rise in Beijing" vs "What time does the sun rise in Shanghai": very close embeddings, completely different correct answers.
• "How do I cancel my order" vs "How do I cancel my subscription": the same problem.

The similarity threshold has no correct setting. Too strict and the hit rate collapses; too loose and wrong answers ship. Auditing what was returned is hard, because the user thinks they asked a new question. I would only use semantic caching if (a) the answer space is small and stable (FAQ, static docs), and (b) an occasional wrong answer is cheap.

---

Part 3: How to choose

Prompt caching and response caching are not mutually exclusive, and a mature system can layer them. Most projects do not need both, though, and layering them without a way to measure each one on its own makes debugging much harder later.

Decision tree:

Is the *exact* same request likely to arrive twice?
├── Yes → Response cache first. Highest leverage, LLM not invoked at all.
│         Use exact-match unless you have a strong reason (FAQ shape) for semantic.
│
└── No → Response cache is dead weight. Skip.
          │
          ├── Does the request share a long (>1024 token) static prefix across calls?
          │   ├── Yes → Prompt cache. Confirm your model supports it FIRST
          │   │         (Mistral on Bedrock does not), and check where it
          │   │         writes (OpenAI GPT-5.6+ only at boundaries). Log
          │   │         cacheRead/cacheWrite from day one.
          │   │
          │   └── No → Neither cache helps. Look elsewhere (output-side
          │             reductions, model swap, batching).

Rules I took away:

1. Check vendor minimums and the supported model list before building a measurement rig. OpenAI's 1024-token floor is documented, and so is Bedrock's list of supported models. Both take a minute to look up, and I have wasted time on each by not doing it.

2. A test that reports 0% when things are broken and 0% when things are fine is not a test. Add an assertion that fails if the setup itself is wrong: assert input_tokens > VENDOR_MINIMUM in the OpenAI script, assert "cacheWriteInputTokens" in usage on Bedrock. If a measurement can only come back one way, it is not measuring anything.

3. Measure at production shape. Every simplification I made to sharpen the signal is what destroyed it. Use a realistic batch size and payload, or you are testing a different system from the one you ship.

4. Instrument before you optimise. Per-batch hit-rate logging is the one thing I would keep if I had to throw everything else out. With it in place, a drop in hit rate shows up in the logs instead of going unnoticed.

5. Look at the output side first. Input caching is the more interesting problem, but output tokens are usually the expensive ones, at 3-4× the price per token. On the OpenAI job-matching workload, deleting three response fields nobody read cut output tokens by 61% and per-call cost by 44%. It was a ten-minute change and it beat every caching trick by a wide margin.

6. Be suspicious of the clean result you wanted. The 47.7%-vs-95.4% result was plausible and matched my hypothesis, and I recorded it without re-running it. It took months to find out I could not reproduce it. A result that confirms what you expected deserves a second run more than a surprising one does.

7. "The model supports caching" and "your calls will cache" are different claims. Support means the marker is not rejected. Whether anything is cached depends on prefix length, prefix stability, TTL, shard routing and the number of markers. Verify with the usage counters.

8. Caching semantics belong to the model generation, not the vendor. "OpenAI caches prefixes automatically" was true, then it became "OpenAI caches at boundaries, and charges for writes". Re-read the caching docs whenever the model changes. That check is as much a part of a migration as re-running evals.

9. A test that sends the same prompt twice cannot tell "the prefix was reused" from "the whole prompt was reused". The 99.9% A/B on gpt-6-luna looked perfect while production got nothing. The 95.8% on gpt-4.1-mini had the same blind spot; it just did not matter on a model that matched any prefix. Production shape includes varying the parts that vary in production.

---

Closing

None of this saved much money. The OpenAI work is worth about $0.15 a month at my scale. The Bedrock optimisation I decided against would have saved maybe 30% on input tokens for one service.

What matters more is that "we added caching" is a claim people put in slides and architecture diagrams, and almost nobody has verified it. When I started digging, I found:

• One project where the script I used to measure the cache had undersized its own input, so on a later run it could only report 0%, whether the cache worked or not.
• Another project where I would have spent a sprint wiring up cachePoint before finding out the model did not support it.
• A third case where a response cache was on the roadmap as an "obvious win" for a workload whose real hit rate would round to zero.
• The first project again, after a model upgrade, where the default cache setup wrote every batch at a 25% premium and never read any of it back.

The first three would have shipped and quietly done nothing. The fourth quietly cost more than having no cache, and the cost line did not show it. The instrumentation is worth more than any specific optimisation: the per-call cacheReadInputTokens / cacheWriteInputTokens / inputTokens triple (on OpenAI, cached_tokens / cache_write_tokens / input_tokens), logged as a hit-rate metric and priced into the cost line, with an alert when it drifts. Once you can see it, the right decision is usually obvious. Without it you are guessing, and guesses about caching are usually wrong.

---

Primary sources

• OpenAI: Prompt caching guide. Source for the 1024-token minimum, prompt_cache_key semantics before and after GPT-5.6, the ~15 req/min per-key routing ceiling, GPT-5.6+ explicit-mode pricing (1.25× cache write, 0.1× cache read, 0.05× on GPT-6.1 Sol), implicit and explicit breakpoints, and the TTL values.
• AWS: Bedrock prompt caching. Source for per-model minimum-prefix numbers, the 4-checkpoint limit, the tools → system → messages cascade rule, the TTL ordering constraint, and the on-demand-only restriction (the batch API does not cache).

I cross-checked every specific number in this note against these two docs when I wrote it. If they differ from what you read here, trust the vendor. Caching parameters change more often than blog posts do.