What similarity search cannot do

Eight case files. A detective asks a question, the system retrieves candidates, and writes a cited report. The corpus is adversarial by construction: three of the eight files are topically identical to the real evidence and explicitly state they are unrelated.

That last property is the whole problem. A document saying "this breach was unrelated to the theft" is, to an embedding model, maximally about the theft. Similarity search ranks it at the top and has no mechanism to tell it apart, because the thing that disqualifies it is a claim inside the text, not a property of its topic.

Retrieval gets you candidates, not evidence

Dense retrieval alone loses here, and the reason is measurable. The terms that actually discriminate between these documents are rare proper nouns — wallet addresses, incident identifiers — which is precisely what lexical search is good at and embeddings smooth away. BM25 separates the real evidence from the distractors by 3.09; dense similarity separates them by 0.00.

So the retriever is hybrid, and the two rankings are fused with Reciprocal Rank Fusion rather than by normalising and adding scores. Scores from a dense retriever and from BM25 are not on a comparable scale and normalising them invents a comparability that does not exist. Ranks are comparable by construction.

Then the obvious idea: if the distractors rank high but score slightly lower, cut at a threshold. This does not work, and I can say exactly how badly. A threshold sweep clears 5 of 8 queries at t=0.50, but no single fixed value clears all of them — the constraints the queries impose on the cutoff conflict by a factor of 3.7. Any threshold tuned to one query is wrong for another.

That conflict is deterministic and reproduces exactly. It is the strongest result on this page, and it is the one that dictates the architecture.

Reading disposes

If no threshold generalises, the system has to actually read each candidate and decide. An LLM assessment step takes the retrieved documents and returns a per-document verdict with reasoning and a citation key.

Measured against the thresholding baseline on distractor exclusion:

Two design details that carry more weight than they look like they should. Excluded documents are shown, not hidden — for a query whose correct answer is "no such incident", the rejection is the finding, and a system that silently drops the rejects has thrown away the answer. And confidence is banded by verdict rather than taken from the raw similarity score, so a demoted document sorts below a relevant one structurally instead of by luck.

What measurement did to this page

Everything above was written after the system was built and evaluated once. Then it was evaluated properly, and five of the claims did not survive. The pattern in how they failed turned out to be worth more than any of them.

The headline was a lucky sample. The published result was 1.00 distractor exclusion. Run the identical configuration five times and it is 0.97, range 0.92–1.00. A 1.00 was always attainable on a good run; it was never a property of the system. Every LLM-dependent number in a RAG evaluation is a draw from a distribution, and reporting one to two decimals without a range reports the sample as though it were the parameter.

The benchmark was not measuring the product. The eval harness called the assessment step directly on raw retriever output. The application calls a different entry point — one that expands the query into sub-queries first. An entire pipeline stage had never been benchmarked. It hid for five phases because at top_k = 8 on an 8-document corpus both paths retrieve everything and converge. Drop to top_k = 3 and they separate immediately: recall 0.83 → 1.00, precision 0.64 → 0.82. Fixing the harness produced the first real measurement of what expansion contributes.

Small samples cannot compare two configurations. One query measured at n=20 on both retrieval paths gave 25% versus 35% failure — Fisher exact p = 0.73, not distinguishable. Separating a real 25-vs-35 point difference at 80% power needs roughly 329 trials per arm. Measuring the same query and same path again later gave 55%. Detecting that failures happen is cheap; comparing two failure rates is a different experiment with a budget an order of magnitude larger.

The obvious fix made it strictly worse. One query drives all the remaining instability: it asks for prior incidents unrelated to the theft, and the assessor accepts the disclaiming documents because — under the prompt's own definition — a document saying "here is an unrelated prior incident" does directly answer it. So I tightened the prompt to say that a disclaimer is not evidence. A/B tested at n=20 per arm:

The target query went from failing half the time to failing every time, and the model began citing the disclaimer as its justification for relevance. To say what to exclude, the added text had to use the words "unrelated to" and "no connection" — and on a query that is itself about unrelated matters, raising the salience of that vocabulary inverted the effect. Naming the trap in the prompt made it easier to fall into. Reverted; the system keeps a known 25–55% failure on that one query because the alternative is 100%.

Significance is the wrong bar for clearing a regression. The acceptance rule is not "no significant regression" but "no regression at all on any previously-clean query, whatever the p-value" — because at n=20 a real 0% → 15% regression scores p = 0.231 and a naive test waves it through. Significance is the right bar for claiming an improvement; for clearing one, the null hypothesis is working against you.

What survived

The corrections narrowed the claims rather than demolishing them:

• Evidence recall 1.00 — in 15 of 15 runs, every configuration. Now the best-supported number here.
• The query the corpus was built around never fails — the one that defeats every threshold holds 1.00 distractor exclusion in every trial.
• Reading still dominates thresholding — 0.97 against 0.00–0.71. The gap is an order of magnitude larger than the variance, so the architectural argument is untouched.
• The 3.7× threshold conflict — deterministic, unaffected by any of this.

Every correction moved a number slightly down and its specification considerably up.

Built, measured, not shipped

The fashionable next feature was an agentic retrieval loop: when a query comes back with no evidence, re-search using the assessor's own rejection reasons as feedback. It was built and measured at three retrieval depths, and found inert at all three, for three different reasons. At k=8 the first round already sees every document. At k=3 expansion has already restored recall to 1.00, so nothing comes back empty. At k=1, where recall genuinely is broken, the trigger never fires — a query that found one of its two evidence documents is not empty. The trigger is negatively correlated with the cases it should serve: it activates almost exclusively on true-negative queries, where retrying is precisely wrong.

It is not wired into the product.

The same technique was adopted elsewhere in the same week. One classification call before expansion, deciding whether the input is a question or a statement, fixes a real defect where a statement whose wording overlapped the domain was silently reinterpreted and answered at high confidence. Measured: 0 false alarms in 8 labelled queries, stable across five trials.

Same technique, opposite decisions. The retry was rejected because three measurements said it did nothing; the scope check shipped because a reproducible defect said it was needed.

The part worth carrying out

Retrieval proposes; reading disposes. Similarity search measures aboutness, and aboutness is not evidence. Any corpus containing negations, disclaimers, exclusions or superseded records — which is most real corpora — needs a step that reads the text and decides.

And one thing about the process. Every number in the first half of this page was true when written, produced by a real run against a real API. Five of them were still wrong, because a single run of a non-deterministic system is a sample rather than a measurement, and because a benchmark can quietly diverge from the thing it claims to benchmark.

The system was easier to build than to measure honestly, and the measurement is what found every real defect.