What similarity search cannot do

Eight case files. A detective asks a question, and the system retrieves candidates and writes a cited report. I built the corpus to be adversarial: three of the eight files are topically identical to the real evidence and explicitly state they are unrelated.

That last property is the whole problem. To an embedding model, a document saying "this breach was unrelated to the theft" is about the theft as much as any document can be. Similarity search ranks it at the top and has no way to tell it apart, because what rules it out is a claim inside the text. Its topic is exactly right.

Hybrid retrieval, and why a threshold fails

Dense retrieval alone loses here, and you can measure why. The terms that actually tell these documents apart are rare proper nouns such as wallet addresses and incident identifiers. Lexical search is good at those, and embeddings smooth them away. BM25 separates the real evidence from the distractors by 3.09. Dense similarity separates them by 0.00.

So the retriever is hybrid, and I fuse the two rankings with Reciprocal Rank Fusion instead of normalising and adding scores. Dense and BM25 scores are on different scales, and normalising them pretends they are comparable when they are not. Ranks are comparable by construction.

The next obvious idea is a threshold: the distractors rank high but score slightly lower, so cut below some value. It does not work. A threshold sweep clears 5 of 8 queries at t=0.50, but no single fixed value clears all of them, because the cutoffs the queries need conflict by a factor of 3.7. Any threshold tuned for one query is wrong for another.

That conflict is deterministic and reproduces exactly. It is the most reliable result on this page, and it is why the system reads each candidate.

An LLM reads each candidate

If no threshold works across queries, the system has to read each candidate and decide. An LLM assessment step takes the retrieved documents and returns a verdict for each one, with its reasoning and a citation key.

Against the threshold baseline, on distractor exclusion:

Excluded documents stay in the report. For a query whose correct answer is "no such incident", the rejections are the answer, and a system that silently drops them has thrown the answer away. Confidence is banded by verdict instead of taken from the raw similarity score, so a rejected document always sorts below a relevant one.

Five claims that did not hold up

I wrote everything above after building the system and evaluating it once. When I evaluated it properly, five of the claims did not survive.

The headline number was one good run

The published result was 1.00 distractor exclusion. Running the identical configuration five times gives 0.97, with a range of 0.92–1.00. A good run can produce 1.00, but that is not what the system typically does. Every LLM-dependent number in a RAG evaluation is a draw from a distribution, and reporting one to two decimals without a range presents a sample as if it were the parameter.

The benchmark was not measuring the product

The eval harness called the assessment step directly on raw retriever output. The application calls a different entry point, one that first expands the query into sub-queries, so an entire pipeline stage had never been benchmarked. It went unnoticed for five phases because at top_k = 8 on an 8-document corpus both paths retrieve everything and end up in the same place. At top_k = 3 they separate immediately: recall goes from 0.83 to 1.00, and precision from 0.64 to 0.82. Fixing the harness gave the first real measurement of what expansion contributes.

Small samples cannot compare two configurations

I measured one query 20 times on each retrieval path and got 25% failure on one and 35% on the other. Fisher's exact test gives p = 0.73, so the two cannot be told apart. Separating a real 25% versus 35% difference at 80% power needs roughly 329 trials per arm. When I measured the same query on the same path again later, it failed 55% of the time. Showing that failures happen is cheap. Comparing two failure rates is a different experiment, with a budget an order of magnitude larger.

The obvious prompt fix made it worse

One query causes all the remaining instability. It asks for prior incidents unrelated to the theft, and the assessor accepts the disclaiming documents because, under the prompt's own definition, a document saying "here is an unrelated prior incident" does answer it. So I tightened the prompt to say that a disclaimer is not evidence, and A/B tested it at n=20 per arm:

The target query went from failing half the time to failing every time, and the model started citing the disclaimer as its reason for marking the document relevant. To say what to exclude, the new text had to use the words "unrelated to" and "no connection". On a query that is itself about unrelated matters, making that vocabulary more prominent reversed the effect. I reverted it. The system keeps a known 25–55% failure rate on that one query, because the alternative is 100%.

A significance test is the wrong bar for a regression

My acceptance rule is no regression at all on any previously clean query, whatever the p-value. A test for significant regression is not enough, because at n=20 a real 0% → 15% regression scores p = 0.231 and would pass. Significance is the right bar when you claim an improvement. When you are checking that nothing got worse, the null hypothesis works against you.

What still holds

After the corrections, these still hold:

• Evidence recall is 1.00 in 15 of 15 runs, across every configuration. It is now the best-supported number on this page.
• The query the corpus was built around never fails. It is the one that defeats every threshold, and it holds 1.00 distractor exclusion in every trial.
• Reading still beats thresholding by a wide margin: 0.97 against 0.00–0.71. The gap is an order of magnitude larger than the run-to-run variation, so the case for the architecture still stands.
• The 3.7× threshold conflict is deterministic and none of this affects it.

In each case the number went down a little, and the conditions under which it holds became much more precise.

A feature I built and did not ship

The fashionable next feature was an agentic retrieval loop: when a query comes back with no evidence, search again using the assessor's own rejection reasons as feedback. I built it and measured it at three retrieval depths, and it did nothing at any of them, for three different reasons. At k=8 the first round already sees every document. At k=3 expansion has already brought recall to 1.00, so nothing comes back empty. At k=1, where recall really is broken, the trigger never fires, because a query that found one of its two evidence documents is not empty. The trigger fires almost only on true-negative queries, which are exactly the cases where retrying is wrong.

It is not wired into the product.

In the same week I did ship a related change. One classification call before expansion decides whether the input is a question or a statement. It fixes a real defect: a statement whose wording overlapped the domain was being silently reinterpreted as a question and answered with high confidence. Measured, it raised 0 false alarms on 8 labelled queries, and that held across five trials.

I rejected the retry because three measurements said it did nothing, and shipped the scope check because a defect I could reproduce said it was needed.

Takeaway

Similarity search tells you what a document is about. It cannot tell you whether the document supports the answer. Any corpus that contains negations, disclaimers, exclusions or superseded records, which is most real corpora, needs a step that reads the text and decides.

About the process: every number in the first half of this page was true when I wrote it, and came from a real run against a real API. Five of them were still wrong, because a single run of a non-deterministic system is a sample, and because a benchmark can quietly drift away from the thing it is supposed to measure.

The system was easier to build than to measure honestly, and the measurement found every real defect.