Embedding similarity cannot decide duplication

Yuqi Wang · 30 July 2026 · ~8 min · the dataset · the script

We built a deduplication gate the way most people build one: embed the entities, compare cosine similarity, flag pairs above a threshold. Then we looked at 2,362 pairs our own adjudicators had ruled on, and discovered the score was doing almost none of the work.

The setup

We maintain a registry of robotics engineering capabilities — currently about 37,000 of them. Capabilities are compiled from literature independently, one topic at a time, so the same underlying competence gets rediscovered under different names constantly. Something has to decide whether Contact Point Sampling and Contact Point Candidate Generation are the same thing.

Our gate encodes each entity into two views — an identity view over names and aliases, and a concept view over the description — and flags a pair when

s = 0.5·s_identity + 0.5·s_concept  ≥ 0.86
      or  max(s_identity, s_concept) ≥ 0.95

The 0.86 was not guessed. We measured the distribution of nearest-neighbour similarity among entities known to be legitimately distinct and put the threshold at its 95th percentile, ≈0.863. That is a defensible way to set a threshold, and we were reasonably pleased with it.

Flagged pairs then go to a two-stage adjudication chain: a non-LLM triage pass that only ever clears obvious non-duplicates, then an LLM adjudicator instructed to answer "different" under uncertainty. Every decision is written to a ledger with the score, the metric, the adjudicator and a written rationale.

That ledger is the point of this post. It is a labelled dataset, and we had never used it as one.

The measurement

Take merged as "true duplicate" and resolved as "distinct". 2,362 pairs carry a final label. Now ask the question the design implicitly answers but never tested: once a pair has been flagged, does the similarity score tell you anything about whether it is actually a duplicate?

AUC uses similarity as a duplicate detector within the flagged set; 0.5 is chance. Confidence intervals from 5,000 bootstrap resamples.

Read the first row slowly. 93% of what the gate flags is not a duplicate. And the AUC of 0.629 — while its confidence interval does exclude chance, so there is signal — is nowhere near strong enough to decide anything. The true duplicates have mean similarity 0.957; the non-duplicates have mean similarity 0.956. Those distributions sit almost on top of each other.

The second row is worse but means less: with eight positive examples the interval spans a wide range. We report it for completeness and draw no conclusion from it, beyond the obvious one — a fallback path exercised this rarely cannot be validated from production data at all.

"Just raise the threshold"

This is the first thing everyone says, including us. Here is what it buys.

Going from 0.862 to 0.943 keeps precision flat near 5.5% — still about one true duplicate in eighteen — while recall of true duplicates falls from 100% to 76%. Past 0.966 the counts collapse and recall drops to single digits.

There is no operating point at which the score alone is a usable decision procedure. This is not a tuning problem.

What actually follows from this

Our system already had a conservative adjudication stage after the threshold. We had adopted it on an argument about asymmetric costs: a false merge welds two legitimately distinct entities together and destroys a distinction that then has to be re-derived from scratch, whereas a false retention leaves one redundant row that a later pass can still clean up. Irreversible versus reversible. So bias toward retention.

That was a design preference. This measurement makes it a requirement.

The adjudication stage is not a safety margin bolted onto a working classifier. It is doing essentially all of the discrimination. The threshold is a recall filter — its job is to make sure real duplicates reach a judge, and it should be tuned for exactly that and nothing else.

Which gives the uncomfortable corollary. Any system that flags near-duplicate entities by embedding similarity and then merges automatically above a threshold would, at our operating point, be wrong more than nine times in ten. That pattern is common. If you have one in production and have never labelled a sample of its decisions, you do not know what it is doing.

Two limits we are not going to bury

Selection bias. Every pair in the ledger is a pair that already passed the threshold. So the AUC above answers a conditional question — "among pairs we flagged, does the score rank duplicates higher?" — and not "how well does the score separate duplicates from non-duplicates in general". The conditional question happens to be the operationally decisive one, because it asks whether the score should influence the decision after flagging. But it does not generalise, and we are not claiming it does.

We cannot measure recall. Nothing labelled the duplicates the gate never flagged, because nothing ever looked at them. A real recall estimate requires adjudicating a random sample of unflagged pairs, which we have not done. It is the obvious next experiment and it is not hard, just tedious.

Why we are publishing this

Partly because it is a genuinely useful result to anyone building entity resolution over technical text, and it cost us nothing to obtain — the data was sitting in a log we had been writing for months and never read as a dataset.

But mostly because of the second-order lesson, which is about instrumentation rather than embeddings. We were only able to run this experiment because the adjudication decisions had been recorded with their scores and their rationales instead of being applied and forgotten. If the merge step had simply mutated the registry and moved on — which is the obvious way to implement it — there would be no dataset, no measurement, and no way to discover that the component was behaving nothing like we assumed.

Log the decision, the input that drove it, and the reason. It is a few extra fields, and it is the difference between having a system and knowing how your system behaves.

Get the data

The ledger is published under CC BY 4.0 with the selection-bias warning attached to the file itself:

• dedup_adjudication_ledger.json — 2,993 pairs, 2,362 labelled, with scores, metrics, adjudicators and rationales
• analysis/measure.py — computes every number above, including the bootstrap intervals and the threshold sweep. Reads only files on disk: no GPU, no network, no service
• The paper, for the same result with the surrounding system described

If you run this on your own adjudication logs, we would like to know what you find — particularly if your AUC is better than ours, because then the interesting question becomes what you did differently.

Part of Entropy Box, a knowledge compiler for embodied AI. Corrections welcome: wangyuqi11and11@163.com. This post is CC BY 4.0.