## What happens

`benchmarks/` is docir's strongest and least portable asset. It is the only published retrieval
measurement across every tool surveyed in ref-a6db21f52427, and it lives in the repository — so
a user who installs docir inherits the numbers as a claim and has no way to reproduce them, or
to find out whether their own corpus behaves the same way.

## Why it matters beyond marketing

This project has one rule about ranking changes: measure, then decide. adr-ab9c454b760c chose an
embedder on numbers and adr-d657a09b8c4a rejected a reranker on numbers. Every future ranking
question inherits that rule, and right now the instrument only exists for the maintainer, on one
corpus, in a git checkout.

That makes it a blocker rather than a nicety. A user who thinks retrieval is underperforming on
their documents cannot produce evidence, and the maintainer cannot ask for any.

## Shape of the fix

A fixture of queries with known-relevant documents, scored over the store the caller already
has. For docir the fixture is cheaper to express than for a path-addressed tool, because
expected results are document **ids** — stable under retitling, retyping and a moved file, which
is exactly what a fixture written six months ago needs to survive.

Reporting recall@k, MRR and precision@k against at least two configurations (with and without
graph expansion) is what makes a result readable, since ref-e7534f1c812d shows expansion lifts
both embedders and hides the difference between them.

## How it shipped

Shipped in `b1d4bb5` as `docir bench <fixture.yaml>`, with the three questions above answered
rather than deferred.

- **A command, not a recipe.** Writing the scoring yourself is the barrier this issue is about,
  so a documented recipe over existing output would have left it in place.
- **A plain file, not a store document.** `yaml.safe_load` parses the JSON spelling too, so one
  loader covers both, and judgments stay out of a corpus meant to hold decisions.
- **An unresolved id is neither an error nor a silent skip.** It is reported under `unresolved`
  and excluded from the judgments. Erroring makes one archived document fail the whole run;
  skipping quietly shrinks recall's denominator, which *raises* the score — a fixture rotting
  would read as retrieval improving. `StrategyScore.tasks` says how many tasks each mean
  covered, for the same reason. A task left with no resolvable ids is returned under `dropped`.

## What it reports, and why three rows

`context` is the shipped read path. `context --expand 0` removes graph expansion, which lifts
every embedder and hides the difference between them (ref-e7534f1c812d) — the pair is what
isolates the semantic signal. `search` is full-text alone, the floor anything semantic must beat.

`benchmarks/example_fixture.yaml` judges eight tasks against docir's own store and scores
`context` 0.88 recall@5 / 0.63 MRR, `--expand 0` 0.75, `search` 0.62. That ordering is the
design working on a real corpus rather than a fixture built to show it.

Two properties follow from what a fixture is. It names document **ids**, not paths, because a
retitle moves the filename and a retype moves the directory. And it does not federate: a
fixture judges ids in one store, and the score is a property of that store's read path.