A full rebuild on every upgrade, even when nothing changed

What was measured¶

Every command in the CLI, timed as a whole python -m docir process against a copy of a real 315-document store (1,326 vectors: 315 document + 1,011 section). Apple M1, docir 0.15.0, --no-daemon, output captured so each one takes the JSON path an agent takes.

This is the half benchmarks/latency.py never covered. That harness times context, search and get, and concluded the read path is fast and interpreter startup dominates — both still true. Nothing measured the write and maintenance path, which is where a command that takes a minute was sitting unreported.

Numbers¶

p50 seconds, --no-daemon, 313–315 documents.

command	s	command	s
docir check --fix	59.8	docir add / docir update	1.2–1.6
docir reindex	58.5	docir check	0.97
docir build	4.3	docir archive / docir delete	~0.7
docir lint --deep	2.4	docir version (floor)	0.5
docir query / search / get / context	1.3–1.7

Two outliers, and the second is the first: check --fix reindexes before allocating ids. Everything else is at or under 4.3 s.

Where the 58 seconds goes¶

Embedding, and nothing else. The same rebuild against the model-free hashing embedder (DOCIR_EMBEDDER=deterministic) costs 2.5 s, so the model accounts for 96%.

run	time
full rebuild, real model	58.4 s
full rebuild, hashing embedder	2.5 s
docir reindex --changed, nothing moved	1.5 s

The 1.5 s row is the point: the work was already skippable, and the flag that skips it already shipped.

Why an unchanged corpus paid for it¶

docir self upgrade ran an unconditional full reindex — including the run that reports "already the newest build", where no package moved and no file changed. A full rebuild re-embedded every document it re-saved, so all 1,326 vectors were recomputed byte-identical to the ones already stored. (It queues them now and the drain skips what matches, which is issue-77dd42e3a03a rather than this issue.)

docir check --fix paid it twice: its repair reindexes in full before allocating ids, and again after re-issuing duplicates.

The fix¶

MaintenanceService.resync reads the index build stamp before rebuilding — both modes write it, so a cheap pass would erase the evidence a full one was needed — and takes the changed-only path only when the stamp equals the running version. A genuine version move still rebuilds in full, which is what adr-6a4718fa7a7d reserves it for.

It compares equality rather than reusing stale_index_build(), which folds "never recorded" into absent so docir check stays quiet. Here unknown has to mean rebuild: a store with no stamp was last built by code that did not write one.

Repair's two rebuilds are now changed-only. Agreeing with the files is all either is for.

Measured after¶

case	before	after
docir self upgrade, stamp equal	65 s	1.5 s
docir self upgrade, stamp moved	65 s	62 s
docir reindex	64 s	64 s

Batching and thread tuning were measured and rejected¶

The obvious next move — feed the model a batch instead of one text per call — makes it slower on this hardware, because onnxruntime already saturates the cores on a single ~300-token sequence and fastembed pads every batch to its longest member.

config	ms/text
shipped default, one text per call	58.2
batch 16, length-sorted	79.3
batch 64, length-sorted	100.3
4 threads	66.1
8 threads	101.4

Do not retry either without a fresh measurement on different hardware.

What not to conclude¶

That embedding got cheaper. It did not: a release that changes the model or the chunking still costs ~62 s per 315 documents. What changed afterwards is the grain of "what did not change" — issue-77dd42e3a03a keys each vector on its own inputs, so a version move alone re-reads the metadata and re-embeds nothing; only the release that actually moves what the model reads pays the floor.

Nor that the daemon helps on the release that does re-embed. It keeps the model warm, which is worth ~0.5 s against a 58 s command — the observed 65 s through a warm daemon and 58.5 s in-process are the same number on a different day.