Doc-Index CLI — the read path

Read path¶

Agent → docir context "<task>" → hybrid scoring (FTS5 + semantic) + related-graph traversal → returns a small relevant subset instead of the whole docs/ folder.

Two-tier retrieval (skeleton → body). context, query, and search return skeletons — frontmatter, tags, typed related, and the owner/verified/stale fields, but not the body. The agent scans those to judge relevance and then fetches only the bodies it needs — all of them in one docir get <id> <id> ..., since process start rather than retrieval is what a read costs (issue-9509f9fa3631). Keeping bodies out of result sets is where the context savings come from; the description field (indexed and shown in listings) is what makes the skeleton enough to judge relevance.

Semantic layer (fastembed)¶

Pure FTS5 misses semantically close but lexically different matches (e.g. a query about "refresh token handling" against a document titled "session renewal strategy"). fastembed (ONNX-based, quantized, CPU-only, no external API call — consistent with the project's privacy-first stance elsewhere) closes this gap without pulling in heavy ML dependencies.

Storage¶

a document vector over title + description + body, plus one vector per ## section — see "Semantic index: every section is embedded" below for why the document vector alone was not enough. Both are BLOBs in SQLite; each row records the model that produced it, so switching embedder recomputes rather than comparing vectors of different widths. The agent-authored description gives the document vector a concise, high-signal summary to anchor on, improving retrieval over embedding raw body text alone. Brute-force cosine similarity is sufficient at this scale (hundreds to low thousands of documents) — no ANN index needed.

docir context scoring¶

Reciprocal rank fusion over the FTS5 ranking and the vector ranking, rather than replacing full-text outright — lexical matches are still valuable and cheap, and the two disagree often enough that either alone is measurably worse (benchmarks/).

score is therefore rank-derived and says only where a document placed; similarity carries the raw cosine through and is the one number with absolute meaning. --explain returns both plus the terms behind them.

With several queries — context --also — pooling still decides each document's numbers and taking turns decides the order, so no one query fills the head of the result (adr-4c21693aac55).

Where it runs¶

entirely inside the daemon (arch-1cfb1b212237, "Daemon process"), so the model is loaded once and reused; per-call added latency is on the order of single-digit to tens of milliseconds, not the cold-start cost of loading the model fresh.

Recompute triggers¶

the embedding is tied to what changed, not to every docir update call — recomputing on metadata-only changes (e.g. a status transition) would be wasted work with no benefit:

• docir add → embedding scheduled (new document).
• docir update changing title, description, or the body (--append-section, --replace-section, --replace-body) → embedding scheduled.
• docir update touching only other frontmatter fields (--status, --tags, related) → embedding left untouched.
• docir archive / deletion → the vector is removed from the index along with its FTS and relation rows, so an archived/deleted document can't resurface via similarity search or Tier 2 DRY checks.

Async recompute (not on the write's critical path)¶

an agent rarely writes a document in a single call — it does docir add then several --append-section edits over a long session. Recomputing the vector synchronously on each write would re-embed the same document repeatedly and put model inference on the write path, serializing the agent's rapid successive calls behind it. Instead, two levels of consistency are separated:

• Synchronous, immediately consistent: file write, metadata, FTS5, relations. Cheap, and search correctness depends on it — ready by the time the command returns.
• Deferred, eventually consistent (embeddings only): a content change sets an embedding_dirty flag on the row (persisted in SQLite, so it survives a daemon crash/restart) and returns immediately. A background worker inside the daemon drains dirty rows with a short debounce window (a few seconds), coalescing a burst of edits to one document into a single recompute.

Consistency window¶

between the write and the vector being ready, the document is still found via FTS (lexically) — the semantic contribution simply joins a few seconds later. A brand-new document is FTS-only until its vector lands; an updated one temporarily keeps its previous vector. On daemon restart, the worker re-embeds anything still flagged dirty, so nothing is silently lost.

Escape hatch¶

--wait-embeddings on a write, or docir embed --flush, forces a synchronous recompute when a semantically-heavy query must run immediately afterward (e.g. in tests).

Model version changes¶

if the embedding model is upgraded, existing vectors become stale — but nothing has to name them. Each row records the model that produced it, so a foreign model_id reads as dirty and docir embed --flush recomputes exactly those. A full docir reindex does it too, since it re-embeds every document it re-saves and reports the count as embeddings_recomputed; there is no separate flag for it (adr-6a4718fa7a7d).

Also powers Tier 2 DRY linting¶

(docir lint --deep): the same vectors are reused to flag content-similarity across documents, so there's no duplicate infrastructure for search vs. lint.

Semantic index: every section is embedded¶

The embedding model (bge-small-en-v1.5) reads about 512 tokens — roughly 1,900 characters of prose — and silently ignores the rest: appending text past that boundary returns a bit-identical vector. 84 of docir's own 103 documents at the time of measurement exceeded it, so more than half the corpus was absent from the semantic index while FTS5 hid the problem by covering the whole body.

So the embedding pass writes a document vector and one vector per ## section (chunk_embeddings, keyed (doc_id, ordinal)). The scorer accepts repeated ids from the semantic ranking and keeps each document's best chunk — RRF fuses rankings of documents, so the collapse has to happen before fusion, not after. The winning candidate survives the collapse, not just its score, which is what lets a ranked hit name the heading that matched (matched_section) and an agent read only that span with docir get <id> --section "<heading>".

Load-bearing details:

• MAX_CHUNK_CHARS (1200) is derived from the measured window, not chosen — and each chunk carries the document title as a prefix, which eats into the budget. A chunk allowed to overflow reintroduces the original bug one level down.
• The splitter tracks fenced code blocks, because a ## comment inside one is not a heading, and cutting there yields two invalid chunks.
• There is no second dirty flag: chunks are rewritten wholesale under the existing embeddings.dirty queue, in the same transaction.
• Tier 2 similarity linting deliberately still compares document vectors only. Chunk vectors would answer "do these two share a section", not "are these the same document".
• indexing may not import documents, so the entity is the seam: Document.embedding_chunks() hands the scheduler positional (ordinal, heading, text) triples.

Because max-pooling structurally favours documents with more sections, the recall gate on the benchmark corpus is not optional — keep it when touching ranking.

Status filtering (default visibility)¶

Closed documents are not deleted or hidden physically — they stay in docs/, in git history, and remain fully queryable. Visibility is controlled purely at the read path:

• docir context, docir query and docir search filter to active statuses by default. "Closed" is per type, not one hardcoded status: each type declares its own inactive_statuses — superseded/rejected for a decision, resolved for an issue, deprecated for architecture.
• --include-inactive widens any of them, and docir get <id> always returns the document whatever its status — an agent checking whether a similar bug was already fixed needs exactly that. (--include-resolved is the old spelling, still accepted; it named a status only two types have.)
• docir archive goes further, removing a document from active search entirely (see arch-0368cc754c15, "Archiving vs. deletion") — for volume rather than for status.

There is exactly one visibility predicate, and expansion runs both ways. DocumentService._is_visible (archived + inactive status) is called by the ranked fusion loop and by the graph-expansion pass; do not inline the check into either. They used to differ — expansion tested only archived — so a resolved issue the caller had excluded came back through a neighbour edge, and the filter that held on three read paths leaked on the fourth.

Expansion follows outgoing edges and incoming successor edges, successors first in each seed's list. A supersedes edge points from the new document to the old one, so before this the replacement sat one hop away backwards and the graph could not answer "is this decision still current?" — the question it exists for. Which kinds count is schema data (successor: true), not a hardcoded pair, so a custom kind with that shape is followed too.

Down-weighting closed documents in scoring instead of excluding them was considered and not built: a status filter is a statement about whether a document is current, and a score is a statement about relevance. Blending them makes neither answerable.

Reading across stores¶

In a multi-repo organisation the decision governing the service you are editing often lives in the platform repo, and an agent reading context in the service repo cannot see it — so it re-decides. .docir/stores.yaml, committed beside docs-schema.yaml, declares peer store homes; --store PATH (repeatable) adds one for a single invocation on top of that list, without editing the file. Relative paths resolve against this store's home, so a file one person commits works for everyone who clones the same layout, and the list is read per request rather than cached at daemon startup.

Reads federate; writes never do. Exactly four commands fan out — get, query, search, context — and that set is asserted against the dispatcher's own command list in the suite, so a new command joins by decision rather than by omission. Every write and every maintenance command runs against the local store alone.

Three properties are load-bearing:

Peers are opened read-only at the database¶

(sqlite:///file:<path>?mode=ro&uri=true), so "docir does not write to a peer" is enforced by SQLite rather than promised by a comment. This is also why peers get their own construction path: the normal container build runs migrations and creates directories, both of which write.

An unreadable peer is skipped with a warning on stderr, never an error.¶

A peer's index is derived and gitignored, so a fresh clone of it simply has none; failing the read would make one repository's state everyone's outage.

The merge sorts on similarity, never score.¶

RRF ranks within one store, so comparing two stores' scores compares the sizes of their corpora. Rows carry a store field only while federating — it is pure cost with one store, which is why the read paths never carried it before.