Repository Indexing

Indexing converts a checkout into a queryable model. It is designed to be incremental: the first pass over a large repository is expensive, every subsequent pass is proportional to what changed rather than to repository size.

Ingestion pipeline#

1. 1

   Discovery

   The tree is walked, respecting ignore rules. Vendored directories, lockfiles, build output and binary assets are classified and excluded from semantic indexing but retained in the corpus layer.

2. 2

   Classification

   Each file is assigned a language and a role — source, test, fixture, configuration, documentation, generated. Role drives how aggressively the file is indexed and whether edits to it require elevated review.

3. 3

   Parsing

   Source files are parsed to concrete syntax trees. Parsing is error-tolerant: a file that does not compile still yields a partial symbol table rather than being dropped.

4. 4

   Symbol extraction

   Definitions, references, imports and type relationships are extracted and written to the symbol graph with stable identifiers that survive reformatting.

5. 5

   Chunking and embedding

   Prose and documentation are chunked along structural boundaries rather than fixed token counts, so a chunk is a coherent unit of meaning.

6. 6

   Behavioural mapping

   Test files are linked to the symbols they exercise, using both static import analysis and, where available, coverage data from a prior run.

Incremental maintenance#

After the initial pass the index is maintained by diff. A changed file invalidates its own symbols and, transitively, any cached analysis that depended on those symbols. Unaffected regions are never re-parsed.

Monorepos#

A monorepo is indexed as a set of linked projects rather than one undifferentiated tree. Package boundaries are inferred from build configuration and become first-class edges in the symbol graph, which lets the agent distinguish an internal refactor from a change that crosses a published interface — a distinction that governs how much review a patch requires.

Generated and vendored code#

Generated files are indexed for reference but marked non-editable. When a change plan would require modifying generated output, the planner instead traces back to the generator input and proposes an edit there. This single rule eliminates a large class of agent failures where a fix is written into a file that the next build silently overwrites.