Skip to content

RFC: Incrementally published standing indexes for AFT #286

Open
Open
RFC: Incrementally published standing indexes for AFT#286

Description

@randomvariable
randomvariable
opened on Aug 31, 2026

RFC: Incrementally published standing indexes for AFT

Summary

AFT maintains three standing indexes for a project root:

  • The exact trigram search index serves grep, glob, and lexical candidate discovery.
  • The semantic index stores embeddings and serves vector similarity queries.
  • The callgraph store persists resolved call edges and serves graph navigation and health analysis.

These indexes have different query contracts. They also share the same operational problem. A large repository cannot wait for a whole-corpus build before the indexes become useful. AFT must publish useful work as it completes. It must continue to serve interactive readers while the remaining corpus converges.

AFT adopts one crash-safe immutable segment substrate for all durable discovery, coverage, exact-search, semantic, callgraph, and worktree-private artifacts. The substrate provides one versioned outer envelope and publication protocol. It permits multiple typed payload encodings within that envelope.

  1. Discover the corpus through bounded, resumable directory-enumeration slices.
  2. Schedule discovery as resource-consuming work. Do not run a whole-root scan during startup or bind.
  3. Build small file batches and publish each valid segment immediately.
  4. Store every durable output in the shared versioned segment envelope.
  5. Track discovered coverage through immutable shards and a small atomic generation manifest.
  6. Query every published segment and use bounded fallbacks for uncovered or undiscovered scope.
  7. Return a concise reliability warning when the request deadline prevents a complete fallback.
  8. Compact published segments without making the index unavailable.
  9. Persist long-lived worktree changes as private immutable generations that reuse a pinned shared base.

The common substrate supplies crash recovery, format coexistence, integrity validation, reader pinning, and collection. Arrow IPC regions store naturally columnar rows. Native typed regions retain encodings that better serve exact trigram postings, FSTs, Roaring bitmaps, CSR adjacency, and similar structures. Separate segments retain independent update, query, and corruption domains. The design uses regular user-space file and memory APIs. It requires no administrator privileges, kernel modules, services, huge pages, or platform-specific asynchronous I/O API.

Motivation

The current implementations make different trade-offs:

  • search_index.rs writes one disk-backed trigram artifact. grep depends on complete candidate enumeration. A ranked full-text engine cannot replace that contract.
  • semantic_index.rs stores f32 embedding vectors and evaluates similarity by scanning the resident entries.
  • callgraph_store/mod.rs uses generation-based SQLite databases. SQLite remains useful initially as a callgraph construction and materialization engine for joins, resolution, validation, and resumable staging. It is not the common crash-recovery mechanism or the serving authority.

AFT already has root-keyed writer leases, reader markers, an executor that separates interactive and maintenance work, a process-wide cold-build concurrency limit, and atomic artifact publication. This RFC extends that base with a bounded index-work scheduler, load-aware admission, bounded memory ownership, background-only OS priority, and incremental publication. Index readiness becomes a coverage value rather than a binary state.

Goals

  • Make the first indexed results available after the first completed discovery and build batches.
  • Converge large repositories through bounded incremental discovery and indexing work.
  • Preserve exact grep and glob completeness when bounded discovery and fallback finish.
  • Tell the calling agent what remains trustworthy and what action to take when a result is partial.
  • Keep interactive reads available during discovery, indexing, refreshes, and compaction.
  • Resume pending discovery and index work after scheduler rotation or process restart.
  • Prioritize files that are likely to satisfy the active query.
  • Prevent corrupt or stale segments from becoming visible.
  • Give every durable index, discovery, coverage, and worktree artifact the same crash-recovery and format-evolution contract.
  • Reduce semantic vector residency without weakening correctness silently.
  • Remove SQLite connection contention from callgraph read queries while retaining SQLite only where it remains useful for construction and staging.
  • Support Linux, macOS, and Windows as an unprivileged user.
  • Keep every index cutover independently reversible.
  • Support long-lived worktrees with many changes without copying shared index segments or relying on volatile in-memory overlays.

Non-goals

  • This RFC does not require whole-corpus indexing before serving queries.
  • This RFC does not replace exact trigram search with Tantivy or BM25.
  • This RFC does not select an approximate nearest-neighbor algorithm or a vector quantizer.
  • This RFC does not require replacing the SQLite callgraph construction engine in the first cutover. SQLite state must remain reconstructible from source plus published coverage and segments.
  • This RFC does not define one physical file that mixes text, vectors, and graph data. It defines one outer segment substrate with typed payload regions.
  • This RFC does not require a fixed 4 KiB mapping or I/O alignment.
  • This RFC does not promise memory or latency reductions before a reproducible baseline exists.

Required query contracts

Exact search

The executor queries the immutable segments referenced by the active project or worktree manifest. It then searches uncovered files through the bounded filesystem fallback. If the fallback finishes, the tool returns the same set of files and matches as the current exact path with complete: true. If the deadline or walk budget stops the fallback, the tool returns the indexed matches with complete: false. It states that returned matches are valid, absence is not reliable, and the agent should narrow the scope if absence matters. Segment ranking can order work. It cannot suppress candidates while the result claims completeness.

Semantic search

The first semantic cutover preserves exact cosine ordering across all published vectors. Files without a current embedding remain internal coverage gaps. aft_search can use lexical candidates as a fallback lane. A partial semantic response states that its ranking covers only indexed files and is not a global top-K guarantee. An approximate or quantized path can ship later only with the exact f32 path as an oracle and fallback.

Callgraph queries

The projection preserves the data required by callers, call trees, impact analysis, traces, dead-code projection, source locations, resolution quality, and test-origin filtering. Queries use published projection segments first. They use the existing local per-file graph for uncovered or stale files when that fallback can answer the operation. Otherwise, the response states that returned edges are valid but the traversal ended at an unindexed frontier. The SQLite store remains the authority during projection construction.

Incremental discovery and publication protocol

Corpus discovery is a first-class scheduler job. AFT does not enumerate, stat, hash, or read every file during startup, bind, or generation activation. These paths open only the current pointer and the referenced immutable metadata. The scheduler admits a bounded discovery slice under the lowest criticality class. Each slice has limits for directory entries, cumulative path bytes, metadata operations, and wall time. It applies ignore and mount-boundary rules before metadata or content access. Discovery does not read file contents.

The durable discovery frontier stores pending directory paths and completed-directory markers. A running process can retain an in-directory iterator. After a restart, AFT can repeat the interrupted directory enumeration, but immutable discovery shards and stable file identities make repeated entries idempotent. This avoids a platform-specific dependency on persistent directory cookies. Watcher events invalidate affected directory and file coverage after the initial frontier has reached them.

Each completed discovery slice writes an immutable discovery shard. A discovery shard contains the newly observed paths, source identity data available without content reads, exclusions, and child directories added to the frontier. It becomes input to bounded index batches as soon as it publishes. Discovery and indexing therefore overlap. AFT can publish useful index results while most of the repository remains undiscovered.

Each durable output uses the same immutable segment substrate. This includes discovery shards, coverage shards, exact-search segments, semantic segments, callgraph projections, and worktree-private objects. A small versioned generation manifest references these objects. It does not rewrite a repository-sized per-file table after every batch.

The outer segment envelope contains:

  • A stable magic value, envelope version, format family, and format-family version.
  • A typed region registry with encoding identifier, region version, offset, length, alignment, and checksum for each region.
  • A segment content identity plus source, configuration, and schema fingerprints.
  • A bounded metadata footer that the reader validates before it maps or decodes payload regions.

Arrow IPC is a region encoding, not the whole segment format. Use it for coverage and path tables, semantic vectors in the first implementation, symbol metadata, and edge attributes. Use specialized versioned regions for trigram postings, FST dictionaries, Roaring bitmaps, CSR offsets and adjacency, Bloom filters, and other structures when their query contract requires them.

stateDiagram-v2
    [*] --> DiscoveryPending
    DiscoveryPending --> Discovering: admit bounded directory slice
    Discovering --> DiscoveryPending: persist frontier and yield
    Discovering --> Pending: publish discovered paths
    Pending --> Building: admit bounded index batch
    Building --> Published: validate and publish segment
    Building --> Failed: publish bounded failure record
    Published --> Stale: watcher invalidates source
    Stale --> Pending: schedule replacement
    Failed --> Pending: retry eligible failure
    Published --> Compacted: replace segment set
    Compacted --> Collected: no pinned generation references it
Loading

The generation manifest contains:

  • A format version and index kind.
  • The project-scope key and configuration fingerprint.
  • The discovery-frontier generation and inventory_complete state.
  • A sorted list of immutable discovery-shard and coverage-shard identifiers.
  • A sorted list of published data-segment identifiers.
  • Per-shard and per-segment lengths and checksums.
  • Aggregate discovered, indexed, pending, stale, failed, and excluded counts for operator telemetry.

Discovery and coverage shards contain the repository-sized state:

  • Stable file identities and source fingerprints.
  • File states for indexed, pending, stale, failed, and excluded.
  • Bounded reasons for failed or excluded files.
  • Directory-frontier additions and completed-directory markers.

Publication uses a versioned compare-and-swap protocol for each discovery, index, or compaction batch:

  1. Read the current generation manifest and its coordination token.
  2. Select one bounded discovery, pending-index, stale-index, or compaction batch.
  3. Write one or more immutable output shards or segments without changing the current manifest.
  4. Validate the outer envelope, typed-region bounds and versions, checksums, source fingerprints, index-specific invariants, and the expected identities of any retired objects.
  5. Write and synchronize every output object.
  6. Re-read the active coordination token under the publication lock. Reject the commit when it no longer matches.
  7. Write a new immutable manifest with the next monotonic generation. Write and synchronize it before activation.
  8. Atomically replace a small cross-platform current-generation pointer file and synchronize its parent directory.
  9. Install the new reader handles. Retire old objects only after no pinned manifest references them.

The protocol uses stale-token rejection, monotonic generations, and one atomic manifest transaction for multi-output replacement. A crashed writer can leave unreferenced objects. Startup recovery identifies them by reachability from the current pointer and collects them after the retention window. A failed compare-and-swap never removes the active generation.

This sequence does not wait for whole-corpus discovery or indexing. A valid first segment becomes queryable immediately. Later shards and segments increase known coverage monotonically unless a watcher event marks coverage stale. A failed batch does not hide previously published work.

Compaction uses the same transaction. It writes all replacement segments first. It validates all retired identities. It commits one manifest generation that adds every replacement and removes every retired segment. Reader pins protect the old generation until collection is safe.

The current-generation pointer uses an atomically replaced regular file. It does not require symbolic links. The replacement helper implements and tests the platform-specific durability sequence. Mapping offsets use the runtime mapping allocation granularity. Windows commonly requires view offsets aligned to the system allocation granularity, which can be larger than the hardware page size.

A reader opens one generation manifest and pins that version for the full query. It can read many discovery shards, coverage shards, and data segments from that manifest. It does not combine membership from different manifest versions.

Crash recovery belongs to this shared substrate. A crash can lose the active in-flight batch, which the durable frontier and coverage state can schedule again. It cannot lose or partly expose a published batch. Recovery opens the current pointer, validates the referenced manifest and segment envelopes, isolates corrupt or unsupported objects, restores reader-safe generations, and collects unreachable objects after the retention window. The scheduler queue order is reconstructible and does not need a write on every scheduling decision.

Durable worktree generations

The current in-memory worktree overlay is not sufficient for a long-lived worktree. It loses local index state after process restart or root eviction. It also cannot represent durable semantic and callgraph changes. AFT replaces it with a worktree-scoped copy-on-write generation.

Each worktree keeps a small durable manifest under its existing canonical-path project-scope key. The manifest pins one exact shared base generation and references private immutable discovery, coverage, exact-search, semantic, and callgraph segments. Shared base segments remain immutable and are not copied into the worktree scope.

Changed paths, added paths, and deletion tombstones in the worktree coverage shards supersede records for the same paths in the pinned base. A query pins one worktree manifest for its full duration. That manifest directly names both shared and private segments, so the reader does not reconcile a separate mutable overlay during the query.

flowchart TB
    B[Shared base generation] --> W[Worktree manifest]
    D[Private discovery and coverage shards] --> W
    E[Private exact-search segments] --> W
    S[Private semantic segments] --> W
    C[Private callgraph segments] --> W
    W --> Q[Pinned worktree query snapshot]
    T[Path supersession and deletion tombstones] --> Q
Loading

Watcher events enqueue bounded private batches through the same scheduler and publication compare-and-swap protocol as repository generations. A restart opens the worktree pointer and resumes its durable frontier. It does not replay all worktree changes or scan the full root. A watcher journal gap marks affected worktree coverage incomplete and schedules bounded reconciliation. Valid private segments remain queryable while reconciliation proceeds.

Advancing the shared base is an explicit bounded reconciliation operation. AFT can reuse a segment from the newer base only when the worktree has not superseded any path covered by that segment. It retains private segments and any still-required old base segments otherwise. The operation publishes one new worktree manifest atomically and never mutates a shared generation.

When a worktree disappears, AFT marks its scope for collection. Reader pins and the retention window protect every referenced base and private generation. Collection removes private manifests and segments only after no live reader or worktree manifest references them.

The volatile worktree overlay remains only as a migration fallback for one release. The durable worktree generation must cover exact search, semantic search, and callgraph queries before AFT removes that path.

Partial results and fallbacks

Every partial response must answer three questions for the calling agent:

  1. What can the agent trust?
  2. What conclusion must the agent not make?
  3. What action is useful now?

Repository-scale counts do not answer these questions. The model-facing response must not include file lists, coverage counters, percentages, internal epochs, segment identifiers, or scheduler telemetry. Those values belong in aft_status, health output, and diagnostics for an operator.

The structured response uses a small result-semantics contract:

{
  "complete": false,
  "partial": {
    "reason": "index_converging",
    "guarantee": "returned_matches_valid",
    "limitation": "absence_not_reliable",
    "action": "narrow_scope_if_absence_matters"
  }
}

The server renders these enums into one short, tool-specific warning:

  • Exact search: Partial exact search: returned matches are valid. Absence is not reliable because indexing is still converging and fallback reached its bound. Narrow the path if absence matters.
  • Semantic search: Partial semantic search: use these results as candidates, not a global top-K. Narrow the path or retry later if ranking completeness matters.
  • Callgraph: Partial callgraph: returned edges are valid. The traversal stopped at an unindexed frontier. Do not infer that no caller or path exists.

The warning appears after the useful results so it does not displace them. It uses stable enums in the structured payload and direct prose in agent-facing text. A tool can omit action when no action can improve the result. It can use inspect_status when the index is healthy but source failures caused the gap.

Index state Tool behavior Agent-facing semantics
All applicable files are current Query the immutable segments in the active project or worktree manifest. complete: true; absence is reliable.
Fallback covers all gaps Merge indexed and fallback work. complete: true; absence is reliable.
Index convergence exceeds the fallback bound Return all valid work completed before the bound. Returned items are valid; absence is not reliable; narrow scope if absence matters.
Corpus discovery is incomplete Query published data and perform only the query-scoped traversal admitted by the interactive fallback budget. Returned items are valid; undiscovered files can change absence or ranking.
Source failures affect the request Return usable results from other files. Returned items are valid; inspect status if the missing result matters.
No safe fallback exists Query covered data only. Returned graph or ranking is a partial candidate set, not a negative result.
The supplied scope resolves to no files Return no result items. no_files_matched_scope: true; this is not an index-coverage warning.

The tool must not hide a coverage gap behind success: true with an empty result. success: true means that the tool performed useful work. The complete field says whether the full applicable scope was covered.

The fallback shares the interactive request deadline and has explicit directory-entry, metadata-operation, content-byte, and wall-time limits. It never starts an unbounded file scan. Exact search uses published path metadata first, then performs query-scoped discovery only within the requested path. Callgraph navigation starts at the requested frontier. Semantic search embeds or ranks lexical candidates when the remaining deadline permits. This order improves early results. It does not change the meaning of complete: true.

Scheduler and resource contract

The standing-root scheduler is the only entry point for incremental index work, including directory enumeration and metadata discovery. It uses deficit round-robin across roots and explicit criticality classes: health and control, interactive query and query-triggered repair, watcher repair, compaction, and background discovery. Health and control are always admitted. Pressure sheds new work from the lowest class upward. Each root receives a measured byte-and-time service quantum. Before dispatch, a discovery slice is bounded by directory entries, path bytes, metadata operations, and wall time. An index slice is bounded by input bytes, files, and wall time. After each slice, the scheduler charges actual elapsed CPU time, bytes read, bytes written, directory entries, and metadata operations. An unfinished root returns to the queue with its remaining deficit.

flowchart LR
    R[Pending work by root and class] --> Q[Deficit round-robin scheduler]
    Q --> L{Host load admits background work?}
    L -- No --> H[Pause background admission]
    H --> Q
    L -- Yes --> M{Bounded work lease available?}
    M -- No --> Q
    M -- Yes --> A{Work permit available?}
    A -- No --> Q
    A -- Yes --> S[Run one bounded discovery or index slice]
    S --> V{Slice valid?}
    V -- No --> F[Record bounded failure]
    V -- Yes --> P[Publish immutable shard or segment]
    F --> C[Charge measured cost]
    P --> C
    C --> D{Pending work remains?}
    D -- Yes --> Q
    D -- No --> I[Root converged]
Loading

The scheduler remains work-conserving within the admitted budget. It combines I/O token buckets with demand-aware deficit round-robin. Separate accounting covers content bytes, directory entries, metadata operations, and publication writes because a directory walk can consume substantial CPU and storage I/O without reading file contents. Refill rates come from measurements on the target repositories and storage classes. Foreground tool latency controls the background share. The controller excludes latency samples taken while another preemption mechanism is active, because those samples cannot identify background indexing as the cause. PSI and native platform pressure signals act as safety inputs. They are not the only throughput controller.

The transport and executor read periodically sampled atomic controller state. They do not collect host metrics on an admission path. A work slice releases its permit after it publishes or yields.

Each subsystem has a stable memory consumer identity, an observer that reports bytes used, and a separate budget owner for memory retained by that subsystem. The initial consumers are exact search, semantic search, callgraph, interactive fallback, and compaction. Configuration validates floor <= cap, validates any pin within that range, and rejects aggregate floors or pins above the total budget.

Memory admission is based on bounded ownership, not a prediction of total allocation cost. Before a batch starts, the scheduler bounds the work unit by input bytes, file count, concurrency, embedding count and dimension, and spill or arena capacity. The subsystem acquires a lease for known inputs and fixed-capacity retained outputs. It charges each owned capacity increase before try_reserve or retention. When the next charge would exceed the cap, the batch spills, publishes, splits, or yields. RAII ownership releases the charge on success, failure, or cancellation. Parsing, native libraries, allocator metadata, thread stacks, and mapped-page residency are not fully chargeable through this ledger. File-size and concurrency caps bound their transient exposure. Process RSS, native allocation telemetry, and mapped bytes remain independent safety signals.

Benchmark calibration can derive conservative expansion factors for choosing batch sizes and reserve headroom. An expansion factor must not certify that an arbitrary future allocation is safe. The first release uses measured static floors and caps. Adaptive budget transfers require a separate benchmark because best-effort setters and benefit estimation create another feedback loop.

The initial user-only resource modes are balanced and performance. balanced reduces the background byte share or pauses new low-criticality batches when foreground latency or sustained CPU, memory, or I/O pressure crosses a measured threshold. It uses hysteresis for recovery. Linux can use PSI and power-supply state. macOS and Windows need native equivalents before those signals become blocking. Missing signals are reported as unavailable. They do not permanently pause indexing. performance bypasses load-based pauses. It does not bypass memory charges, queue bounds, publication compare-and-swap, or build-concurrency limits.

Explicit query fallbacks remain interactive work. They use the request deadline and bounded fallback budget. They do not wait behind background convergence. Query demand can raise a pending file from background convergence to the query-demand queue. It cannot bypass memory admission or correctness gates.

The executor keeps interactive work at normal OS priority. It demotes only JobClass::Maintenance execution to background CPU and I/O priority and restores the exact previous thread policy after the job. It does not demote interactive heavy initialization. The SubC transport thread performs no corpus walk, allocator scan, index merge, or pressure sampling. Health and heartbeat control frames use a bounded priority lane so sustained control traffic cannot starve tool data.

The scheduler exports operator telemetry for queued work by class, current root, admitted resource mode, pause reason, charged and observed memory, process RSS, batch CPU time, bytes read and written, and rejection counts. Model-facing tool responses receive only the reliability semantics in the previous section.

Service state contract

Resource pressure and incomplete indexes change background permissions. They do not make the service binary healthy or unavailable. AFT exposes one explicit service state with defined permissions:

State Query Bounded fallback Publish Compact Discover
healthy Full As needed Allow Allow Allow
converging Published coverage Allow Allow Allow when admitted Allow
pressure_paused Published coverage Interactive only Finish an already-safe commit Stop Stop
partial_coverage Published coverage Allow Allow repair Stop unrelated work Allow repair discovery
corrupt_segment_isolated Exclude corrupt segment Allow Allow replacement Stop affected compaction Allow repair discovery
read_only_cache Pinned valid coverage Read-only fallback Stop Stop Stop

State transitions are exhaustive and appear in health telemetry. Queries remain available from valid published coverage during convergence, pressure, and isolated corruption. A state never permits a corrupt segment to enter a query snapshot.

Exact search design

The exact trigram index becomes a set of immutable segments. A normal project manifest references shared segments. A worktree manifest references a pinned shared base plus private segments and path tombstones.

flowchart TB
    W[File watcher changes] --> B[Bounded private batch]
    C[Bounded discovery shards] --> B
    B --> P[Publish immutable shared or worktree segments]
    P --> E[Exact query executor]
    U[Uncovered or undiscovered scope] --> F[Bounded query fallback]
    F --> E
    E --> M[Deterministic union and verification]
    M --> D{Full scope covered?}
    D -- Yes --> R[complete true]
    D -- No --> G[partial with reliability warning]
Loading

Each segment stores only the data required by exact candidate enumeration:

  • A sorted file table.
  • File metadata and content ranges.
  • A trigram dictionary.
  • Compressed posting lists.
  • Segment-local checksums and source coverage.

The query executor probes every segment referenced by one pinned project or worktree manifest. Worktree-private path records and tombstones supersede matching base records. It then uses the fallback for files that the coverage manifest marks pending, stale, or failed. It forms a deterministic union before content verification. Segment scores can rank work, but they cannot cap work while the result claims completeness.

Compaction rebuilds replacement segments from already published source records. It does not block new batch publication. The coverage manifest changes atomically when either an incremental segment or a compacted replacement is valid.

Tantivy is a candidate for a separate ranked lexical lane in aft_search. It is not a candidate for exact grep completeness. If AFT adds that lane, the hybrid query layer should combine its rank with semantic rank through rank fusion rather than compare unrelated BM25 and cosine score scales directly.

For rank lists M, weighted reciprocal rank fusion is:

$$ \mathrm{RRF}(d) = \sum_{m \in M} \frac{w_m}{k + r_m(d)} $$

The implementation must tune k and the lane weights on an AFT retrieval corpus. This RFC does not adopt the commonly cited value k = 60 as an unmeasured product constant.

Semantic index design

The semantic index becomes immutable vector segments plus the current embedding pipeline.

flowchart TB
    E[Embedding batch] --> P[Publish vector segment]
    Q[Query embedding] --> S[Search published vectors]
    P --> S
    U[Files without current vectors] --> L[Lexical fallback candidates]
    S --> G[Global exact score merge]
    L --> G
    G --> D{Semantic coverage complete?}
    D -- Yes --> K[complete true with top K]
    D -- No --> X[partial candidate ranking]
    K --> H[Optional heterogeneous rank fusion]
    X --> H
Loading

The first implementation stores the same normalized f32 vectors in incrementally published segments. It computes the same similarity function and performs a global score merge across published vectors. A newly embedded batch becomes searchable without waiting for the rest of the corpus. This step isolates publication, scheduling, and residency changes from approximate-search error.

A later benchmark can compare open-source approximate and compressed options such as DiskANN, HNSW implementations, RaBitQ, and TurboQuant. Each candidate must satisfy these gates:

  • The license is compatible with AFT distribution.
  • The Rust integration is maintained or small enough to own.
  • The file format is versioned and validated.
  • Recall is measured against the exact f32 oracle on code-search queries.
  • Build memory, build time, query latency, index bytes, and peak process memory are measured.
  • A corrupt or unsupported segment falls back to exact search or returns a source-failure reliability warning.
  • Scalar code remains available on CPUs without the candidate SIMD path.

Bit width alone is not an end-to-end memory claim. A b-bit code uses b / 32 of the raw f32 vector payload before graph, identifier, alignment, and allocator overhead. A 3-bit code therefore reduces only the raw vector payload by 1 - 3 / 32 = 90.625%. It does not establish the process RSS reduction.

Filtered semantic queries should first use ordinary pre-filtering, post-filtering, or candidate expansion with measured behavior. This RFC does not adopt an unverified drift-guided walk, anchor atlas, or fixed centroid formula.

Callgraph query projection

SQLite remains an initial mutable construction and validation store. It is not a recovery boundary. Each validated file batch publishes a read-optimized projection segment through the shared immutable substrate. SQLite checkpoints carry the same source and configuration fingerprints as the generation they construct. AFT can discard and reconstruct them from source plus published coverage and segments.

flowchart LR
    P[Parser and resolver batch] --> D[SQLite staging rows]
    D --> V[Batch validation]
    V --> X[CSR segment builder]
    X --> C[Publish coverage manifest]
    C --> Q[Read-only query]
    O[Local graph fallback] --> Q
    U[Uncovered graph frontier] --> O
Loading

Each projection segment uses dense vertex ordinals within that segment. The outer envelope registers specialized CSR topology regions and Arrow IPC attribute regions:

  • A sorted symbol-key to segment-local vertex-ordinal region.
  • Forward CSR offsets and edge ordinals.
  • Reverse CSR offsets and edge ordinals.
  • Arrow IPC columns for edge resolution, source location, call kind, and test origin.
  • An Arrow IPC symbol table for paths, ranges, and display names.
  • Cross-segment targets as stable symbol keys until compaction rewrites them.

A projection query maps the requested symbol to a segment and ordinal. It reads the required adjacency range and decodes only the required attribute columns. It follows stable cross-segment keys through the manifest symbol directory. Readers do not acquire the SQLite builder connection.

The first projection should use plain fixed-width or existing bit-packed arrays. Partitioned Elias-Fano is a benchmark candidate for monotone offset and adjacency lists. A PGM index is a benchmark candidate for symbol-key lookup only if it beats binary search, an FST, or a minimal perfect hash under the real symbol distribution. Neither mechanism is required for the first cutover.

SQLite and the current local callgraph provide temporary fallback data while projection coverage converges. If these sources cover the requested graph frontier, the tool can return complete: true. If they do not, the tool returns the covered graph with complete: false and states that returned edges are valid but absence is not reliable beyond the unindexed frontier. Worktree-local projection segments serve changed files before shared-base reconciliation. Their coverage remains bounded by the worktree manifest. SQLite can be removed from a serving path once the projection and bounded local fallback satisfy that path's contract.

A neighborhood cache is optional. If graph profiles show repeated adjacency decoding as material, compare no cache, the current cache, SIEVE, and another established policy under interactive navigation and repository-wide scan workloads. Do not select a policy from paper results alone.

Physical layout

The indexes use separate segments and directories because they have different update, query, and corruption domains. Every segment uses the same versioned outer envelope and manifest protocol.

flowchart TB
    C[Active project and worktree manifests]
    C --> B[Pinned shared base generations]
    C --> W[Worktree-private generations]
    B --> E[Versioned segment envelopes]
    W --> E
    E --> A[Arrow IPC column regions]
    E --> N[Native typed index regions]
    E --> M[Checksums identities and fingerprints]
    W --> X[Path supersession and tombstones]
Loading

The envelope uses explicit typed-region offsets, lengths, versions, checksums, and alignment. The writer can align large regions to a runtime-selected mapping granularity. The reader must still accept valid unaligned logical subranges by mapping an aligned enclosing region and applying an internal offset. Reader dispatch uses the envelope and region versions. Old and new formats can coexist while manifests reference both. An unsupported required region isolates that segment instead of making the generation partly readable without warning.

Alternative substrate comparison

The implementation must compare the proposed purpose-built substrate with the current DuckDB, Delta Lake, and Lance options before committing to a storage implementation. These components are not one coherent stack. DuckDB is a query engine with native storage and extensions. Delta Lake and Lance are separate versioned table formats. DuckDB FTS and VSS overlap with Lance FTS and vector indexes.

Option What it provides Mismatch with AFT Disposition
DuckDB native storage with DuckDB FTS and VSS Embedded SQL, transactions, BM25 retrieval, and HNSW vector search Native read-write ownership is process-scoped. FTS is ranked token search rather than exact grep enumeration. VSS is experimental, keeps the complete index outside DuckDB's memory limit, requires the index to fit in RAM, warns of incomplete WAL recovery, rewrites the complete persistent index at checkpoints, and reloads it wholly after restart. Reject as the serving substrate. Retain DuckDB SQL and exact vector scans as optional benchmark tools.
DuckDB with Delta Lake Versioned Parquet tables, time travel, checkpoints, idempotent append coordination, and analytical scans Delta does not provide the exact-search, semantic-index, callgraph, partial-coverage, or worktree index lifecycle. Adding FTS or VSS requires a separate DuckDB materialization and index authority. Its transaction log duplicates the local generation manifest without replacing index-specific segments. Reject for local AFT index storage.
DuckDB with the Lance extension SQL access to Lance datasets plus Lance vector, FTS, and hybrid search DuckDB adds a second execution and extension boundary on the latency-critical serving path. The extension's documented binaries omit macOS x86_64 and Windows ARM64. It does not remove the need for AFT-specific exact trigram and callgraph indexes. Keep as a benchmark and inspection adapter, not the runtime authority.
Lance through its Rust crates Immutable versioned manifests, optimistic transactions, partial index coverage, indexed-plus-unindexed query plans, progressive index loading, stable row identifiers, shallow clones through base paths, vector indexes, FTS partitions, a built-in scalar n-gram index, and index compaction Lance FTS is ranked token search. Its built-in n-gram index lowercases and ASCII-folds alphanumeric character trigrams, so it does not provide byte-exact grep. The public extension registry currently accepts vector extensions and rejects scalar extensions. AFT still needs exact byte-trigram and callgraph CSR index families. Shallow clones need an AFT mapping from source-path supersession and tombstones. Cross-platform binary and dependency cost remain unmeasured. Prototype as the principal alternative to a purpose-built substrate. Adopt only if both custom index families can ship without a long-lived fork and all AFT contracts pass.
DuckDB, FTS, Delta, VSS, and Lance together The union of the features above Delta and Lance duplicate versioned table ownership. DuckDB FTS/VSS and Lance FTS/vector duplicate index ownership. Two catalogs, two transaction protocols, and overlapping indexes create more recovery and invalidation boundaries than AFT needs. Reject as an assembled stack.

DuckDB FTS now has an optional trigger-maintained incremental mode for inserts and deletes. This improves its suitability for ranked lexical experiments, but it does not satisfy exact byte or regex enumeration and it keeps the index inside a mutable DuckDB table authority. DuckDB VSS does not satisfy this RFC's bounded residency or crash-recovery requirements.

Lance is the only alternative that substantially overlaps the proposed common substrate. Its index model already permits immutable independent segments with explicit fragment coverage. A query can combine indexed fragments with a scan of unindexed fragments. Its transaction model publishes immutable manifests and allows an index to cover only part of a dataset. Its built-in scalar n-gram index also provides useful implementation precedent for bounded spill, merge, on-demand posting reads, remapping, and candidate verification. The tokenizer and scalar-extension boundary prevent direct reuse for AFT exact search today.

The Lance prototype must use the Rust crates directly. It must model coverage and path rows, exact f32 vectors, partial index fallback, and a long-lived worktree shallow clone. It must determine whether upstream Lance can expose stable scalar and graph index extension points. A prototype that requires patching Lance or wraps opaque AFT files does not pass Stage 0 because it retains two lifecycle implementations without gaining upstream query planning or index maintenance.

The benchmark must compare Lance-direct and the purpose-built substrate on the same revisions and fixed-arrival workload. It must record time to first valid result, convergence latency, foreground tail latency, CPU time, directory and content I/O, peak RSS, mapped bytes, retained and temporary storage, write amplification, restart recovery, compaction cost, worktree update cost, binary size, dependency build time, and supported target coverage.

Licensing does not decide the architecture. DuckDB and its FTS, VSS, and Delta extensions use the MIT license. Lance and the DuckDB Lance extension use Apache-2.0. Both are compatible with evaluation and redistribution subject to their notice requirements. The material risks are operational maturity, extension stability, dependency weight, and whether AFT's custom index families can remain upstream-compatible.

Candidate mechanism disposition

Mechanism Disposition Reason
Shared versioned segment substrate Adopt Gives every durable artifact one crash-recovery, validation, evolution, pinning, and collection contract.
Arrow IPC typed regions Adopt for columnar payloads Fits coverage, path, vector, symbol, and attribute rows without forcing index-specific structures into Arrow.
Specialized typed regions Adopt Exact postings, FSTs, bitmaps, and CSR topology need query-specific encodings under the common envelope.
Incrementally published exact trigram segments Adopt Makes each completed batch searchable and preserves bounded exact fallback.
Versioned coverage manifest Adopt Gives every query an explicit indexed, pending, stale, failed, and undiscovered scope.
Worktree copy-on-write generations Adopt Persist local exact, semantic, and callgraph changes while reusing immutable shared base segments.
SQLite callgraph construction store Retain initially Supports construction, joins, validation, and resumable staging but is reconstructible and not a serving or recovery authority.
Incremental columnar CSR projections Adopt in stages Moves validated graph batches onto the read path without forcing an immediate constructor rewrite.
Incremental exact f32 semantic segments Adopt first Makes completed embeddings searchable and preserves the correctness oracle.
Lance Rust substrate Stage 0 competing prototype It already implements much of the manifest, partial-coverage, progressive-load, vector, FTS, compaction, and clone lifecycle. Adoption depends on upstream custom exact-search and callgraph index integration plus all AFT gates.
DuckDB runtime authority Reject Its process ownership and extension boundaries add mutable state and latency without satisfying exact search or callgraph serving.
Delta Lake substrate Reject It duplicates versioned table ownership and does not supply AFT index semantics.
DuckDB VSS Reject Its persistent path is experimental, unbounded by DuckDB memory limits, wholly resident, and wholly rewritten at checkpoint.
DuckDB and Lance adapters Benchmark and inspection only They provide useful SQL, exact-vector, ranked lexical, and operational comparison surfaces without owning AFT serving state.
Tantivy Benchmark for ranked lexical aft_search only BM25 ranking does not satisfy exact grep enumeration.
RRF Benchmark for heterogeneous ranking lanes Rank fusion avoids direct comparison of BM25 and cosine scales.
DiskANN or HNSW Benchmark after exact segmentation Approximation needs measured recall and build-cost evidence.
TurboQuant or RaBitQ Benchmark after exact segmentation Compression claims must include graph and process overhead.
Partitioned Elias-Fano Benchmark for monotone graph columns Benefits depend on the real adjacency distribution.
PGM index Benchmark for symbol lookup The current key distribution may favor simpler structures.
SIEVE Benchmark only if cache profiles justify a cache Cache policy is independent from the projection format.
Lance built-in n-gram index Benchmark as a candidate engine It has immutable scalar-index segments, Roaring postings, bounded spill and merge, on-demand posting reads, remapping, and candidate-only semantics. Its fixed lowercase ASCII-folded alphanumeric character trigrams are not byte-exact grep semantics.
greplm-core exact segments Benchmark as a candidate engine It combines immutable mmap-backed FST and Roaring segments with exact verification and incremental updates. Its multi-file metadata, cache, and tombstone lifecycle does not replace the common crash-safe substrate.
qndx sparse n-grams Benchmark against byte trigrams Its subset invariant has property tests and its hybrid postings are versioned and checksummed. Its sparse extraction has quadratic worst-case work, its durable update path rebuilds, and its published performance figures are author measurements rather than AFT evidence.
tgrep bounded external builder Reuse the construction pattern Its fixed-byte arena, compact spill segments, shared merge read-ahead budget, k-way merge, large-corpus tests, and mapped/private memory telemetry are directly relevant. Its complete index format and server lifecycle are not the AFT publication protocol.
sux or sbits succinct graph columns Benchmark for CSR regions sux is the stronger production precedent and supports Elias-Fano, unaligned access, and mmap through epserde. sbits includes Partitioned Elias-Fano but is a small static-structure implementation. Neither supplies incremental graph publication.
epserde internal typed-region encoding Benchmark behind envelope validation It can map immutable succinct structures with minimal copying, but deserialization is unsafe and it performs no data validation or padding cleaning. It cannot be the outer envelope or trust boundary.
culpert allocation sampling Optional diagnostic arm only It is experimental and observes Rust GlobalAlloc traffic only. It cannot account for SQLite, tree-sitter, ONNX, mapped residency, fragmentation, CPU, or lock contention.
mmap-io Reject as a required substrate dependency It does not provide generation publication or recovery. Its mutable and atomic mapping features are unnecessary for immutable segments, and external concurrent modification can make mapped access undefined.
psi crate Reject as a required dependency It is a small Linux-only wrapper and does not implement cgroup-v2 support. AFT already has a focused direct PSI sampler and still needs non-Linux signals.
fastgrep Reject as architecture precedent It uses all logical CPUs by default, freshness based on mtime plus size, intentional file omission, and synthetic small-corpus benchmarks. Those policies conflict with AFT completeness and resource contracts.
Fixed 4 KiB mappings Reject Mapping allocation granularity differs by platform.
One combined text/vector/graph file Reject The indexes have different update, query, and corruption domains.
Whole-corpus readiness gate Reject Large repositories must return indexed and fallback results during convergence.
Drift-guided filtered walk and anchor atlas Reject from this RFC The cited evidence does not establish this design for filtered ANN.

Implementation plan

Stage 0: Freeze contracts and record baselines

  • Use the Linux kernel, kubernetes/kubernetes, and Chromium source trees as the permanent large-repository corpus.

  • Add deterministic queries for exact search, semantic ordering, callgraph navigation, and edit churn.

  • Record time to first result, time to each coverage milestone, peak process memory, index bytes, refresh time, fallback work, and query latency with profiling disabled.

  • Capture CPU profiles in separate diagnostic runs against the same revisions, workloads, and build artifacts.

  • Record the exact revisions, build, operating system, storage type, resource-policy configuration, and profiler state.

  • Add the opt-in pprof endpoint and benchmark profile capture before evaluating index mechanisms.

  • Add a Lance-direct prototype as the competing substrate. Use Lance Rust crates without DuckDB on the serving path.

  • Prove coverage/path rows, exact vector rows, partial-index fallback, and worktree shallow-clone behavior.

  • Determine whether exact trigram and callgraph CSR index families can integrate through stable upstream extension points without a long-lived fork.

  • Compare Lance-direct, the purpose-built substrate, and the current implementation under the same workload. Include binary and dependency cost.

  • Select Lance only if it satisfies exact-search, semantic, callgraph, partial-coverage, worktree, crash-recovery, resource, and platform contracts. Otherwise record the failing gates and continue with the purpose-built substrate.

Stage 1: Add scheduler and load-control foundations

  • Run fixed-arrival interactive traffic while background convergence proceeds. Correct latency histograms for coordinated omission using the configured arrival interval.
  • Separate warmup from measured intervals. Mark a load point invalid when sustained dispatch lag shows that the driver cannot maintain the configured arrival rate.
  • Persist each completed load point immediately. A stopped sweep retains completed points and marks the result partial.
  • Write the benchmark manifest atomically after redacting secrets. Record toolchain, AFT build, harness version, host, kernel, filesystem, mount options, storage device class, correction interval, saturation policy, and profile artifacts.
  • Capture off-CPU evidence in addition to CPU profiles where the platform supports it.
  • Add bounded, class-aware discovery and index-work queues with deficit round-robin service across roots.
  • Define directory-entry, metadata-operation, content-byte, publication-byte, and time quanta from benchmark evidence rather than fixed unmeasured constants.
  • Add bounded memory ownership with RAII leases for known inputs and retained capacities.
  • Bound unchargeable transient work by file size, batch size, embedding count, and concurrency. Add process RSS, native-library, mapped-byte, and logical-charge telemetry as independent safety checks.
  • Add measured CPU, memory, and I/O pressure sampling with hysteresis. Treat unavailable platform signals as non-blocking and visible.
  • Demote only maintenance jobs to background OS priority and restore the exact previous thread policy.
  • Keep pressure sampling, allocator measurement, corpus discovery, and index finalization off the SubC transport thread.
  • Add bounded channel-0 priority so health and heartbeat traffic remains responsive without starving tool data.
  • Verify interactive latency, peak RAM, discovery CPU time, directory and metadata operations, content bytes, publication bytes, and I/O throughput under concurrent convergence on all three target repositories.

Stage 2: Add the segment substrate and incremental publication

  • Define the versioned outer envelope, typed-region registry, content identity, source and configuration fingerprints, validation rules, and reader dispatch.
  • Implement Arrow IPC regions for naturally columnar data and native typed regions for exact postings and graph topology.
  • Add the durable directory frontier, immutable discovery and coverage segments, manifest compare-and-swap, reader pins, orphan recovery, and retired-object collection.
  • Admit discovery only through bounded scheduler slices. Do not enumerate or stat the whole root during startup or bind.
  • Publish one valid batch before corpus discovery finishes.
  • Make every index-backed response derive complete and agent-facing result semantics from the pinned coverage state.
  • Verify interrupted discovery and index slices resume from durable coverage without hiding earlier segments.
  • Verify old and new envelope or region versions coexist in one manifest during rolling migration.
  • Expose discovery, indexing, stale, failed, and published aggregates in health telemetry.
  • Add worktree-scoped manifests that pin a shared base generation and publish private discovery, coverage, exact-search, semantic, and callgraph segments.
  • Add path supersession, deletion tombstones, restart recovery, bounded watcher-gap reconciliation, explicit base advancement, and reader-safe worktree collection.

Stage 3: Publish incremental exact-search segments

  • Discover and partition files in bounded batches.
  • Publish each exact trigram segment immediately after validation.
  • Query every segment referenced by the pinned project or worktree manifest plus the bounded fallback as one deterministic result.
  • Verify exact set equality whenever the fallback reports complete: true.
  • Verify bounded requests return useful matches with the correct guarantee, limitation, and next action before convergence.

Stage 4: Publish incremental exact semantic segments

  • Store the existing f32 representation in Arrow IPC regions within one immutable segment per completed embedding batch.
  • Merge exact scores across all published vector segments.
  • Use lexical and on-demand embedding work as bounded fallbacks for missing vectors.
  • Report complete: false while an applicable file can still change semantic top-K.

Stage 5: Publish incremental callgraph projections

  • Build specialized forward and reverse CSR regions plus Arrow IPC symbol and edge-attribute regions from validated SQLite construction batches.
  • Resolve cross-segment edges through stable symbol keys.
  • Use the local per-file graph for an uncovered requested frontier when possible.
  • Return the covered graph with an absence warning when fallback cannot finish.
  • Profile decode cost before adding compression or a neighborhood cache.

Stage 6: Benchmark optional mechanisms independently

  • Compare Lance built-in n-grams, greplm-core, qndx sparse n-grams, tgrep-style byte trigrams, and the current exact index under one exact differential workload. Keep only engines that preserve the AFT completeness contract.
  • Compare plain CSR offsets with sux Elias-Fano and sbits Partitioned Elias-Fano on the measured AFT callgraph distribution.
  • Compare ANN and vector compression against the exact oracle.
  • Compare cache policies only when profiles identify repeated decode work.
  • Use culpert only in a separate diagnostic arm after proving that its allocator wrapper composes with mimalloc. Keep RSS, native, SQLite, and mapped-byte telemetry authoritative.
  • Record negative results. Do not combine independent mechanisms in one benchmark arm.

Stage 7: Complete each cutover

  • Remove an old serving path only after incremental publication, fallback, completeness reporting, crash recovery, format coexistence, and rollback pass on all three target repositories.
  • Keep SQLite builder data only while it remains useful for construction. Prove that deleting its checkpoint causes bounded reconstruction rather than loss of published progress.
  • Update storage migration, doctor output, status, architecture documents, and release notes.
  • Run the repository validation lane and focused cross-platform envelope, publication, recovery, and corruption tests.

Each stage is independently reviewable. Stages 3, 4, and 5 can proceed separately after Stage 2.

Target corpus

The release gate uses three large source trees:

Source tree Primary stress
Linux kernel Large C tree, generated configuration boundaries, and dense include relationships.
kubernetes/kubernetes Large Go modules, generated clients, interfaces, and broad test trees.
Chromium Very large mixed C++, Java, JavaScript, and build-metadata tree. Chromium is the upstream source tree used by Google Chrome.

The benchmark records immutable revisions for all three trees. It runs the same revision before and after a change. Smaller fixtures remain useful for deterministic failure injection. They do not replace the large-repository release corpus.

Benchmark profiling endpoint

AFT adds an opt-in, loopback CPU profiling endpoint for benchmark and diagnostic runs. The endpoint is disabled unless the process starts with --pprof <loopback-address>. AFT rejects non-loopback addresses. An explicit bind failure fails process startup so a benchmark cannot run without the requested evidence.

The initial surface exposes one Go-compatible route:

GET /debug/pprof/profile?seconds=N

The route returns the standard pprof profile protobuf. The benchmark harness consumes it with go tool pprof. AFT does not render flamegraphs or reports in the daemon. The client can produce top tables, call graphs, or interactive views from the same captured profile.

The first implementation evaluates pprof-rs with protobuf output on supported POSIX targets. Its current profiler uses SIGPROF and restores the previous signal handler when capture stops. The prototype must verify unwind safety, handler coexistence, and measured overhead in the AFT process before adoption. AFT does not include server-side flamegraph generation. The build remains valid on Windows. A Windows benchmark reports CPU profiling as unsupported until AFT has a verified Windows sampling backend that emits the same protobuf contract. The benchmark result never silently omits a requested profile.

The endpoint permits one active capture. A concurrent capture receives 409 Conflict. The seconds argument is required, positive, and bounded by a server safety ceiling. Daemon shutdown cancels an active capture. The response includes the protobuf content type and does not buffer more than one completed profile in process memory.

The benchmark protocol separates measurement from diagnosis:

  1. Run latency, throughput, convergence, and memory measurements with profiling disabled.
  2. Repeat the same workload with profiling enabled and capture the relevant interval.
  3. Use the profile to explain measured cost. Do not compare profiled timings with unprofiled timings as equivalent samples.
  4. Retain the raw protobuf profile with the benchmark metadata and result bundle.

The first implementation uses pprof-rs with protobuf output on supported POSIX targets. The dependency enables protobuf output only. AFT does not include server-side flamegraph generation. The build remains valid on Windows. A Windows benchmark reports CPU profiling as unsupported until AFT has a verified Windows sampling backend that emits the same protobuf contract. The benchmark result never silently omits a requested profile.

Acceptance criteria

  • The first valid batch becomes queryable before corpus discovery or indexing completes.
  • Corpus discovery runs only in bounded scheduler slices. Startup, bind, and generation activation perform no whole-root enumeration or metadata scan.
  • Exact search returns set-equal results whenever it reports complete: true.
  • An exact-search deadline returns useful indexed and fallback matches with complete: false, a validity guarantee, an absence limitation, and a useful next action.
  • A partial semantic response identifies its results as candidates rather than a global top-K.
  • A partial callgraph response states that returned edges are valid and forbids a negative inference beyond the unindexed frontier.
  • A killed worker can lose only its in-flight batch. Every published discovery, coverage, exact-search, semantic, callgraph, and worktree segment remains readable after restart.
  • Recovery reconstructs pending work from the durable frontier and coverage state. It does not depend on persisted scheduler queue order.
  • A malformed envelope, invalid typed-region bound or version, checksum failure, fingerprint mismatch, or source-coverage failure cannot enter the active manifest.
  • Reader markers prevent collection of segments pinned by an active manifest version.
  • Readers can open a manifest that references supported old and new envelope or region versions during migration.
  • An unsupported or corrupt required region isolates its segment and produces explicit partial-coverage semantics. It cannot make a generation silently partly readable.
  • Health and interactive reader requests remain responsive while all three target repositories converge.
  • A long-lived worktree reopens its last valid private generation after process restart or root eviction without a whole-root rescan.
  • Worktree-private path records and tombstones supersede base records consistently across exact search, semantic search, and callgraph queries.
  • Base advancement reuses only shared segments whose covered paths the worktree has not superseded, and a failed advancement leaves the prior worktree generation active.
  • Removing a worktree cannot collect a private or shared segment while a reader or manifest still pins it.
  • Background convergence keeps owned retained capacity within the configured logical memory budget. A batch that cannot acquire its initial bounded lease does not start.
  • Each owned capacity increase is charged before reserve or retention. Unchargeable transient work remains within measured file-size, batch-size, and concurrency bounds.
  • Peak process RSS, native allocation telemetry, mapped bytes, and logical charges are recorded independently for each target repository.
  • The Stage 0 report compares the current implementation, Lance-direct, and the purpose-built substrate under identical corpus revisions and workloads.
  • The Lance prototype cannot pass by storing opaque AFT indexes as unmanaged sidecars. It must prove transactional index coverage, partial fallback, compaction, recovery, and worktree lifecycle through the selected Lance extension points.
  • Lance adoption requires exact trigram and callgraph CSR index families without a long-lived downstream fork.
  • The report records binary size, clean build time, dependency footprint, and Linux, macOS, and Windows target support for each alternative.
  • Removing the SQLite construction checkpoint does not remove serving data or published progress. AFT reconstructs the checkpoint in bounded work from source plus durable coverage and segments.
  • Interactive reads and channel-0 health checks meet their latency gate while background discovery and indexing use all admitted permits.
  • No corpus walk, pressure sample, allocator scan, merge, or publication finalization executes on the SubC transport thread.
  • Sustained control traffic cannot starve tool data, and sustained tool traffic cannot delay a ready health response beyond its gate.
  • balanced pauses only after measured pressure crosses its configured threshold and resumes only after the hysteresis recovery threshold.
  • An unavailable host-pressure signal is visible in health telemetry and does not permanently pause indexing.
  • Only maintenance jobs run at background OS priority. Interactive heavy initialization retains normal priority.
  • The benchmark records background CPU share, directory entries, metadata operations, bytes read and written, discovery and index service time, pause time by reason, and interactive latency under load.
  • Linux, macOS, and Windows publication helpers use only unprivileged APIs.
  • Health output reports inventory state, coverage aggregates, active segments, bounded failure aggregates, mapped bytes, resident estimates, and scheduler pause reason.
  • Each cutover records time to first result, discovery progress, index progress, fallback work, peak memory, CPU time, I/O work, and query latency on the Linux kernel, Kubernetes, and Chromium revisions.
  • An explicit profiling bind exposes /debug/pprof/profile, rejects non-loopback addresses, and fails startup when the listener cannot bind.
  • Concurrent profile requests receive 409 Conflict, and a successful response loads in go tool pprof.
  • Benchmark bundles retain raw profiles and distinguish profiled diagnostic runs from unprofiled measurements.
  • No approximate or quantized semantic path becomes the default without a maintainer-approved recall threshold recorded before measurement.

Risks

  • Risk: Segment fan-out increases query overhead. Compact published segments without blocking new batch publication.
  • Risk: Corpus discovery competes with interactive work for CPU and I/O. Admit directory enumeration and metadata operations through the same bounded background scheduler as index construction. Pause both under foreground latency or resource pressure.
  • Risk: Incomplete discovery creates unknown coverage. Report inventory_complete: false and admit only bounded query-scoped traversal as fallback work.
  • Risk: Discovery and coverage segments cause write amplification. Publish immutable segments only after a measured amount of useful work or at an interruption boundary. Compact them through the generation protocol.
  • Risk: A shared envelope couples unrelated index formats. Version each typed region independently and retain separate segments and corruption domains for each index kind.
  • Risk: Arrow adds metadata or decode overhead where native encodings are better. Use Arrow only for naturally columnar rows. Benchmark payload and decode cost before expanding its use.
  • Risk: Projection schemas omit a callgraph attribute. Compare focused query behavior against SQLite before routing that query to projections.
  • Risk: SQLite construction state diverges from published generations. Bind each checkpoint to source and configuration fingerprints. Discard and reconstruct a mismatched checkpoint.
  • Risk: Mapping behavior differs by operating system. Centralize mapping and file replacement behind focused platform tests.
  • Risk: Compression reduces bytes but increases decode cost. Add one codec at a time and retain the plain format until the measured trade-off is accepted.
  • Risk: Worktree generations retain old shared segments. Reconcile base advancement in bounded slices and collect old generations only after path supersession and reader-pin checks.
  • Risk: A watcher gap makes durable worktree coverage stale. Keep valid private segments available, mark absence unreliable, and run bounded reconciliation through the scheduler.
  • Risk: Logical memory accounting misses native or mapped memory. Gate admission on logical ownership and verify the result against process RSS and mapped-byte telemetry.
  • Risk: Load thresholds encode an unmeasured policy. Derive defaults from the target-corpus benchmark and publish the measured basis.
  • Risk: A pressure controller oscillates. Use separate pause and resume thresholds and record time spent in each state.
  • Risk: Background priority delays query-triggered repair. Classify query-triggered fallback as interactive work and demote only maintenance jobs.
  • Risk: Control prioritization starves data traffic. Bound consecutive control dispatches and test both traffic directions under saturation.
  • Risk: CPU sampling changes the measured workload. Keep profiling disabled for primary measurements and use a repeated diagnostic run for attribution.

Open questions

  1. Should the first ranked lexical experiment use Tantivy, or should it wait until exact incremental publication is complete?
  2. Which internal switch format should govern one-release rollback for each index?
  3. Should callgraph projection segments contain both edge directions from the first batch?
  4. Should approximate semantic search remain opt-in for one release after it passes the recall gate?
  5. How long should AFT retain an absent worktree generation before collection?

References

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions