Skip to content

Stream lineage families with two-pass discovery #33

Description

@iam-brain

Question

Complete the memory-bound scanner architecture after the prepared-family cache seam. Index only lineage metadata and file signatures first, then parse and reconcile full observations one family at a time so total retained observation memory is bounded by the largest active family rather than the whole history.

Acceptance

Preserve normalized-home scope, bounded exceptional ancestor lookup, evidence quality, deterministic fingerprints, cancellation, atomic publication, and active/archive behavior. Add structural peak-memory diagnostics plus normal-suite and opt-in multi-million-observation coverage. Do not claim a fixed RSS ceiling across platform/runtime implementations.

Why this was added

Issue #17 can bound reconciliation scratch state but current discovery retains every parsed Document observation. This follow-up is required before claiming end-to-end memory-bounded scanning.

Parent map: #10

Metadata

Metadata

Assignees

No one assigned

    Labels

    wayfinder:taskPrerequisite task on a Wayfinder map

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions