Skip to content

Feature/Fix: Profile and optimize shadow lineage accounting#44

Open
iam-brain wants to merge 2 commits into
iam-brain/lineage-branch-frontier-diagnosticsfrom
iam-brain/lineage-performance-profiling
Open

Feature/Fix: Profile and optimize shadow lineage accounting#44
iam-brain wants to merge 2 commits into
iam-brain/lineage-branch-frontier-diagnosticsfrom
iam-brain/lineage-performance-profiling

Conversation

@iam-brain

Copy link
Copy Markdown
Owner

Summary

  • add stage-level timing for lineage discovery, loading, reconciliation, and composition
  • eliminate repeated copy-on-write growth and timestamp parsing in family reconciliation
  • hash JSONL bytes during parsing so two-pass mutation checks need fewer full-file reads
  • add a bounded two-million-observation regression test

This is another isolated component of the planned scanner rearchitecture tracked in #10. It keeps lineage accounting in shadow mode and does not remove the legacy fallback.

Real-data validation

On the retained 570-document / 819,779-observation corpus, daily UTC totals stayed identical while lineage runtime fell from about 225s in the instrumented baseline to about 164s. Compared with the earlier roughly 360s replay, the current path is about 54% faster. Family reconciliation dropped from 72.4s to 15.2s; discovery and document loading remain the next performance constraints.

Tests

  • swift test --filter CodexLineageEngineTests --filter CodexLineageTwoPassDiscoveryTests --filter CodexLineageLedgerTests
  • two-million-observation regression test (29.5s in the full suite)
  • retained local UTC replay (exact daily totals preserved)
  • make check
  • make test (649 selections, 55/55 groups, no failures, retries, or timeouts)

Closes #42.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant