perf: release JS parse trees between symbol-fact collection files

[VasuBansal7576]

What changes

JS/TS symbol-fact collection keeps every parsed tree alive and repeatedly walks whole trees.
Process each unique resolved file in its own scope and index relevant syntax nodes in one preorder walk, preserving ordered facts and resolution behavior.
Duplicate resolved paths retain the original batch behavior, including mixed-grammar symlinks; those inputs still retain the batch's trees.

Independent regression coverage

Nine focused tests now cover ordered facts, aliases, exports, calls, class references, pre-existing fact prefixes, duplicate paths, and both mixed-grammar symlink orders.
Explicit expected facts were checked against the original 937e59a collector and do not use the modified batch helper as their oracle.
A separate ordering case preserves a first class's references before a later class's inheritance facts.

The lifetime test observes the actual native tree-sitter Tree reference count on CPython.
It checks that no root or descendant still owns the previous tree before the next parse, then releases the test's final reference.
Negative controls that drop aliases, bypass the duplicate fallback, or retain a native descendant make the appropriate tests fail.

Upstream CI

CI run 34088979232 passed on head c2a8667, tested as merge 5138d13 into v8 at c9f9901:

• Linux Python 3.10: 5,443 passed, 11 skipped.
• Linux Python 3.12: 5,442 passed, 12 skipped.
• All five generated-artifact checks and the CLI installation steps passed.

Ruff and the required graphify update . also passed locally.
The security job is green under the repository's existing non-blocking policy; its Bandit and pip-audit steps report findings in unchanged code and build dependencies.
This is not a clean security-scan claim, and this PR does not change those scan policies or dependency specifications.

Full-repository JS/TS measurements

Three additional pinned repositories were tested on all tracked eligible JS/TS-family files, including fixtures, generated bundles, and symlinks.
Each row is the median of three alternating before/after pairs in fresh processes, comparing upstream c9f9901 with CI-tested PR merge 5138d13 in the same Python environment.

Repository	Files	Workload	Time, before → after	Peak RSS, before → after
webpack	15,273	Fact collection	7.72s → 5.30s	1,097 → 151 MiB
webpack	15,273	Graph extraction	29.99s → 26.88s	1,167 → 279 MiB
Next.js	23,909	Fact collection	48.33s → 31.54s	3,031 → 3,219 MiB
Next.js	23,909	Graph extraction	137.57s → 114.11s	3,515 → 3,605 MiB
VS Code	13,458	Fact collection	92.09s → 47.47s	4,030 → 489 MiB
VS Code	13,458	Graph extraction	304.38s → 250.82s	4,326 → 1,812 MiB

All 18 full-corpus before/after pairs preserve exact ordered output; every graph run reports zero failed sources.
The VS Code graph contains 223,509 nodes and 865,391 edges; Next.js has 120,345 nodes and 270,285 edges; webpack has 42,655 nodes and 56,495 edges.

Limitation exposed by Next.js: five groups of duplicate resolved paths, containing six additional aliases, trigger the whole-batch compatibility fallback.
Next.js median collector RSS rises 6.2% and graph parent RSS rises 2.5%, so this PR does not deliver a memory reduction on that full corpus.
Collector RSS ranges overlap substantially; these are descriptive observed medians, not a statistically established regression magnitude.
webpack graph timing ranges also overlap, so its 10.4% median time reduction should not be treated as a guaranteed speedup.
Scoping tree retention to colliding path groups while preserving ordering and mixed-grammar behavior is a follow-up opportunity; it is not implemented in this PR.

Immutable full-corpus receipts include pinned source commits, full file manifests, all 36 measurements, ranges, counts, hashes, commands, and validation metadata.

Concrete repository examples

The before/after results above use these exact source revisions.
Check them out in sibling directories named webpack, next.js, and vscode:

Example	Pinned source	What it exercises
webpack	92c4650	15,273 eligible files, including many small bundler fixtures; graph peak RSS falls 76.1%.
Next.js	4fed8ea	23,909 eligible files, including compiled bundles and symlinks; the duplicate-path fallback prevents a memory reduction.
VS Code	58ae409	13,458 eligible files totaling 158.9 MiB; graph time falls 17.6% and parent peak RSS falls 58.1%.

For a concrete Next.js fallback example, these tracked paths under test/development/app-dir/hmr-symlink/app/ resolve to the same file:

symlink-chain/page.tsx -> ../symlink-link/page.tsx
symlink-link/page.tsx  -> ../symlink-target/page.tsx
symlink-target/page.tsx

This one group is enough to select whole-batch collection; the complete corpus contains five such groups.
The benchmark includes these aliases as tracked, preserving the original collector's compatibility behavior.

To repeat both workloads for all three examples, use graphify-before at c9f99018774e2e0380e9f65b3959944559a0d5f6 and graphify-after at 5138d13dad082da1a84f005917cf925dbe44478c.
Run from the PR checkout with the same Python environment for both arms; the recorded runs used Python 3.13.11, tree-sitter 0.25.2, tree-sitter-javascript 0.25.0, and tree-sitter-typescript 0.23.2.

for repo in webpack next.js vscode; do
  for mode in facts graph; do
    python scripts/benchmark_js_fact_collection.py \
      --baseline ../graphify-before --candidate ../graphify-after \
      --source "../$repo" --files 0 --runs 3 --mode "$mode" \
      --output "/tmp/${repo}-${mode}.json"
  done
done

Each command validates the loaded collector, uses fresh processes, and compares complete ordered output hashes.
The immutable receipts above contain the exact commands used for the published measurements.

Earlier repeated OpenClaw measurements

The table compares the exact CI-tested merge 5138d13 with its upstream base c9f9901.
Both run on pinned OpenClaw 4db554f4cb3aaab69c1be4931244ce36376e4906, with the same Python 3.13.11 environment and locked parser versions.
Values are medians of alternating before/after pairs in fresh processes.

Workload	Pairs	Elapsed, before → after	Peak RSS, before → after
Fact collection, 2,000 files	5	6.94s → 4.15s	1,325.0 → 71.8 MiB
Fact collection, 8,000 files	3	35.47s → 20.40s	4,191.4 → 135.1 MiB
Graph extraction, 2,000 files	3	20.74s → 18.09s	1,381.0 → 185.7 MiB

All pairs preserve the exact ordered output within their comparison set.
The current-base graph sample produces 15,903 nodes and 51,217 edges, with no failed sources and identical ordered serialization hashes.
A second complete measurement set compares original PR base 937e59a with branch head c2a8667; all 22 before/after pairs across the two sets match.

Immutable raw receipts and validation metadata include the selected file manifests, source and collector hashes, every individual timing/RSS measurement, ranges, counts, and output hashes.
The reproducible benchmark script verifies which collector was loaded and rejects incomplete graph extraction.

Example with separate checkouts of the before and after revisions:

python scripts/benchmark_js_fact_collection.py \
  --baseline ../graphify-before --candidate ../graphify-after \
  --source ../openclaw --files 2000 --runs 5 --output /tmp/facts.json

Use --files 8000 --runs 3 for the larger collector sample, or --mode graph --files 2000 --runs 3 for graph extraction.

Measurement limits

These measure fact collection and library graph extraction, not full CLI execution or extraction of every language in each repository.
The OpenClaw workloads are sampled; the three additional corpora use all tracked eligible JS/TS-family files.
Filesystem caches are not flushed and desktop load is not controlled, so the observed speedups are not statistical guarantees.
RSS is the process high-water mark before result serialization; graph RSS excludes extraction worker processes.
The OpenClaw samples resolve uniquely; the full Next.js corpus exercises the duplicate-path memory fallback described above.

Earlier full CLI measurement on a different base

The earlier single before/after run extracted 34,432 OpenClaw files in 18m 57.58s versus 5m 11.154s, with byte-identical 527,865,058-byte graph output.
Both arms used Graphify 02b7c220 with identical pre-existing local patches, with this collector as the changed component.
That full CLI run has not been repeated on the newer bases and should not be conflated with the repeated measurements above.
Its graph SHA-256 was 6d48b60dc5c0c8206d37ba998aab3532ddd0d99d1834ca8f7e093de9d79c8dd8.
Archived measurement image.

[graphify-labs]

Graphify reviewed this change.

Worth a look — the grounded gate found no coupling regressions or blocking issues, but 1 advisory finding(s) below merit a look before merge.

Formal verification. No changes could be formally verified in this run.

---

Graphify review — findings

Refactors JS/TS symbol-resolution collection to process files one at a time via a new _collect_js_symbol_resolution_facts_batch, so each source's parse tree is released before the next is parsed instead of holding every tree in memory at once. Parsing now indexes only the relevant node kinds (imports/exports, lexical declarations, classes) per file rather than re-walking the whole tree for each pass, and the streaming path preserves the original fact ordering (all calls uses before class type uses). Falls back to the batch collector when the input contains duplicate resolved paths (e.g. symlinks), keeping the "last parsed tree wins" behavior for mixed grammars.

Worth a look

• Only 'calls' vs non-'calls' partition assumed, but 'references'/'inherits' ordering not guaranteed to match batch — graphify/extractors/resolution.py:1511 · Escalate · medium
  • agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review

Analysis details — impact, health, verification

Impact & health

Graphify review

Impact — 1731 functions depend on the 132 functions this change touches.

Health — this change adds coupling hotspots:

• new: extract() — 537 callers, 43 callees
• new: _rebuild_code() — 113 callers, 50 callees
• new: _extract_generic() — 18 callers, 25 callees
• new: extract_js() — 85 callers, 4 callees
• new: extract_xaml() — 19 callers, 17 callees
• new: dispatch_command() — 2 callers, 124 callees
• new: extract_objc() — 27 callers, 9 callees
• new: _resolve_js_module_path() — 27 callers, 6 callees
• …and 34 more — each is listed as a finding

Verification — 1731 functions in the blast radius were not formally verified this run (proofs are advisory here).

Gate & verification

graphify gate

PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.

Advisory (not blocking):

• verification_scope: 738 function(s) in the blast radius were not formally verified this run

Formal verification

Could not verify: Could not verify \_collect\_js\_symbol\_resolution\_facts.

The verifier did not have enough to check \_collect\_js\_symbol\_resolution\_facts, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: parameter `facts` is annotated `_SymbolResolutionFacts` — outside the synthesizable primitive/collection set

· 1 grounded finding(s) anchored inline below; 41 more finding(s) on lines outside this diff (see the check run).

[graphify-labs]

Graphify reviewed this change.

Looks safe to merge — no coupling regressions and no blocking issues, checked against the code graph (not a self-assessment).

Formal verification. No changes could be formally verified in this run.

---

Graphify review — findings

Splits JS symbol-resolution collection so that when a path batch has no duplicate resolved paths, each file is parsed in its own _SymbolResolutionFacts scope via _collect_js_symbol_resolution_facts_batch and its native tree is released before the next parse; batches containing duplicate/symlinked paths fall back to the original multi-pass batch collector to keep the last-tree-wins and mixed-grammar behavior. Preserves the existing fact ordering by emitting all calls uses before class-type uses across files, and speeds the batch pass by pre-indexing only import/export, lexical-declaration, and class nodes per tree instead of re-walking the whole tree for each phase. Adds scripts/benchmark_js_fact_collection.py, which samples tracked JS/TS files from a source checkout and compares two git checkouts for elapsed time, peak RSS, and byte-identical ordered output across fresh alternating worker processes.

No blocking issues surfaced. 9 lower-confidence candidates did not survive cross-model review.

Analysis details — impact, health, verification

Impact & health

Graphify review

Impact — 1739 functions depend on the 140 functions this change touches.

Health — this change adds coupling hotspots:

• new: extract() — 539 callers, 43 callees
• new: _rebuild_code() — 113 callers, 50 callees
• new: _extract_generic() — 18 callers, 25 callees
• new: extract_js() — 85 callers, 4 callees
• new: extract_xaml() — 19 callers, 17 callees
• new: dispatch_command() — 2 callers, 124 callees
• new: extract_objc() — 27 callers, 9 callees
• new: _resolve_js_module_path() — 27 callers, 6 callees
• …and 35 more — each is listed as a finding

Verification — 1739 functions in the blast radius were not formally verified this run (proofs are advisory here).

Health delta baseline: last indexed commit c9f9901 (diverged from this PR's base — delta is approximate).

Gate & verification

graphify gate

PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.

Advisory (not blocking):

• verification_scope: 746 function(s) in the blast radius were not formally verified this run

Formal verification

Could not verify: Could not verify \_collect\_js\_symbol\_resolution\_facts.

The verifier did not have enough to check \_collect\_js\_symbol\_resolution\_facts, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: parameter `facts` is annotated `_SymbolResolutionFacts` — outside the synthesizable primitive/collection set

· 2 grounded finding(s) anchored inline below; 41 more finding(s) on lines outside this diff (see the check run).