BUG

[linchen-LC]

Title: extract --code-only hangs forever on large monorepo (no graph.json written); worker spins 100% in re.sub — reproduced on 0.9.53 and 0.9.54

Environment

• macOS 15.7.7 (Apple Silicon), Python 3.11.15 (uv-managed), graphifyy 0.9.53 and 0.9.54 (both reproduced)
• Corpus: DeepSeek Harness monorepo (dsh) v0.1.3-alpha.1, ~2,470 code files after excluding tests/snapshots via .graphifyignore (**/tests/, **/*.spec.ts, **/*.test.ts, **/*.e2e.ts, snapshots/). Total repo incl. tests ≈ 4,037 files.

Steps to reproduce

1. Add the .graphifyignore exclusions above in repo root.
2. graphify extract . --code-only (default workers)
   → AST extraction progress freezes: parallel runs stall at ~97–99% (2400/2470; inside packages/ alone 3300/3362; also 3400/3415) and graph.json is never written.
3. Variants all reproduce: --max-workers 1/2/4/10, --no-dedup, 0.9.53 and 0.9.54.
4. Serial run (--max-workers 1) freezes earlier at 500/2469 (20%), also indefinitely.
5. Repo subsets that are small (each top-level dir except packages/) complete fine; a ~60-file repo (xiaohongshu-mcp) completes fine — so not a universal scale issue.

Diagnostics

• While frozen: the active worker shows 100% CPU and opens NO files (lsof lists only the python binary).
• sample (macOS) of the spinning worker shows 2485/2485 samples inside _sre_SRE_Pattern_sub → sre_search → sre_ucs1_match — i.e., a Python re.sub operating on a large UCS-1 (ASCII) string is the hot path. No traceback; process exits only when killed, with resource_tracker semaphore-leak warnings at teardown.
• Freeze point shifts with file set → looks content-triggered: likely a regex normalization/sanitization pass over large serialized per-chunk/per-file payloads that is pathologically slow (or catastrophic backtracking) for some content in this corpus.

Request
Could you check the post-parse regex normalization paths (e.g., anything running re.sub over chunk/file payloads) for patterns that are linearity-unsafe on large ASCII inputs? Happy to provide more logs/cache entries on request. This blocks graphify extract on moderately large TypeScript monorepos.

Notes

• I did not retain graphify-out/cache (deleted between retries); can keep it on a repro run if useful.
• Corpus is a public GitHub repo: deepseek-ai/deepseek-harness (tag v0.1.3-alpha.1).

[Sagexd08]

Reproduced, and a fix has been put together. The diagnosis in the report is close, but the hot path isn't a linearity-unsafe regex. Every re.sub and every pattern in the extractor is linear. The re frames showing up in your sample output are the innermost frames of a quadratic caller, not the cause of it.

Root cause

_normalize_ts_import_types in graphify/extract.py (the #3154 TypeScript import(...) type normalizer) filters regex matches against tree-sitter type_arguments byte ranges using a nested linear scan:

matches = [m for m in matches
           if not any(start <= m.start() < end for start, end in original_type_ranges)]

The same shape is repeated in the second pass. The cost of that is O(matches x ranges). On top of it, m.start() gets re-evaluated inside any() once per range, which is why the profile ends up dominated by re.Match.start and the genexpr, and why sample attributes the time to _sre.

Both of those counts scale with file size in TS code that mixes generic calls (fn<A, B>(x), which contributes one type_arguments range each) with import(...) types. The filter was measured in isolation, with everything else held constant:

file	import() sites	type_arguments	before	after
0.8 MB	10,000	10,000	~21 s	0.003 s
2.1 MB	25,000	25,000	~2.6 min	0.008 s
4.2 MB	50,000	50,000	~11.8 min	0.019 s

Every symptom in the report is accounted for by this:

• 100% CPU with zero open files. All of the work happens in memory once the file has been read, so nothing is held open while it spins.
• The freeze point shifting with the file set. It gets triggered by content, meaning a handful of large generated or barrel .ts files, rather than by file count. Those live under packages/; your other top-level dirs and the 60-file repo don't contain any.
• --max-workers 1 stalling earlier (500/2469). A single worker reaches the pathological file sooner, instead of ten workers racing past ~2400 first. The stall point isn't meaningful on its own.
• Never completing. With several such files in one tree the total runs into hours, so graph.json never gets written at all.
• --no-dedup, 0.9.53 and 0.9.54 all reproducing. The code sits in the extract path and is unchanged across those three releases, so none of those knobs can route around it.

Fix

Both scans are replaced by a binary search over start-sorted ranges, combined with a prefix maximum of the end values, so a lookup only has to touch nesting depth rather than every range in the file. Type-argument ranges nest but never partially overlap, which is what makes the prefix max a correct stop condition. The subtle part is that an inner closed sibling must not be allowed to terminate the walk early.

Normalization behaviour itself is left untouched. Byte length and line count are still preserved, the same masking decisions are made, and the runtime-vs-type disambiguation from 6baa5e0 is kept exactly as is. The new lookup was checked against the old predicate with a 500,000-case differential test, including randomly generated non-nested ranges so the nesting assumption is exercised from both sides. No mismatches were found.

A regression test has been added as test_ts_normalizer_scales_linearly_on_large_files. It asserts on scaling, meaning that doubling the input must not roughly quadruple the time, rather than on wall clock, so it doesn't become machine dependent or flaky on slower CI boxes. On the current code it fails at 3.4x; after the change it passes.

One thing worth mentioning: graphify-out/cache shouldn't need to be kept around for this. The failure reproduces from the shape of the generated source alone, so a synthetic file is enough to see it.

[Sagexd08]

The fix has been opened as #3402. It comes from a fork, since push access here isn't something I have.

On the CI delta, that looks like Windows environment flakiness rather than anything the change introduced. The failure diff lists 13 new failures and also 5 that disappeared, and a change confined to the TS normalizer can't produce movement in that second direction at all. All of the failures sit in serve_http, install*, hooks and watch. None land in extract, resolution, or any TypeScript path, and none of the failing tests reference the functions that were touched.

The install tests are also state dependent, which was easy enough to confirm by running them against unmodified c9f9901 with no edits applied: 8 failures on the first run, then 3, 3 and 3 on the next three. Run head to head three times each, the baseline and the fix branch come out identical at 3 failed / 155 passed, so the branch isn't contributing anything of its own.

Happy to re-run any specific test in a loop if one of those failures still looks suspicious from your side, or to rebase if the branch has drifted by the time this gets looked at.

[safishamsi]

Fixed in v0.9.56 by #3402 (thanks @Sagexd08) — TS import-type normalization no longer does an O(matches×ranges) scan. Release: https://github.com/Graphify-Labs/graphify/releases/tag/v0.9.56