Skip to content

feat(detect): enumerate large trees via git, repo, and submodules - #3287

Open
albertbu wants to merge 3 commits into
Graphify-Labs:v8from
albertbu:albertb/detect-git-enum
Open

feat(detect): enumerate large trees via git, repo, and submodules#3287
albertbu wants to merge 3 commits into
Graphify-Labs:v8from
albertbu:albertb/detect-git-enum

Conversation

@albertbu

@albertbu albertbu commented Sep 2, 2026

Copy link
Copy Markdown

Use git ls-files, Google repo manifests, and nested worktrees so ignored trees are pruned at directory granularity instead of a Python walk.
slice review: v8...albertbu:graphify:albertb/detect-git-enum

@graphify-labs graphify-labs Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Graphify reviewed this change.

Worth a look — the grounded gate found no coupling regressions or blocking issues, but 5 advisory finding(s) below merit a look before merge.

Formal verification. No changes could be formally verified in this run.


Graphify review — findings

Adds Git-aware enumeration to detect() so large trees are listed via git ls-files (--cached plus --others --exclude-standard), Google repo project.list worktrees, and nested git submodules/worktrees rather than walked, keeping gitignored and submodule-internal paths out of the scan. Prunes Buildroot output//output_<board> sysroots and .repo metadata via _is_buildroot_output and expanded _SKIP_DIRS, and drops non-classifiable names under --code-only through _maybe_code_filename. Falls back to the existing os.walk when git ls-files fails or times out (300s), parallelises word counting and regular-file checks via the thread pool, and emits rate-limited --verbose heartbeats during long walks.

Worth a look

  • Symlink outside root now scanned when not a symlink but resolves outside rootgraphify/detect.py · Escalate · high
    • agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
  • Files reached through an out-of-root symlinked directory are no longer rejectedgraphify/detect.py · Escalate · high
    • agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
  • Memory scan follows out-of-root symlink directoriesgraphify/detect.py:2391 · Escalate · high
    • agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
  • Files under out-of-root symlinked directories can be admittedgraphify/detect.py:2562 · Escalate · high
    • agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
  • Symlink files with targets outside root are no longer skipped for non-symlinks; but non-symlink out-of-root resolution check droppedgraphify/detect.py · Escalate · medium
    • agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
Analysis details — impact, health, verification

Impact & health

Graphify review

Impact — 2140 functions depend on the 807 functions this change touches.

Health — this change adds coupling hotspots:

  • new: extract() — 520 callers, 43 callees
  • new: _rebuild_code() — 113 callers, 50 callees
  • new: detect() — 116 callers, 17 callees
  • new: save_manifest() — 40 callers, 11 callees
  • new: extract_files_direct() — 17 callers, 20 callees
  • new: extract_xaml() — 19 callers, 17 callees
  • new: extract_corpus_parallel() — 26 callers, 11 callees
  • new: dispatch_command() — 2 callers, 123 callees
  • …and 33 more — each is listed as a finding

Verification — 2140 functions in the blast radius were not formally verified this run (proofs are advisory here).

Gate & verification

graphify gate

PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.

Advisory (not blocking):

  • verification_scope: 1165 function(s) in the blast radius were not formally verified this run

Formal verification

Could not verify: Could not verify detect.

The verifier did not have enough to check detect, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: parameter `root` is annotated `Path` — outside the synthesizable primitive/collection set

Could not verify: Could not verify detect\_incremental.

The verifier did not have enough to check detect\_incremental, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: parameter `root` is annotated `Path` — outside the synthesizable primitive/collection set

Could not verify: Could not verify \_git\_tracked\_path\_keys.

The verifier did not have enough to check \_git\_tracked\_path\_keys, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: parameter `root` is annotated `Path` — outside the synthesizable primitive/collection set

Could not verify: Could not verify \_is\_noise\_dir.

The verifier did not have enough to check \_is\_noise\_dir, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: parameter `parent` is annotated `'Path | None'` — outside the synthesizable primitive/collection set

· 6 grounded finding(s) anchored inline below; 35 more finding(s) on lines outside this diff (see the check run).

Comment thread graphify/detect.py
return name[dot:].lower() in _CORPUS_EXT_LOWER


def _is_noise_dir(part: str, parent: "Path | None" = None) -> bool:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Health regression_is_noise_dir()

10 callers depend on it (afferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Health flags are expected: git enumeration reuses _is_noise_dir so walk and ls-files prune the same trees. detect()’s 116 callers are pre-existing. Not splitting those helpers in this PR.

Comment thread graphify/detect.py
return _path_is_under(worktree, root) or _path_is_under(root, worktree)


def _git_enumerate_named_worktrees(

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Health regression_git_enumerate_named_worktrees()

fans out to 7 callees (efferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.

Comment thread graphify/detect.py
)


def _git_enumerate_files(

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Health regression_git_enumerate_files()

fans out to 8 callees (efferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.

Comment thread graphify/detect.py


def detect(root: Path, *, follow_symlinks: bool | None = None, google_workspace: bool | None = None, extra_excludes: list[str] | None = None, cache_root: Path | None = None, gitignore: bool = True) -> dict:
def detect(root: Path, *, follow_symlinks: bool | None = None, google_workspace: bool | None = None, extra_excludes: list[str] | None = None, cache_root: Path | None = None, gitignore: bool = True, code_only: bool = False) -> dict:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Health regressiondetect()

fans out to 17 callees (efferent coupling); 116 callers depend on it (afferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.

Comment thread graphify/detect.py
vprint(f"scanning {root} ...")
heartbeat = _Heartbeat("scanning")

def _walk_from(

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Health regression_walk_from()

fans out to 8 callees (efferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.

Comment thread graphify/detect.py

for p in all_files:
# For memory dir files, skip hidden/noise filtering
def _admit(p: Path) -> tuple:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Health regression_admit()

fans out to 8 callees (efferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.

@albertbu

albertbu commented Sep 2, 2026

Copy link
Copy Markdown
Author

Out-of-root resolve() guard is restored

@graphify-labs graphify-labs Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Graphify reviewed this change.

Worth a look — the grounded gate found no coupling regressions or blocking issues, but 5 advisory finding(s) below merit a look before merge.

Formal verification. No changes could be formally verified in this run.


Graphify review — findings

Adds a git-aware enumeration path to detect() that lists large trees via git ls-files (cached + untracked, excluding standard ignores), Google repo .repo/project.list project sets, and nested git worktrees/submodules, so gitignored and submodule directories are enumerated from their own indexes instead of being walked. Prunes Buildroot output//output_<board>/ sysroots (detected by host/target/images/staging layout) and .repo metadata, drops non-classifiable filenames under --code-only, and parallelizes word counting via a thread pool with a serial fallback. Emits rate-limited stderr heartbeats during long walks under --verbose, with a 300s git ls-files timeout that falls back to the Python os.walk scan when Git is unavailable or times out.

Worth a look

  • memory_dir files may be lost when git enumeration replaces all_filesgraphify/detect.py · Escalate · high
    • agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
  • code_only path skips Google Workspace/Office conversion but stores original binary as readable filegraphify/detect.py · Escalate · medium
    • agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
  • non-descend walk clears dirnames only when not in_memory, leaking subtree files into top-level bucketgraphify/detect.py · Escalate · medium
    • agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
  • Escaping .gitmodules path can make scanner run Git outside the scan rootgraphify/detect.py:1542 · Escalate · medium
    • agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
  • Concurrent subtree walkers mutate shared ignored_dirs without synchronizationgraphify/detect.py:2287 · Escalate · medium
    • agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
Analysis details — impact, health, verification

Impact & health

Graphify review

Impact — 2142 functions depend on the 809 functions this change touches.

Health — this change adds coupling hotspots:

  • new: extract() — 520 callers, 43 callees
  • new: _rebuild_code() — 113 callers, 50 callees
  • new: detect() — 117 callers, 17 callees
  • new: save_manifest() — 40 callers, 11 callees
  • new: extract_files_direct() — 17 callers, 20 callees
  • new: extract_xaml() — 19 callers, 17 callees
  • new: extract_corpus_parallel() — 26 callers, 11 callees
  • new: dispatch_command() — 2 callers, 123 callees
  • …and 33 more — each is listed as a finding

Verification — 2142 functions in the blast radius were not formally verified this run (proofs are advisory here).

Gate & verification

graphify gate

PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.

Advisory (not blocking):

  • verification_scope: 1167 function(s) in the blast radius were not formally verified this run

Formal verification

Could not verify: Could not verify detect.

The verifier did not have enough to check detect, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: parameter `root` is annotated `Path` — outside the synthesizable primitive/collection set

Could not verify: Could not verify detect\_incremental.

The verifier did not have enough to check detect\_incremental, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: parameter `root` is annotated `Path` — outside the synthesizable primitive/collection set

Could not verify: Could not verify \_git\_tracked\_path\_keys.

The verifier did not have enough to check \_git\_tracked\_path\_keys, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: parameter `root` is annotated `Path` — outside the synthesizable primitive/collection set

Could not verify: Could not verify \_is\_noise\_dir.

The verifier did not have enough to check \_is\_noise\_dir, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: parameter `parent` is annotated `'Path | None'` — outside the synthesizable primitive/collection set

· 6 grounded finding(s) anchored inline below; 35 more finding(s) on lines outside this diff (see the check run).

Comment thread graphify/detect.py
return name[dot:].lower() in _CORPUS_EXT_LOWER


def _is_noise_dir(part: str, parent: "Path | None" = None) -> bool:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Health regression_is_noise_dir()

10 callers depend on it (afferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as on the previous review: reuse of _is_noise_dir is intentional so walk and git ls-files prune the same trees. detect()’s caller count is pre-existing. Not splitting these helpers in this PR.

Comment thread graphify/detect.py
return _path_is_under(worktree, root) or _path_is_under(root, worktree)


def _git_enumerate_named_worktrees(

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Health regression_git_enumerate_named_worktrees()

fans out to 7 callees (efferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as on the previous review: reuse of _is_noise_dir is intentional so walk and git ls-files prune the same trees. detect()’s caller count is pre-existing. Not splitting these helpers in this PR.

Comment thread graphify/detect.py
)


def _git_enumerate_files(

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Health regression_git_enumerate_files()

fans out to 8 callees (efferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as on the previous review: reuse of _is_noise_dir is intentional so walk and git ls-files prune the same trees. detect()’s caller count is pre-existing. Not splitting these helpers in this PR.

Comment thread graphify/detect.py


def detect(root: Path, *, follow_symlinks: bool | None = None, google_workspace: bool | None = None, extra_excludes: list[str] | None = None, cache_root: Path | None = None, gitignore: bool = True) -> dict:
def detect(root: Path, *, follow_symlinks: bool | None = None, google_workspace: bool | None = None, extra_excludes: list[str] | None = None, cache_root: Path | None = None, gitignore: bool = True, code_only: bool = False) -> dict:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Health regressiondetect()

fans out to 17 callees (efferent coupling); 117 callers depend on it (afferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as on the previous review: reuse of _is_noise_dir is intentional so walk and git ls-files prune the same trees. detect()’s caller count is pre-existing. Not splitting these helpers in this PR.

Comment thread graphify/detect.py
vprint(f"scanning {root} ...")
heartbeat = _Heartbeat("scanning")

def _walk_from(

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Health regression_walk_from()

fans out to 8 callees (efferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as on the previous review: reuse of _is_noise_dir is intentional so walk and git ls-files prune the same trees. detect()’s caller count is pre-existing. Not splitting these helpers in this PR.

Comment thread graphify/detect.py

for p in all_files:
# For memory dir files, skip hidden/noise filtering
def _admit(p: Path) -> tuple:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Health regression_admit()

fans out to 8 callees (efferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as on the previous review: reuse of _is_noise_dir is intentional so walk and git ls-files prune the same trees. detect()’s caller count is pre-existing. Not splitting these helpers in this PR.

@albertbu

albertbu commented Sep 8, 2026

Copy link
Copy Markdown
Author

Reply to the Graphify reviews

First review (symlink / 6805542) — those findings were real. Fixed on f976450: every candidate goes through _resolves_under_root again (not only islink() files), and the memory walk prunes out-of-root symlink directories so followlinks=True cannot admit regular files from the target.

Health / coupling — expected. Git enumeration reuses _is_noise_dir so walk and ls-files prune the same trees. detect()’s caller count is pre-existing. Not splitting those helpers in this PR.

Second review (f976450) — Worth a look

  1. memory_dir files may be lost when git enumeration replaces all_files — not a bug. After git_enum assigns all_files, we still merge graphify-out/memory via _walk_from(..., in_memory=True, descend=True) whenever that directory exists. Git usually never lists graphify-out/ (gitignored); the extra walk is what keeps memory files. The out_prefix/mem_prefix filter only drops other graphify-out/ paths if Git happened to list them.

  2. code_only skips Google Workspace/Office conversion but stores the original binarycode_only here means “skip word-count / skip expensive conversion”, not “drop documents”. test_detect_code_only_skips_word_count still expects .md documents in the result. Office/Workspace names pass _maybe_code_filename because they are corpus extensions. Storing the original path without converting is the cheap path; extract cannot read those binaries as text. Skip-vs-convert for those types is a separate detect-policy change, not a git-enum bug.

  3. non-descend walk clears dirnames only when not in_memory — not reachable. _walk_from(..., descend=False) is only used for the scan root (in_memory=False). The memory walk is always descend=True. dirnames.clear() is not used to stop a memory walk.

  4. Escaping .gitmodules path can make scanner run Git outside the scan rootgit -C may run on an overlapping ancestor when the scan root is a subfolder of a superproject. _ingest_git_relpaths drops any path that is not under the scan root. _worktree_overlaps_scan skips worktrees that neither contain nor are contained by the scan root, so a path = /tmp/evil gitmodules entry is not enumerated unless it actually overlaps. Admitted files cannot escape the scan root.

  5. Concurrent subtree walkers mutate shared ignored_dirs — True-only memo. Duplicate add of the same Path is idempotent; a missed memo only repeats _is_scan_ignored. Walkers do not iterate the set. CPython’s GIL makes set.add / in atomic. A lock would not change scan results.

No further commits planned for these comments.

albertbu and others added 2 commits September 8, 2026 12:58
Use git ls-files, Google repo manifests, and nested worktrees so ignored
trees are pruned at directory granularity instead of a Python walk.

Co-authored-by: Cursor <cursoragent@cursor.com>
Gating the out-of-root check on islink() admitted regular files reached
by following a symlink directory, including graphify-out/memory/ which
skips ignore pruning. Restore the v8 resolve()-under-root admit guard
and prune those dirs during the memory walk too.

Co-authored-by: Cursor <cursoragent@cursor.com>
@albertbu
albertbu force-pushed the albertb/detect-git-enum branch from f976450 to fced5b7 Compare September 8, 2026 09:59

@graphify-labs graphify-labs Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Graphify reviewed this change.

Worth a look — the grounded gate found no coupling regressions or blocking issues, but 5 advisory finding(s) below merit a look before merge.

Formal verification. No changes could be formally verified in this run.


Graphify review — findings

Adds a Git-aware enumeration path to detect() that lists large trees via git ls-files (--cached plus --others --exclude-standard), Google repo project.list project sets, and nested worktrees/submodules, so gitignored and untracked-but-ignored directories are never walked. Prunes Buildroot output//output_<board>/ sysroot trees and .repo metadata during fallback os.walk, filters to classifiable names under --code-only, and parallelizes word-counting and regular-file checks through the shared thread pool. Emits rate-limited _Heartbeat progress to stderr under --verbose, and falls back to the Python walk when git ls-files fails, is unavailable, or exceeds the 300s timeout.

Worth a look

  • Side-effecting conversions run concurrently and write to shared converted_dirgraphify/detect.py · Escalate · high
    • agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
  • Git pathspec is not escaped for literal subdirectoriesgraphify/detect.py · Escalate · medium
    • agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
  • .gitmodules path traversal reads nested manifests outside scan rootgraphify/detect.py · Escalate · medium
    • agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
  • ignored_dirs cache persists across sibling parallel walks with divergent nested patternsgraphify/detect.py · Escalate · medium
    • agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
  • Child .graphifyignore state can leak into sibling walksgraphify/detect.py · Escalate · medium
    • agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
Analysis details — impact, health, verification

Impact & health

Graphify review

Impact — 2628 functions depend on the 811 functions this change touches.

Health — this change adds coupling hotspots:

  • new: extract() — 559 callers, 43 callees
  • new: _rebuild_code() — 115 callers, 51 callees
  • new: detect() — 117 callers, 17 callees
  • new: _extract_generic() — 18 callers, 26 callees
  • new: save_manifest() — 40 callers, 11 callees
  • new: extract_files_direct() — 17 callers, 20 callees
  • new: extract_js() — 85 callers, 4 callees
  • new: extract_xaml() — 19 callers, 17 callees
  • …and 48 more — each is listed as a finding

Verification — 2628 functions in the blast radius were not formally verified this run (proofs are advisory here).

Gate & verification

graphify gate

PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.

Advisory (not blocking):

  • verification_scope: 1238 function(s) in the blast radius were not formally verified this run

Formal verification

Could not verify: Could not verify detect.

The verifier did not have enough to check detect, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: parameter `root` is annotated `Path` — outside the synthesizable primitive/collection set

Could not verify: Could not verify detect\_incremental.

The verifier did not have enough to check detect\_incremental, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: parameter `root` is annotated `Path` — outside the synthesizable primitive/collection set

Could not verify: Could not verify \_git\_tracked\_path\_keys.

The verifier did not have enough to check \_git\_tracked\_path\_keys, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: parameter `root` is annotated `Path` — outside the synthesizable primitive/collection set

Could not verify: Could not verify \_is\_noise\_dir.

The verifier did not have enough to check \_is\_noise\_dir, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: parameter `parent` is annotated `'Path | None'` — outside the synthesizable primitive/collection set

· 6 grounded finding(s) anchored inline below; 50 more finding(s) on lines outside this diff (see the check run).

Comment thread graphify/detect.py
return name[dot:].lower() in _CORPUS_EXT_LOWER


def _is_noise_dir(part: str, parent: "Path | None" = None) -> bool:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Health regression_is_noise_dir()

10 callers depend on it (afferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.

Comment thread graphify/detect.py
return _path_is_under(worktree, root) or _path_is_under(root, worktree)


def _git_enumerate_named_worktrees(

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Health regression_git_enumerate_named_worktrees()

fans out to 7 callees (efferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.

Comment thread graphify/detect.py
)


def _git_enumerate_files(

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Health regression_git_enumerate_files()

fans out to 8 callees (efferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.

Comment thread graphify/detect.py


def detect(root: Path, *, follow_symlinks: bool | None = None, google_workspace: bool | None = None, extra_excludes: list[str] | None = None, cache_root: Path | None = None, gitignore: bool = True) -> dict:
def detect(root: Path, *, follow_symlinks: bool | None = None, google_workspace: bool | None = None, extra_excludes: list[str] | None = None, cache_root: Path | None = None, gitignore: bool = True, code_only: bool = False) -> dict:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Health regressiondetect()

fans out to 17 callees (efferent coupling); 117 callers depend on it (afferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.

Comment thread graphify/detect.py
vprint(f"scanning {root} ...")
heartbeat = _Heartbeat("scanning")

def _walk_from(

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Health regression_walk_from()

fans out to 8 callees (efferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.

Comment thread graphify/detect.py

for p in all_files:
# For memory dir files, skip hidden/noise filtering
def _admit(p: Path) -> tuple:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Health regression_admit()

fans out to 8 callees (efferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.

…ignores

GitHub review on Graphify-Labs#3287 after the v8 rebase: subdirectory ls-files pathspecs
were globs, .gitmodules could point outside the superproject, sibling walks
shared ignore memos, and Office/Workspace conversion raced on converted/.

Co-authored-by: Cursor <cursoragent@cursor.com>
@albertbu

albertbu commented Sep 8, 2026

Copy link
Copy Markdown
Author

Reply to the latest Graphify review (fced5b7)

Addressed on 0988353 (stacked PRs rebased).

  1. Side-effecting conversions run concurrently and write to shared converted_dir — Office/Workspace conversion and the follow-up ignore check now run under _admit_lock. Parallel admit still does stat/classify; only the shared converted/ writes are serialized.

  2. Git pathspec is not escaped for literal subdirectories — subdirectory git ls-files now uses :(literal)… so a scan of app cannot glob-match app-test. Covered by test_detect_git_pathspec_is_literal_not_glob.

  3. .gitmodules path traversal reads nested manifests outside scan root — submodule paths are rejected unless they stay under the superproject (.., absolute, and ~ entries dropped). Nested .gitmodules is only read for those safe rels. Covered by test_detect_ignores_gitmodules_path_outside_superproject.

  4. ignored_dirs cache persists across sibling parallel walks — each _walk_from now has its own True-only memo instead of sharing ignored_dirs across thread-pool siblings.

  5. Child .graphifyignore state can leak into sibling walks — sibling walks already start from an ancestor snapshot; nested patterns are local copies. _is_ignored is anchor-scoped, so a kept.py rule in pkgA cannot drop pkgB/kept.py. Covered by test_detect_sibling_graphifyignore_does_not_ignore_other_tree.

Health/coupling flags: same as before — not splitting detect() helpers in this PR.

@graphify-labs graphify-labs Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Graphify reviewed this change.

Worth a look — the grounded gate found no coupling regressions or blocking issues, but 5 advisory finding(s) below merit a look before merge.

Formal verification. No changes could be formally verified in this run.


Graphify review — findings

Adds Git-index-driven enumeration to detect(): large trees are walked via git ls-files, Google repo project.list, and nested worktrees/submodules so gitignored directories are never traversed, with an os.walk fallback when Git is unavailable or times out (300s). Prunes Buildroot output//output_<board> sysroots and the .repo metadata store, filters non-classifiable names under --code-only, and parallelizes regular-file checks and word counting through a thread pool with rate-limited --verbose heartbeats. Hardens the path handling so subdirectory pathspecs match literally rather than as globs (_git_literal_pathspec), submodule/.gitmodules paths escaping the superproject are rejected (_safe_relpath_under), and directory gitlinks are enumerated from their own worktree rather than treated as files.

Worth a look

  • gitignore=False is ignored on git-enumerated rootsgraphify/detect.py · Escalate · high
    • agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
  • git ls-files subprocess optimization removed, tracked_files/tracked_dirs now always emptygraphify/detect.py · Escalate · high
    • agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
  • _git_tracked_path_keys now records non-regular paths (deleted/gitlink entries) as tracked filesgraphify/detect.py:1624 · Escalate · medium
    • agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
  • detect no longer preserves git-tracked ignored filesgraphify/detect.py:2292 · Escalate · medium
    • agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
  • --code-only admits Office/Google document files instead of excluding non-codegraphify/detect.py:2630 · Escalate · medium
    • agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
Analysis details — impact, health, verification

Impact & health

Graphify review

Impact — 2638 functions depend on the 821 functions this change touches.

Health — this change adds coupling hotspots:

  • new: extract() — 559 callers, 43 callees
  • new: _rebuild_code() — 115 callers, 51 callees
  • new: detect() — 120 callers, 17 callees
  • new: _extract_generic() — 18 callers, 26 callees
  • new: save_manifest() — 40 callers, 11 callees
  • new: extract_files_direct() — 17 callers, 20 callees
  • new: extract_js() — 85 callers, 4 callees
  • new: extract_xaml() — 19 callers, 17 callees
  • …and 48 more — each is listed as a finding

Verification — 2638 functions in the blast radius were not formally verified this run (proofs are advisory here).

Gate & verification

graphify gate

PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.

Advisory (not blocking):

  • verification_scope: 1248 function(s) in the blast radius were not formally verified this run

Formal verification

Could not verify: Could not verify detect.

The verifier did not have enough to check detect, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: parameter `root` is annotated `Path` — outside the synthesizable primitive/collection set

Could not verify: Could not verify detect\_incremental.

The verifier did not have enough to check detect\_incremental, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: parameter `root` is annotated `Path` — outside the synthesizable primitive/collection set

Could not verify: Could not verify \_git\_tracked\_path\_keys.

The verifier did not have enough to check \_git\_tracked\_path\_keys, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: parameter `root` is annotated `Path` — outside the synthesizable primitive/collection set

Could not verify: Could not verify \_is\_noise\_dir.

The verifier did not have enough to check \_is\_noise\_dir, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: parameter `parent` is annotated `'Path | None'` — outside the synthesizable primitive/collection set

· 6 grounded finding(s) anchored inline below; 50 more finding(s) on lines outside this diff (see the check run).

Comment thread graphify/detect.py
return name[dot:].lower() in _CORPUS_EXT_LOWER


def _is_noise_dir(part: str, parent: "Path | None" = None) -> bool:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Health regression_is_noise_dir()

10 callers depend on it (afferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.

Comment thread graphify/detect.py
return _path_is_under(worktree, root) or _path_is_under(root, worktree)


def _git_enumerate_named_worktrees(

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Health regression_git_enumerate_named_worktrees()

fans out to 7 callees (efferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.

Comment thread graphify/detect.py
)


def _git_enumerate_files(

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Health regression_git_enumerate_files()

fans out to 8 callees (efferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.

Comment thread graphify/detect.py


def detect(root: Path, *, follow_symlinks: bool | None = None, google_workspace: bool | None = None, extra_excludes: list[str] | None = None, cache_root: Path | None = None, gitignore: bool = True) -> dict:
def detect(root: Path, *, follow_symlinks: bool | None = None, google_workspace: bool | None = None, extra_excludes: list[str] | None = None, cache_root: Path | None = None, gitignore: bool = True, code_only: bool = False) -> dict:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Health regressiondetect()

fans out to 17 callees (efferent coupling); 120 callers depend on it (afferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.

Comment thread graphify/detect.py
vprint(f"scanning {root} ...")
heartbeat = _Heartbeat("scanning")

def _walk_from(

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Health regression_walk_from()

fans out to 8 callees (efferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.

Comment thread graphify/detect.py

for p in all_files:
# For memory dir files, skip hidden/noise filtering
def _admit(p: Path) -> tuple:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Health regression_admit()

fans out to 8 callees (efferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant