Skip to content

Create pipeline for symbol reachability#2151

Open
ziadhany wants to merge 16 commits into
aboutcode-org:mainfrom
ziadhany:add-symbol-reachability
Open

Create pipeline for symbol reachability#2151
ziadhany wants to merge 16 commits into
aboutcode-org:mainfrom
ziadhany:add-symbol-reachability

Conversation

@ziadhany

@ziadhany ziadhany commented Apr 22, 2026

Copy link
Copy Markdown
Collaborator

Issues

Changes

Add Pipeline(s) that can retrieve vulnerable / fix symbols, collect local symbols (pur2sym) and match them
Pipeline Graph:

flowchart TD
    %% Main Entry Point
    Start([collect_and_store_symbol_reachability_results]) --> GetResources[Filter candidate_resources]
    GetResources --> GroupPatches[Group fixed_in_patches by vcs_url]
    GroupPatches --> RepoLoop{For each repository}

    %% Repository Processing
    RepoLoop -->|Process Repo| GitContext[GitRepositoryContext]
    GitContext -->|Clone & Enter| PatchLoop{For each patch}
    
    %% Patch Processing
    PatchLoop -->|Next Patch| GitCheckout[Checkout commit_hash]
    GitCheckout --> GenReport[generate_reachability_report]
    
    %% Patch Analysis Stage
    GenReport --> PatchAnalyzer[PatchAnalyzer.collect_patch_symbols]
    PatchAnalyzer --> AnalyzeDiff[Get Diff & Changed Files]
    AnalyzeDiff --> PatchAnalyzeLoop[Analyze vulnerable & fixed texts]
    PatchAnalyzeLoop --> DetectLang[detect_language_with_scancode]
    DetectLang --> ParsePatchAST[LanguageQuery.parse_code_to_ast]
    ParsePatchAST --> ExtractPatchSymbols[SymbolExtractor: Extract Changed Symbols]
    ExtractPatchSymbols --> DiffSymbols[diff_changed_symbols]
    DiffSymbols --> ResourceLoop{For each candidate_resource}

    %% Resource Matching Stage
    ResourceLoop -->|Next Resource| LangCheck{Language matches patch?}
    LangCheck -->|No| ResourceLoop
    LangCheck -->|Yes| ResAnalyzer[ResourceAnalyzer.build_index]
    
    ResAnalyzer --> ParseResAST[Parse Resource to AST]
    ParseResAST --> IndexResSymbols[Extract definitions, imports, calls]
    IndexResSymbols --> Matcher[ResourcePatchMatcher.match]
    
    Matcher --> MatchVuln[Match Vulnerable Symbols]
    Matcher --> MatchFixed[Match Fixed Symbols]
    
    MatchVuln & MatchFixed --> CheckEvidence{Evidence found?}
    CheckEvidence -->|No| ResourceLoop
    
    %% Reporting Stage
    CheckEvidence -->|Yes| Classify[classify_reachability]
    Classify --> StatusReach[REACHABLE / POTENTIALLY / NOT_REACHABLE]
    StatusReach --> UpdateData[Update resource.extra_data]
    UpdateData --> ResourceLoop

    %% Loop Backs and Exit
    ResourceLoop -->|All resources checked| PatchLoop
    PatchLoop -->|All patches checked| RepoLoop
    RepoLoop -->|All repos checked| End([End Process])

    %% Styling
    classDef functionNode fill:#e1f5fe,stroke:#03a9f4,stroke-width:2px;
    classDef logicNode fill:#fff3e0,stroke:#ff9800,stroke-width:2px;
    classDef loopNode fill:#f3e5f5,stroke:#9c27b0,stroke-width:2px;
    
    class Start,GenReport,PatchAnalyzer,ResAnalyzer,Matcher,Classify functionNode;
    class RepoLoop,PatchLoop,ResourceLoop loopNode;
    class LangCheck,CheckEvidence logicNode;
Loading

Checklist

  • I have read the contributing guidelines
  • I have linked an existing issue above
  • I have added unit tests covering the new code
  • I have reviewed and understood every line of this PR

@ziadhany ziadhany force-pushed the add-symbol-reachability branch from 944dcbe to da128db Compare June 3, 2026 11:39
@ziadhany

Copy link
Copy Markdown
Collaborator Author

The pipeline requires the unidiff dependency to parse diffs/patch text

@ziadhany ziadhany force-pushed the add-symbol-reachability branch from d5fed3d to 08512ab Compare June 16, 2026 15:54
@ziadhany ziadhany marked this pull request as ready for review June 16, 2026 16:24
@ziadhany

Copy link
Copy Markdown
Collaborator Author

This depends on:

We have an API support for patching VulnerableCode, see:

@ziadhany ziadhany force-pushed the add-symbol-reachability branch from 08512ab to 503af03 Compare June 24, 2026 14:58
ziadhany added 12 commits July 1, 2026 16:04
Signed-off-by: ziad hany <ziadhany2016@gmail.com>
Signed-off-by: ziad hany <ziadhany2016@gmail.com>
Signed-off-by: ziad hany <ziadhany2016@gmail.com>
Signed-off-by: ziad hany <ziadhany2016@gmail.com>
Fix the test

Signed-off-by: ziad hany <ziadhany2016@gmail.com>
Signed-off-by: ziad hany <ziadhany2016@gmail.com>
Add missing logic for imports

Signed-off-by: ziad hany <ziadhany2016@gmail.com>
Signed-off-by: ziad hany <ziadhany2016@gmail.com>
Signed-off-by: ziad hany <ziadhany2016@gmail.com>
Signed-off-by: ziad hany <ziadhany2016@gmail.com>
Improve pipeline structure

Signed-off-by: ziad hany <ziadhany2016@gmail.com>
Fix formating tests

Signed-off-by: ziad hany <ziadhany2016@gmail.com>
@ziadhany ziadhany force-pushed the add-symbol-reachability branch from 3686a85 to dfbb4db Compare July 1, 2026 13:06
Comment thread scanpipe/pipes/symbols.py Outdated
ziadhany added 4 commits July 1, 2026 16:52
Signed-off-by: ziad hany <ziadhany2016@gmail.com>
Fix formating

Signed-off-by: ziad hany <ziadhany2016@gmail.com>
Signed-off-by: ziad hany <ziadhany2016@gmail.com>
Signed-off-by: ziad hany <ziadhany2016@gmail.com>
Comment thread uv.lock

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure if we need to update all these dependency. I just ran uv add unidiff, which updated it automatically.

@ziadhany ziadhany requested a review from TG1999 July 2, 2026 00:58
@ziadhany

ziadhany commented Jul 2, 2026

Copy link
Copy Markdown
Collaborator Author

I'm also not sure why Run unit tests on macOS / run-unit-tests (3.13) is failing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant