Discussion: statistical significance qualifier filtering — filtering belongs in Gandalf, Shepherd-side filter removal

[SkyeAv]

Context

Following the Translator engineering call, we had an async cross-team discussion about how the new statistical significance qualifier should be filtered by default, and where in the stack that filtering should live. Participants: @tokebe (Retriever), @maximusunc (Gandalf), @SkyeAv (ARAX + Shepherd). This issue records those decisions in writing so they can be referenced from subsequent issues and PRs; until now they existed only in the engineering call notes.

Decisions

• Filtering is a Gandalf thing. Any implicit filtering should happen in Gandalf itself — filtering before retrieving all possible paths is always going to be faster, and Gandalf is not the performance bottleneck. It does not belong in Shepherd or anywhere else in the stack.
• Shepherd does not filter. The filtering portion of PR feat: ranking for biolink:statistical_significance_qualifier #135 will be removed; the ranking and scoring work stays (tracked in feat: ranking & filtering support for biolink:statistical_significance_qualifier #134).
• Defaults are overridable via TRAPI. The statistical significance qualifier gets a default significance cutoff (value TBD) that standard TRAPI qualifier constraints can override. The same discussion covered negated edges, which follow the same default-and-override shape (excluded by default, clients opt in via query).

Open questions

• What is the exact TRAPI standard for communicating the cutoff override (and filter overrides generally)? A companion Retriever issue will coordinate the override standard and the Retriever-side pass-through.

Related

• feat: ranking & filtering support for biolink:statistical_significance_qualifier #134 — ranking and filtering support for the statistical significance qualifier (tracking issue)
• feat: ranking for biolink:statistical_significance_qualifier #135 — PR: ranking and filtering for the statistical significance qualifier (filtering portion to be removed per above)
• Filter negated evidence from Aragorn creative results #127 — PR: filter negated evidence from Aragorn creative results
• feat: rank and filter logic for biolink:statistical_significance_qualifier RTXteam/RTX#2859 — PR: rank and filter logic for the statistical significance qualifier (equivalent ARAX work)
• retriever#197 — Retriever pass-through for the statistical significance qualifier
• TRAPI standard for overriding implicit filtering defaults (statistical significance cutoff, negated edges) retriever#198 — Companion Retriever coordination issue: TRAPI override standard and Retriever pass-through
• feat: statistical significance qualifier on associations biolink/biolink-model#1766 — upstream qualifier definition

[SkyeAv]

Shepherd-side filter removal done in #135: filter_edges_by_statistical_significance (shared function, thin worker, compose service, and the Aragorn/BTE lookup-workflow wiring) has been dropped from that PR; ranking/scoring stays. Remaining #136 items — Gandalf-side filtering, the TRAPI override standard, and the Retriever pass-through — are tracked in their respective repos.