[RFC / Benchmark Proposal]: Twin Architecture Harness for Evaluating False Positives vs Exploit Verification

[yunusarda-sudo]

Background & Motivation

Autonomous security scanners and SAST engines often face a sharp trade-off between under-reporting (false negatives) and alert fatigue (false positives). A common failure mode in heuristic and early agentic evaluations is flagging standard defensive idioms (e.g. prepared SQL statements, allowlisted protocol/domain parsing, safe path boundary assertions) simply because source-sink tokens match.

To establish reproducible metrics for agentgg's agentic verification across model backends, we propose a standardized Twin Architecture Benchmark Suite.

Benchmark Architecture: The Twin Matrix

Each evaluation unit consists of a paired, functionally comparable implementation:

• Variant A (Vulnerable): Contains a verified, exploitable pattern.
• Variant B (Benign Twin): Implements the same business intent using recommended secure defensive practices.

Initial Test Matrix

Category	Vulnerable Variant (A)	Benign Twin (B)	Expected Verdict
SSRF	Raw requests.get(user_url)	Strict scheme + domain allowlist validation	A: Flagged / B: Clear
SQLi	String concatenation in query	Parameterized query execution	A: Flagged / B: Clear
Path Traversal	open(os.path.join(base, path))	os.path.commonpath boundary assertion	A: Flagged / B: Clear
Command Injection	subprocess.run(f"ping {ip}", shell=True)	Array args with strict IP validation (shell=False)	A: Flagged / B: Clear

Evaluation Metrics & Harness

The harness evaluates the analyzer output against the twin dataset and computes:

• True Positives (TP) & False Positives (FP)
• True Negatives (TN) & False Negatives (FN)
• Precision, Recall, and False Positive Rate (FPR):
  FPR = FP / (FP + TN)

Proof of Concept

We have prototyped a standalone executable harness with paired synthetic samples. Running a naive regex-based detector against it immediately surfaces the expected baseline failure:
{'total_samples': 6, 'precision': 0.5, 'recall': 1.0, 'false_positive_rate': 1.0}

Proposed Next Steps

1. Would the maintainers be interested in having this benchmark harness integrated under packages/cli/__tests__/ (or an evaluation fixture package)?
2. We can provide either TypeScript/Vitest test suites matching the existing monorepo structure, or standalone dataset fixtures.

Feedback and suggestions from the team are welcome!