passk-reliability

Your agent works in the demo and fails in production. Here's the math.

passk-reliability is a domain-agnostic reliability harness for tool-using agents. It measures repeated-run degradation with a strict pass^k curve, variance profile, and error-accrual chart that teams can use as a CI gate.

pass^k vs pass@k (critical disambiguation)

• pass^k (this repo): task passes only if all k runs succeed.
• pass@k (different metric): task passes if any one of k runs succeeds.

pass^k is intentionally stricter and reflects production reliability under repeated execution.

Core outputs

• pass^1..k curve (strict multiplicative reliability)
• variance profile (mean success rate, stddev, flakiness index)
• error-accrual curve: P(still on-policy | turn N)
• policy-adherence scoring (separate from task success)

CLI

passk-reliability run \
  --config config/reliability_config.json \
  --agent langgraph:./examples/run_langgraph_agent.py \
  --k 8 \
  --task-suite ./examples/generic_task_suite.yaml \
  --output-db report/results.duckdb

passk-reliability dashboard --db report/results.duckdb --serve

passk-reliability ci-gate \
  --db report/results.duckdb \
  --min-passk-rate 0.70 \
  --max-flakiness-index 0.15

GitHub Actions CI Gate snippet

- name: Reliability CI gate
  run: |
    passk-reliability ci-gate \
      --db report/results.duckdb \
      --min-passk-rate 0.70 \
      --max-flakiness-index 0.15

References and attribution

• τ-bench methodology and pass^k framing inspired by Yao et al. (2024), arXiv:2406.12045.
• Task-format inspiration from Sierra's τ2-bench project.
• This project is not affiliated with Sierra.

Open source and governance

• License: MIT
• Contribution guide: CONTRIBUTING.md
• Security policy: SECURITY.md
• Architecture context: docs/ARCHZOS_AGENT_ARCHITECTURE_CONTEXT.md