ai-coding-reviewer

中文版: README.zh-CN.md

AI-to-AI code review gate for GitHub PRs. Two LLM models independently review every diff; both must vote PASS, and confidence thresholds (0.90 for security/correctness, 0.80 for performance/maintainability) gate the merge whenever the reviewers disagree or a finding of MEDIUM severity or higher is reported. The voting rule itself is selectable per change via a declared low | medium | high risk level — see Risk levels.

---

Quick start (3 steps)

1. Add the workflow

Copy .github/workflows/ai-review.yml to your repo:

name: AI Code Review

on:
  pull_request:
    branches: [main]

jobs:
  ai-review:
    name: AI Review Gate
    runs-on: ubuntu-latest
    permissions:
      contents: read

    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Set image tag
        run: |
          IMAGE=$(echo "ai-reviewer-pr:${{ github.event.pull_request.head.sha || github.sha }}" | tr '[:upper:]' '[:lower:]')
          echo "IMAGE=${IMAGE}" >> "$GITHUB_ENV"

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Build reviewer image from PR source
        uses: docker/build-push-action@v6
        with:
          context: .
          load: true
          push: false
          tags: ${{ env.IMAGE }}
          cache-from: type=gha
          cache-to: type=gha,mode=max

      - name: Generate diff
        run: git diff origin/${{ github.base_ref }}...HEAD > pr.diff

      - name: Run AI review
        env:
          REVIEWER_1_API_KEY: ${{ secrets.REVIEWER_1_API_KEY }}
          REVIEWER_2_API_KEY: ${{ secrets.REVIEWER_2_API_KEY }}
          # Optional provider overrides — see "Changing providers" below.
          # Empty values fall back to the built-in defaults (minimax + deepseek).
          REVIEWER_1_MODEL:    ${{ vars.REVIEWER_1_MODEL }}
          REVIEWER_2_MODEL:    ${{ vars.REVIEWER_2_MODEL }}
          REVIEWER_1_BASE_URL: ${{ vars.REVIEWER_1_BASE_URL }}
          REVIEWER_2_BASE_URL: ${{ vars.REVIEWER_2_BASE_URL }}
        run: |
          docker run --rm \
            -v "${{ github.workspace }}:/repo" \
            -e REVIEWER_1_API_KEY \
            -e REVIEWER_2_API_KEY \
            -e REVIEWER_1_MODEL \
            -e REVIEWER_2_MODEL \
            -e REVIEWER_1_BASE_URL \
            -e REVIEWER_2_BASE_URL \
            "$IMAGE" \
            --diff /repo/pr.diff \
            --policy /repo/policy.md \
            --source-root /repo \
            --output /repo/review-report.md

      - name: Upload report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: ai-review-report
          path: review-report.md

2. Add a policy file

Create policy.md in your repo root. This is injected verbatim into the LLM prompt:

## Security
- SEC-001: Never log credentials, tokens, or PII
- SEC-002: All SQL queries must use parameterized statements

## Correctness
- LOGIC-001: Every error must be handled or explicitly propagated

See review-policy.example.md for a full template.

3. Add secrets, variables, and enable branch protection

Both live under repo Settings → Secrets and variables → Actions — but on different tabs.

Secrets (required) — Secrets tab:

Secret	Description
REVIEWER_1_API_KEY	API key for the first reviewer model
REVIEWER_2_API_KEY	API key for the second reviewer model

Variables (optional) — Variables tab. Skip these to use the built-in defaults (MiniMax + DeepSeek):

Variable	Description	Example
REVIEWER_1_MODEL	Model ID for reviewer 1	MiniMax-M2.7
REVIEWER_1_BASE_URL	OpenAI-compat endpoint for reviewer 1	https://api.minimax.chat/v1
REVIEWER_2_MODEL	Model ID for reviewer 2	deepseek-chat
REVIEWER_2_BASE_URL	OpenAI-compat endpoint for reviewer 2	https://api.deepseek.com/v1

Don't put model IDs / base URLs in the Secrets tab — Secrets are write-only in the UI and aren't sensitive here. Variables are editable and visible.

Branch protection — Settings → Branches → Add rule:

• Branch: main
• Enable Require status checks to pass before merging
• Add status check: AI Review Gate

---

Changing providers

Defaults are MiniMax (reviewer 1) + DeepSeek (reviewer 2). Override either slot via these env vars:

Variable	Purpose
REVIEWER_1_BASE_URL	OpenAI-compat base URL for reviewer 1
REVIEWER_2_BASE_URL	OpenAI-compat base URL for reviewer 2
REVIEWER_1_MODEL	Model ID for reviewer 1
REVIEWER_2_MODEL	Model ID for reviewer 2

Where to set them:

• GitHub Actions — repo Settings → Secrets and variables → Actions → Variables tab. Add REVIEWER_1_MODEL etc. as repo Variables. The Quick start workflow above already wires them through; leave them unset to fall back to the defaults.
• Local CLI — export them in your shell before cargo run (or pass --reviewer-1-model <id> on the command line).
• Docker locally — docker run -e REVIEWER_1_MODEL=... -e REVIEWER_1_BASE_URL=... ....

Any provider with an OpenAI-compatible /v1/chat/completions endpoint works (MiniMax, DeepSeek, Anthropic via proxy, Gemini, OpenAI, local Ollama, etc.).

---

How it works

PR diff
  └─ parse → AST context extraction (13 languages)
       └─ prompt assembly (system: policy + schema | user: diff + symbols)
            └─ Model A ──┐
                          ├─ consensus: both PASS + (no ≥MEDIUM finding OR per-focus confidence cleared)?
            └─ Model B ──┘
                 └─ review-report.md  +  exit 0 (PASS) / 1 (FAIL) / 2 (error)

• Both models run concurrently and review the same diff independently.
• A finding is reported only when both models agree on the same file + line + rule.
• The gate fails immediately if either model votes FAIL.
• When both vote PASS, the per-focus confidence threshold — 0.90 for security/correctness, 0.80 for performance/maintainability — is enforced as a tiebreaker, applied only when the merged findings contain something at MEDIUM severity or higher. A clean PASS with only LOW/INFO findings is allowed through regardless of confidence, since "uncertain style judgement" shouldn't be gated as hard as RCE-class defects.
• The voting rule above is the medium risk level — the default. The gate also supports low and high levels declared per change. See Risk levels below.
• Transient errors (timeout, 5xx, 429) are retried with exponential backoff; auth errors (401, 403, 404) fail fast.
• AST context (full symbol definitions + intra-file callee name hints) is attached for 13 languages: Rust, Python, Go, JS/TS, Java, C, C++, Ruby, C#, Bash, Scala. Other languages are diff-only. See Limitations & roadmap for what "callee name hints" actually means — it is not a resolved cross-file call graph.

---

Risk levels

A docs typo and a crypto rewrite shouldn't share the same review bar. The developer (or the AI agent doing the change) declares a risk level for the change, and the gate picks a voting rule to match. Per-focus confidence thresholds are unchanged across all levels.

Risk	Pass condition	Use for
low	either reviewer votes PASS — confidence ignored	docs, tests, comment-only edits, isolated typo fixes
medium	both PASS, with per-focus confidence enforced when verdicts disagree or a MEDIUM+ finding is present (the historical default)	ordinary feature work, refactors, bug fixes
high	both PASS and both clear the per-focus confidence threshold unconditionally — even with zero findings	auth, crypto, payments, schema migrations, public API contracts

The declaration is trusted verbatim. The reviewer does not second-guess a low declaration, even when the diff touches sensitive code; an out-of-band reviewer (human or another tool) is expected to police obviously bad calls. The declared level is rendered prominently in review-report.md and printed on stdout to make abuse easy to spot.

Declaring risk on a PR

The simplest path is a PR label or a body line. The CLI reads --risk-level (or env var REVIEWER_RISK_LEVEL); your CI step extracts the value before invoking the binary:

- name: Resolve risk level from PR label
  id: risk
  env:
    LABELS: ${{ toJSON(github.event.pull_request.labels.*.name) }}
  run: |
    level=$(echo "$LABELS" | jq -r '.[]' | grep -E '^risk:(low|medium|high)$' | head -n1 | cut -d: -f2)
    echo "level=${level:-medium}" >> "$GITHUB_OUTPUT"

- name: Run AI Review
  env:
    REVIEWER_1_API_KEY: ${{ secrets.REVIEWER_1_API_KEY }}
    REVIEWER_2_API_KEY: ${{ secrets.REVIEWER_2_API_KEY }}
    REVIEWER_RISK_LEVEL: ${{ steps.risk.outputs.level }}
  run: |
    docker run --rm -i ${{ env.IMAGE }} --diff - --policy policy.md < pr.diff

Apply a PR label risk:low, risk:medium, or risk:high; missing label → defaults to medium (zero behaviour change vs. the historical gate).

Locally:

git diff HEAD~1 | cargo run -- --diff - --policy policy.md --risk-level high

---

Recommended PR size

Every one of the 8 concurrent LLM calls receives the full diff plus AST context for all changed files — per-call prompt size grows roughly linearly with the PR. Review quality degrades in long contexts (typically around 32k+ tokens) well before any hard window limit, so keeping PRs small is the cheapest way to keep reviews sharp.

Tier	Files	Net diff	Expectation
Sweet spot	≤ 10	≤ ~500 lines	Fast, low-noise reviews
Acceptable ceiling	≤ 30	≤ ~1500 lines	Works, but slower and noisier
Split required	> 30 or > 1500 lines		Partition before review

How to split: prefer atomic commits organized by module or feature, then open one PR per logical unit. "One change, one PR" is the goal — not "one commit, one PR".

Natural exceptions — these are legitimately large and should be routed through a dedicated review process rather than this gate:

• Mass renames / signature changes that ripple across the codebase
• Dependency upgrades and their mechanical call-site adaptations
• Generated code: lock files, protobuf stubs, OpenAPI clients, migrations

---

Local usage

# Requires REVIEWER_1_API_KEY and REVIEWER_2_API_KEY in env
cargo run -- --diff path/to/file.diff --policy policy.md --source-root .

# From stdin
git diff HEAD~1 | cargo run -- --diff - --policy policy.md

# Verbose
cargo run -- --diff - --policy policy.md -v

Exit codes: 0 = PASS, 1 = FAIL, 2 = fatal error.

---

For AI tools

tool: ai-reviewer
type: CLI binary / Docker image
language: Rust
purpose: >
  Dual-LLM code review gate. Parses a unified diff, extracts AST context,
  sends to two LLM models in parallel, and produces PASS/FAIL + Markdown report.

interface:
  binary: ai-reviewer
  docker: ghcr.io/<owner>/<repo>:latest
  args:
    --diff:        path to unified diff file, or "-" to read from stdin (required)
    --policy:      path to Markdown policy file injected into the system prompt (required)
    --source-root: repo root for AST context extraction and tool-call file access (optional)
    --output:      path for the Markdown report (default: review-report.md)
    --risk-level:  low | medium | high — selects the gate's voting rule (default: medium)
    -v/--verbose:  enable verbose tracing output

env_vars:
  required:
    REVIEWER_1_API_KEY: API key for the first LLM reviewer
    REVIEWER_2_API_KEY: API key for the second LLM reviewer
  optional:
    REVIEWER_1_BASE_URL:  OpenAI-compat base URL (default: provider-specific)
    REVIEWER_2_BASE_URL:  OpenAI-compat base URL (default: provider-specific)
    REVIEWER_1_MODEL:     model ID override for reviewer 1
    REVIEWER_2_MODEL:     model ID override for reviewer 2
    REVIEWER_RISK_LEVEL:  declared risk level for this change — low | medium | high (default: medium)
    RUST_LOG:             tracing filter, e.g. "ai_reviewer=debug"

exit_codes:
  0: consensus PASS — both models voted PASS and any required confidence threshold was met
  1: consensus FAIL — verdict mismatch, both-FAIL, or per-focus confidence below threshold when a >= MEDIUM finding or disagreement was present
  2: fatal error — bad diff path, missing policy file, diff too large, etc.

outputs:
  stdout: single summary line (PASS/FAIL + finding count)
  stderr: gate-fail message when exit code is 1
  file:   full Markdown cross-comparison report (default: review-report.md)

key_source_files:
  src/main.rs:          CLI entrypoint, provider wiring, concurrent orchestration
  src/diff.rs:          unified diff parser → FileDiff / HunkRange
  src/ast.rs:           tree-sitter AST extraction + intra-file callee name hints (13 languages)
  src/prompt.rs:        system prompt + user prompt assembly
  src/models/reviewer.rs: shared retry loop, JSON self-correction, strip helpers
  src/models/mod.rs:    ReviewResult / ConsensusResult types + JSON schema
  src/consensus.rs:     PASS/FAIL gate logic, finding dedup
  src/report.rs:        Markdown report renderer
  src/tools.rs:         LLM tool implementations (read_file, find_symbol)
  src/telemetry.rs:     Prometheus metrics export

consensus_rule: >
  Voting rule is selected by the declared --risk-level (default: medium).
    low    — either reviewer voting PASS is enough; confidence ignored.
    medium — both must vote PASS; per-focus confidence (0.90 sec/correct,
             0.80 perf/maint) enforced only when verdicts disagree or a
             >= MEDIUM finding is reported. Historical default.
    high   — both must vote PASS AND both must clear the per-focus
             confidence threshold unconditionally, even with zero findings.
  Any ReviewError becomes a synthetic Verdict::Fail with confidence=1.0.
  Finding dedup key: (file, line_start, rule_id).

retry_policy: >
  Up to max_retries+1 attempts per LLM call.
  Retryable: timeout, 5xx, 429, network reset — exponential backoff 1s→2s→4s→8s→16s.
  Non-retryable: 401, 403, 404, invalid API key — fail immediately.
  JSON parse failure also retries with a self-correction prompt (self-contained, no history accumulation).

pr_size_guidance:
  rationale: >
    All 8 concurrent LLM calls receive the full diff plus AST context for every changed file.
    Per-call prompt size grows roughly linearly with the PR; review quality degrades in long
    contexts (≈ 32k+ tokens) well before the model's hard window limit.
  sweet_spot:        "<= 10 files, <= ~500 lines net diff"
  acceptable_ceiling: "<= 30 files, <= ~1500 lines net diff"
  split_required:    "> 30 files OR > 1500 lines — partition before review"
  split_strategy:    "atomic commits per module/feature; one logical change per PR"
  natural_exceptions: >
    Mass renames, signature ripples, dependency upgrades, and generated code (lock files,
    protobuf stubs, OpenAPI clients, DB migrations) are legitimately large — route through
    a dedicated review process rather than this gate.

---

Limitations & roadmap

What the "AST context" actually contains today

To set expectations correctly for anyone wiring this up: the AST context shipped to each model contains full symbol definitions for changed functions/types plus single-file callee name hints — not a resolved call graph.

Concretely, for each changed symbol the tool emits a list of bare identifier strings that appear at call sites inside that symbol's body, scoped to the same file:

• ✅ Full source of every changed function / class / struct / impl / trait
• ✅ Bare names of callees that appear inside changed symbols (e.g. verify_token, push, new)
• ✅ Inlined definitions of any callee whose name happens to match another symbol in the same file
• ❌ No symbol resolution — auth::verify_token, self.verify_token, and a local verify_token all collapse to the string "verify_token"
• ❌ No type inference — vec.push() and string.push() both produce the edge … → "push"
• ❌ No cross-file edges — callers in other files are never linked, and callees defined in other files are emitted as bare names with no body
• ❌ No reverse edges — the tool never answers "who calls this changed function?"

So the callee data should be read as "these names were referenced from inside the changed code", not "these specific functions are called". The internal type is still named CallEdge for historical reasons, but the prompt now labels the section "Callee Name Hints" so reviewer models don't over-trust it.

Roadmap

A real cross-file call graph (resolved fully-qualified names, type-aware method dispatch, reverse edges, blast-radius queries) is on the planned-work list. The likely path is to delegate resolution to a real indexer rather than reimplementing it on top of tree-sitter — candidates being evaluated:

• rust-analyzer / SCIP / LSIF indexes consumed at review time
• stack-graphs for multi-language name resolution
• LSP "find references" against a warm language server

If you have a strong opinion on which direction to take, open an issue.

---

License

MIT