Skip to content

feat(ci): add opt-in review gate to GitHub Action and GitLab CI - #1461

Open
Qiyuanqiii wants to merge 5 commits into
alibaba:mainfrom
Qiyuanqiii:codex/result-gate-ci
Open

Qiyuanqiii wants to merge 5 commits into
alibaba:mainfrom
Qiyuanqiii:codex/result-gate-ci

Conversation

@Qiyuanqiii

@Qiyuanqiii Qiyuanqiii commented Sep 19, 2026

Copy link
Copy Markdown
Contributor

Description

Summary

  • Add an opt-in shared review gate to the GitHub composite Action and the GitLab CI example, disabled by default.
  • Resolve immutable base/head IDs before review and pass the same IDs to ocr gate over the original review JSON.
  • Publish available review feedback, evaluate the gate, and preserve diagnostic artifacts before enforcing the final CI outcome.
  • Fail gated jobs on review errors, incomplete publication, missing gate evidence, incomplete selected-file coverage, or configured severity violations.
  • Add executable shell-contract tests and GitLab publication regressions.
  • Document both integrations in the English, Chinese, Japanese, Korean, and Russian website guides, with configuration examples, outcome rules, and diagnostic artifact paths.

Dependency and Review Scope

Depends on #1457; merge the shared CLI gate before this integration. This PR targets upstream main, so its aggregate diff currently includes the prerequisite gate implementation as well as the CI integration.

After #1457 is merged, this branch can be rebased onto the resulting upstream history so the prerequisite disappears from the diff, including if the prerequisite is squash-merged.

Motivation and Scope

A successful review process is not sufficient evidence that a CI review requirement has been met. A budget-limited run may leave selected files unreviewed, a complete run may report blocking findings, and a publishing failure may leave contributors without the resulting feedback. Separate interpretations of these outcomes in each CI integration can produce different merge decisions for the same review artifact.

#1457 supplies the common offline policy evaluator. This PR connects the GitHub Action and GitLab example to it while retaining each platform's responsibility for process execution, publication, and artifact retention. For example, a partial budget-limited review can still publish useful findings, but its gate remains inconclusive and the job fails after diagnostics have been saved. Conversely, a gate pass cannot turn a failed review or publishing step into a successful job.

The optional gate also rejects budget-only incomplete coverage through the saved manifest. This complements #1030's operational partial-failure exit policy and works independently of that PR.

Usage and Defaults

Add these inputs to an existing GitHub Action step:

with:
  gate: 'true'
  fail_on_severity: high

For the GitLab example:

variables:
  OCR_GATE: 'true'
  OCR_FAIL_ON_SEVERITY: high
Setting Default Behavior
GitHub gate / GitLab OCR_GATE false Enable the shared gate and publication enforcement. Boolean strings are case-insensitive.
GitHub fail_on_severity Empty Optional critical, high, medium, or low threshold; requires gate: 'true'.
GitLab OCR_FAIL_ON_SEVERITY with the gate enabled Empty Same shared CLI threshold policy.
GitLab OCR_FAIL_ON_SEVERITY with the gate disabled Existing behavior Retain the publisher's legacy severity check.

Enabled severity thresholds ignore case and surrounding whitespace. Invalid policy values fail before installation/model calls. Empty severity leaves the severity check disabled while retaining coverage, delivery, revision, and publication requirements.

Opting in requires an OCR build containing the gate command. Select it through the GitHub ocr_version input or GitLab OCR_VERSION variable. Both integrations probe ocr gate --help before calling the model and report an upgrade instruction if the pinned CLI lacks support.

GitHub Action Implementation

  • Validate the new inputs and expose normalized policy through step outputs.
  • When gated, bypass checkpoint range narrowing and checkpoint updates. Review the full merge-base-to-head range so an older blocking finding cannot disappear merely because a later push only reviews its newest changes. Incremental comment posting and routing remain available.
  • Resolve the head commit and actual merge base immediately before review, then use these immutable IDs for both review and gate expectations.
  • Allocate a fresh temporary directory per Action invocation, including repeated invocations in the same job. Review output, stderr, gate output, gate stderr, and the optional progress FIFO belong to that invocation.
  • Record the original review exit code. In gated mode, attempt publication even when review exits non-zero, then evaluate the original JSON regardless of publication failure.
  • Upload review and gate artifacts before final enforcement, including when policy evaluation fails. Explicit status conditions allow these steps after earlier failures while avoiding publication/enforcement after cancellation.
  • Enforce review success independently of the gate. With the gate enabled, also require successful publication, zero failed inline submissions, a published summary URL, and a successful gate command. Missing outputs are failures, not implicit success.
  • Expose gate_exit_code, which is empty when disabled or not reached. This describes only the gate command, not overall job success.

GitLab CI and Publication Implementation

  • Clear all review, gate, and statistics outputs before fallible setup, preventing a reused runner from uploading a stale passing artifact after validation or installation fails.
  • Freeze the expected head and merge base before review and forward them unchanged to the shared evaluator.
  • In gated mode, run post_review.py --require-publication with its legacy severity setting cleared; the CLI evaluates severity over all original findings.
  • Track confirmed final-summary publication independently of the best-effort URL. An existing sticky-summary URL after a failed update, or the MR landing-page fallback, must not count as evidence that this run's summary was written.
  • Preserve the existing URL-returning helpers and best-effort behavior for callers that do not opt into publication enforcement. Add OCR_SUMMARY_PUBLISHED to the dotenv statistics.
  • Attempt publication and gate evaluation before the final exit even if either returns non-zero. Preserve the review failure first, then publication failure, then gate failure; missing stage results fail closed.
  • Retain review and gate JSON/stderr with artifacts: when: always. Gate files remain empty when disabled or not reached.
  • Correct the configuration note: GitLab's eight-character minimum concerns masked variables, not ordinary visible boolean policy variables.

Outcome Rules

The table assumes other required stages succeed unless stated otherwise:

Condition with the gate enabled Outcome
Complete selected-file coverage, valid evidence, matching revisions, successful publication, and no threshold violation Job passes
A finding meets or exceeds the threshold, including routed or deduplicated findings Gate fails; job fails after publication/artifact handling
Partial coverage, including budget stops; waived items; zero selected items; missing or inconsistent evidence Gate is inconclusive; job fails
Recorded code_comment failure or an expected revision mismatch Gate cannot pass; job fails
Review exits non-zero even if its artifact passes the gate Job fails
An inline submission remains failed, even if a fallback summary succeeds Job fails under the conservative publication policy
Final summary publication is unconfirmed, including a stale or fallback URL Job fails
Gate is unavailable, crashes, or leaves no successful outcome Job fails

Compatibility and Boundaries

  • Both gate switches default to off. Existing supported CLI builds remain usable with the gate disabled.
  • The GitHub Action retains existing review-success/failure job semantics when disabled; GitLab retains its publisher-only severity behavior.
  • Thresholds evaluate the original saved findings; publication routing and deduplication cannot remove them from the gate input.
  • Revision checks compare the reviewed event snapshot. They do not query the live PR/MR head after the model finishes or implement merge-time freshness enforcement.
  • A pass covers the selected set only. Excluded files, artifact authenticity, policy trust, and waiver policy remain outside this integration.

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Refactoring (no functional changes)
  • Documentation update
  • CI / Build / Tooling

How Has This Been Tested?

Validation Environment

  • Windows/amd64, Go 1.26.5, Node.js 24.14.0, Python 3.14, and Git Bash.
  • Current integration baseline: prerequisite commit 8d612afb0975955d96c3ecfddc171c5f9b2f4b58.
  • Initial implementation validation head: fd79de92ebc4d913348c29103f3538a42dd9815e.

Initial Implementation Validation

  • make license-add and project make check through mingw32-make with Git Bash: license headers, English-only source check, module tidying, formatting, and go vet.
  • make test: complete Go suite with the race detector.
  • node scripts/github-actions/action-contract.test.js: 55 tests passed, including the seven new gate contract cases and their scenario matrices.
  • Existing GitHub publication and translation-sync test suites passed.
  • python -m unittest discover -s examples/gitlab_ci -p '*_test.py': 156 tests passed across publication and executable pipeline contracts.
  • Parsed action.yml, the GitLab pipeline, and the modified workflow using the repository's YAML dependency.
  • Action SHA-pin validation, git diff --check, and LF-only verification of all nine changed files. Ran git add --renormalize . before committing.

The full npm run test:github-actions command is not claimed as passing locally: after the passing publication, translation, and Action suites, its existing plugin-contract suite fails to create a symbolic link with Windows EPERM. The same failure was reproduced at the unchanged prerequisite baseline. Linux CI must confirm that suite and the newly added GitLab contract job.

Documentation Completion

Commit b49c13079ca4a7e583b618eca6eec4e2ced59fa3 updates the five website CI guides and the GitHub Actions example README. The guides cover the opt-in settings, CLI requirement, three gate decisions, full-range review cost, publication failures, and review/gate artifacts. GitLab setup now includes post_review.py, the version variable, and project-relative .ocr/ paths. GitHub troubleshooting explains the effect of failed inline comments when the gate is enabled.

Documentation checks passed: UTF-8/LF, Markdown fences, Python snippet syntax, five-language configuration consistency, translation sync, repository links, and git diff --check. The 55 Action contract tests passed after the documentation update.

Regression and Integration Coverage

  • Gate-disabled compatibility with a CLI lacking the command, invalid input rejection, and capability checks before model execution.
  • Full-range review despite configured checkpoints, immutable revision forwarding, and independent per-invocation artifacts.
  • Gate evaluation after review failures; preservation of decision JSON/stderr; missing revision rejection.
  • Independent review, publication, and gate failures, including absent outputs and shell/step failures.
  • GitLab severity ownership, publish-before-enforce ordering, stale output removal, and failure artifact declarations.
  • Confirmed versus stale summary publication, failed inline submission with a successfully published fallback summary, malformed input, empty/null findings, and default best-effort compatibility.

Executed 10 additional offline integration scenarios through the actual YAML shell blocks using the built ocr gate binary: complete review, high-severity finding, budget-only partial review, mismatched head, and non-zero review exit with an otherwise passing artifact, for both platforms. Review and hosting operations were test doubles; the gate command was real. No live model or hosting API was used in these integration scenarios.

Pre-commit Review

Completed the repository-required ocr review --audience agent --background ... with exit 0: 4 selected production files reviewed, 0 skipped; no critical, high, or medium findings. The test and documentation diffs were also inspected separately.

OCR reported two low-severity suggestions, both checked against the implementation before committing:

  • The alleged uppercase gate/checkpoint mismatch does not occur because GitHub expression string comparisons ignore case.
  • A global safe.directory '*' setup is unnecessary for the new GitLab contract job: the harness places its Git doubles first inside Bash and runs the pipeline in temporary fixtures, rather than operating on the checkout with real Git.

The checkpoint condition now consistently consumes the normalized validation output. A follow-up pre-commit review of that change completed with 0 findings; all 55 Action contract tests and make check passed. The synchronized prerequisite also rejects zero-count tool failure entries: its gate/CLI race tests and make check passed, with 100% gate-package coverage. The initial model review recovered two transient request errors and completed the selected scope.

Known Limitations and Follow-up

  • The upstream aggregate diff includes feat: add opt-in gate for saved review results #1457 until the branch is updated after that prerequisite lands.
  • Enabling the integration requires an OCR CLI build that includes the prerequisite command.
  • Local checks execute the real shell blocks with controlled subprocesses, and the additional smoke checks use the real gate binary. A live hosted GitHub/GitLab end-to-end run with model and publication credentials has not been performed.
  • Live-head freshness, artifact authentication, selection/exclusion evidence, and waiver semantics remain separate follow-up work.
  • The full Node aggregate has the documented baseline Windows symlink limitation; it remains for Linux CI to validate.

Checklist

  • Code follows the project's coding style and required automated checks described above.
  • Added regression tests for the new behavior.
  • Updated both integration guides and documented defaults, failure behavior, and compatibility.
  • Completed automated pre-commit review and inspected the generated changes.
  • All aggregate suites are confirmed passing in CI; the local Node aggregate limitation is documented above.
  • Contributor has personally reviewed all generated code and text before requesting maintainer review.
  • Contributor has confirmed CLA coverage for this contribution.
  • Contributor will answer maintainer questions and review comments from their own understanding, as required by the contribution guidelines.

AI/LLM Disclosure

Codex (GPT-6) was used to implement inspect the changes, run validation and prepare this PR description. The repository-required model review used open-code-review CLI v1.11.9 with deepseek-v4-flash.

Related Issues and Pull Requests

Depends on #1457. Tracked in #1489 alongside #1030. Related operational failure report: #1027.

@github-actions

github-actions Bot commented Sep 19, 2026

Copy link
Copy Markdown
Contributor

🔍 OpenCodeReview found 2 issue(s) in this PR.

  • ✅ Successfully posted inline: 2 comment(s)

Comment thread .github/workflows/action-contract.yml
Comment thread action.yml Outdated
@Qiyuanqiii

Copy link
Copy Markdown
Contributor Author

I forgot to add the relevant md docs; I’ll complete them at noon.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant