Skip to content

fix(overseer): refuse decisions on a queued result a rerun replaced - #42

Open
vbabenko97 wants to merge 2 commits into
mainfrom
babenko/overseer-revision
Open

vbabenko97 wants to merge 2 commits into
mainfrom
babenko/overseer-revision

Conversation

@vbabenko97

@vbabenko97 vbabenko97 commented Sep 25, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

The overseer queue holds one entry per spec hash, and a rerun of the spec replaces that entry. decide() looked the entry up by spec hash alone. So an overseer who reviewed result A could click approve after a rerun had queued result B, and B was released without anyone reviewing it.

Each queued result now gets a fresh revision token; spec_hash is unchanged. Approving or rejecting must name the revision that was reviewed, and decide() checks it under the existing RELEASE_LOCK.

Behaviour

Request names… Result
the current revision decision goes through, as before
any other revision while the spec is queued, or a revision queued for it before (a rerun replaced it, or it was decided already) stale: API 409 Conflict, CLI exits with an error; nothing is released or changed
a revision never queued for this spec, and the spec is not queued "not in queue": API 404, as before
no revision API 422; CLI argument error
  • API
    • POST /overseer/{spec_hash}/{approve|reject} now requires {"revision": ...}.
    • GET /overseer lists each item's revision.
    • Flagged RunStatus and AnalysisResponse bodies carry the revision they were queued under.
  • CLI
    • approve and reject take a required --revision.
    • list prints the revision, and show prints it on stderr so stdout stays the result JSON.
  • UI
    • A decision sends the queue item's revision.
    • On 409 the UI reloads the queue and says the queue changed since it was loaded (a newer run, or already decided), so the request decided nothing.
    • The UI re-reads the displayed run only when that run queued the decided revision.
    • api() moved to frontend/app/api-client.js, so the HTTP status it attaches is tested.
  • Run status
    • A decision updates only the run that queued the decided revision.
    • An earlier run whose result was replaced before any decision stays flagged. It no longer reports a result it never produced.
  • Release log
    • QUEUED lines and the decisions on them carry the revision.
    • Queue entries written before this change get a stable legacy-<queued> revision, so they can still be decided.

Related fixes found in review

  • A decision made before its run finished could be lost. If the decision landed after record() but before the run wrote its status, and a rerun then recorded before that run finished, the run stayed flagged. The old recovery path read the latest release-log line and the current released.json, and both belong to the rerun by then. The API now keeps such a decision in memory, keyed by revision, and applies it when the run finishes.
  • A rerun failing partway could leave an old revision pointing at new data. record() now withdraws the earlier queue entry and release before overwriting the run files. A write that fails midway therefore leaves no revision that could release data it was not issued for.
  • A decision failing partway could be taken twice. decide() now dequeues, then logs, then releases. If a step fails, the revision cannot be decided again (possibly the opposite way).
  • A release could exist without its log line. The disclosure check's differencing rule reads the release log, so record() and decide() now log before they write released.json. A failed write then leaves a logged release that is not on disk, which errs on the safe side.
  • decide() raises a dedicated NotQueued (a KeyError subclass), so the API's 404 no longer catches unrelated KeyErrors.

Verification

  • python -m pytest -q -m "not slow": 161 passed, including schema freshness. Run locally with the project venv.
  • python -m pytest -q -m slow: 4 passed (FLARE simulator). FLARE controllers call record().
  • git diff --check: clean.
  • Acceptance test: view A, rerun to produce B, approve A → refused, and a byte-for-byte snapshot of the output directory is unchanged; approve B with its revision → released. Covered at queue level (approve and reject), CLI level and API level (409; both runs' status unchanged).
  • Further Python tests: fresh revisions for identical reruns, stale after an auto-released rerun, a rerun failing midway, a decision failing midway, early decisions around a rerun (with parking disabled, 4 tests fail), legacy queue entries.
  • Review: two adversarial review rounds, one of them a bypass hunt over record() failing at each statement, reruns, approve vs reject, legacy revisions, concurrent runs of one spec and sync-endpoint results. No bypass found; the fixes above came out of these rounds.
  • Frontend (CI): npm test 22/22 and npm run build pass. New tests cover the 409 path, including one through the real api-client.js with a mocked fetch, an already-decided item, and api() status handling. A local policy hook blocks node and npm where this was written, so CI was their first run. That run caught a bug in the test's fake API: it matched queued-once revisions across specs rather than per spec. It is fixed in 4d1f550; the product code was unaffected.
  • Schemas: docs/schemas/ edited by hand for the same reason. The freshness test compares them with scripts/export_schemas.py output and passes.

Not changed

  • No authentication or authorization on the governance endpoints. It is still required before any shared deployment. A revision is a staleness guard, not a credential.
  • One process per SERVER_OUT. The check runs under an in-process lock, and a CLI decision in another process is not seen by the API's in-memory run state.

The overseer queue holds one entry per spec hash, and a rerun of the
spec replaces it. decide() looked the entry up by spec hash alone, so an
overseer who reviewed result A could approve after a rerun had queued
result B, and B was released without anyone reviewing it.

Each queued result now gets a fresh revision token (spec_hash is
unchanged). Approving or rejecting must name the revision that was
reviewed, and decide() checks it under RELEASE_LOCK. A revision that
was queued for the spec but is not the current one (a rerun replaced
it, or it was decided already) is stale: nothing is released or
changed. A revision never queued for the spec, on a spec not queued,
is still "not in queue".

- API: POST /overseer/{spec_hash}/{approve|reject} requires
  {"revision": ...}; stale is 409 Conflict, not in queue 404.
  GET /overseer lists each item's revision; flagged run and analysis
  responses carry the revision they were queued under.
- CLI: approve/reject take a required --revision; list prints it and
  show prints it on stderr (stdout stays the result JSON); a stale
  revision exits with an error.
- UI: the decision sends the queue item's revision; on 409 it reloads
  the queue and asks for another review. It re-reads the displayed run
  only when that run queued the decided revision. api() moved to
  app/api-client.js so its error status is tested.
- A decision updates only the run that queued the decided revision. An
  earlier run whose result was replaced before any decision stays
  flagged instead of reporting a result it never produced. A decision
  that lands before its run has written its status is kept in memory
  by revision and applied when the run finishes, so a rerun in between
  no longer loses it (it previously relied on the latest release-log
  line and the current released.json).
- record() withdraws the earlier run's queue entry and release before
  overwriting the run files, so a write failing midway leaves no
  revision pointing at files it was not issued for. decide() dequeues,
  then logs, then releases, so a failure midway cannot leave the
  revision open to a second decision. Both log a release before
  writing released.json, which the differencing check relies on.
  decide() raises NotQueued (a KeyError) for "not in queue".
- Release-log lines carry the revision for queued results and the
  decisions on them. Queue entries written before this change get a
  stable "legacy-<queued>" revision so they can still be decided.

Contracts and docs/schemas are updated. Tests cover the acceptance
case (view A, rerun to B, approve A: refused with nothing changed;
approve B with its revision: released) at queue, CLI and API level,
plus stale-after-auto-release, a rerun or decision failing midway,
early decisions around a rerun, legacy entries, and the UI's 409
handling. Fast suite 161, FLARE simulator suite 4.
…as_queued

The fake API treated a revision queued under any spec as queued before
under every spec, so a decision on an unqueued spec got 409 instead of
404 and "a failed decision is shown and changes nothing else" failed in
CI.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant