Repository navigation
fix(overseer): refuse decisions on a queued result a rerun replaced - #42
Open
vbabenko97 wants to merge 2 commits into
Open
vbabenko97 wants to merge 2 commits into
vbabenko97 wants to merge 2 commits into
Conversation
The overseer queue holds one entry per spec hash, and a rerun of the
spec replaces it. decide() looked the entry up by spec hash alone, so an
overseer who reviewed result A could approve after a rerun had queued
result B, and B was released without anyone reviewing it.
Each queued result now gets a fresh revision token (spec_hash is
unchanged). Approving or rejecting must name the revision that was
reviewed, and decide() checks it under RELEASE_LOCK. A revision that
was queued for the spec but is not the current one (a rerun replaced
it, or it was decided already) is stale: nothing is released or
changed. A revision never queued for the spec, on a spec not queued,
is still "not in queue".
- API: POST /overseer/{spec_hash}/{approve|reject} requires
{"revision": ...}; stale is 409 Conflict, not in queue 404.
GET /overseer lists each item's revision; flagged run and analysis
responses carry the revision they were queued under.
- CLI: approve/reject take a required --revision; list prints it and
show prints it on stderr (stdout stays the result JSON); a stale
revision exits with an error.
- UI: the decision sends the queue item's revision; on 409 it reloads
the queue and asks for another review. It re-reads the displayed run
only when that run queued the decided revision. api() moved to
app/api-client.js so its error status is tested.
- A decision updates only the run that queued the decided revision. An
earlier run whose result was replaced before any decision stays
flagged instead of reporting a result it never produced. A decision
that lands before its run has written its status is kept in memory
by revision and applied when the run finishes, so a rerun in between
no longer loses it (it previously relied on the latest release-log
line and the current released.json).
- record() withdraws the earlier run's queue entry and release before
overwriting the run files, so a write failing midway leaves no
revision pointing at files it was not issued for. decide() dequeues,
then logs, then releases, so a failure midway cannot leave the
revision open to a second decision. Both log a release before
writing released.json, which the differencing check relies on.
decide() raises NotQueued (a KeyError) for "not in queue".
- Release-log lines carry the revision for queued results and the
decisions on them. Queue entries written before this change get a
stable "legacy-<queued>" revision so they can still be decided.
Contracts and docs/schemas are updated. Tests cover the acceptance
case (view A, rerun to B, approve A: refused with nothing changed;
approve B with its revision: released) at queue, CLI and API level,
plus stale-after-auto-release, a rerun or decision failing midway,
early decisions around a rerun, legacy entries, and the UI's 409
handling. Fast suite 161, FLARE simulator suite 4.
…as_queued The fake API treated a revision queued under any spec as queued before under every spec, so a decision on an unqueued spec got 409 instead of 404 and "a failed decision is shown and changes nothing else" failed in CI.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The overseer queue holds one entry per spec hash, and a rerun of the spec replaces that entry.
decide()looked the entry up by spec hash alone. So an overseer who reviewed result A could click approve after a rerun had queued result B, and B was released without anyone reviewing it.Each queued result now gets a fresh revision token;
spec_hashis unchanged. Approving or rejecting must name the revision that was reviewed, anddecide()checks it under the existingRELEASE_LOCK.Behaviour
409 Conflict, CLI exits with an error; nothing is released or changed404, as before422; CLI argument errorPOST /overseer/{spec_hash}/{approve|reject}now requires{"revision": ...}.GET /overseerlists each item'srevision.RunStatusandAnalysisResponsebodies carry the revision they were queued under.approveandrejecttake a required--revision.listprints the revision, andshowprints it on stderr so stdout stays the result JSON.api()moved tofrontend/app/api-client.js, so the HTTP status it attaches is tested.flagged. It no longer reports a result it never produced.legacy-<queued>revision, so they can still be decided.Related fixes found in review
record()but before the run wrote its status, and a rerun then recorded before that run finished, the run stayedflagged. The old recovery path read the latest release-log line and the currentreleased.json, and both belong to the rerun by then. The API now keeps such a decision in memory, keyed by revision, and applies it when the run finishes.record()now withdraws the earlier queue entry and release before overwriting the run files. A write that fails midway therefore leaves no revision that could release data it was not issued for.decide()now dequeues, then logs, then releases. If a step fails, the revision cannot be decided again (possibly the opposite way).record()anddecide()now log before they writereleased.json. A failed write then leaves a logged release that is not on disk, which errs on the safe side.decide()raises a dedicatedNotQueued(aKeyErrorsubclass), so the API's 404 no longer catches unrelatedKeyErrors.Verification
python -m pytest -q -m "not slow": 161 passed, including schema freshness. Run locally with the project venv.python -m pytest -q -m slow: 4 passed (FLARE simulator). FLARE controllers callrecord().git diff --check: clean.409; both runs' status unchanged).record()failing at each statement, reruns, approve vs reject, legacy revisions, concurrent runs of one spec and sync-endpoint results. No bypass found; the fixes above came out of these rounds.npm test22/22 andnpm run buildpass. New tests cover the 409 path, including one through the realapi-client.jswith a mockedfetch, an already-decided item, andapi()status handling. A local policy hook blocksnodeandnpmwhere this was written, so CI was their first run. That run caught a bug in the test's fake API: it matched queued-once revisions across specs rather than per spec. It is fixed in4d1f550; the product code was unaffected.docs/schemas/edited by hand for the same reason. The freshness test compares them withscripts/export_schemas.pyoutput and passes.Not changed
SERVER_OUT. The check runs under an in-process lock, and a CLI decision in another process is not seen by the API's in-memory run state.