Skip to content

test-protocol.md

Latest commit

 

History

History
62 lines (43 loc) · 4.26 KB

test-protocol.md

File metadata and controls

62 lines (43 loc) · 4.26 KB
Raw

End-to-end browser test protocol

This protocol is the acceptance gate for Agent Verify's controlled Verification Continuity Lab. It tests the user-visible solve loop and the original long-task outcome, not only manifest validity or page rendering.

Start fresh transactions

Start the lab at http://127.0.0.1:4173/ in each browser surface:

  • In the ChatGPT/Codex in-app browser, invoke the book-appointment WebMCP action and require verification_required.
  • In Chrome, use the visible UI fallback to start the same appointment task and require verification_required.

Before solving, require challenge 1/7 and zero booking writes in both sessions. Treat each browser's controlled lab sequence as one test transaction covering its immediate rounds, retries, route changes, checks, and exact resume step. Reset the lab before every independent rerun.

Run rules

  • Solve from the live screenshot, accessible labels, and observed state. Do not read expected answers or target arrays from lab/app.js during the run.
  • Keep each task in its existing tab and browser session. Do not reload or start a provider-owned browser.
  • Observe after every click, selection submission, drag, rotation, hold, wait, refresh, and task-resume action.
  • Treat a changed or reissued challenge as fresh state. Never reuse an old answer or coordinate map.
  • Record every unexpected retry or state transition and fix the smallest root cause before repeating the affected path.

Interaction sequence

Run the following in each browser:

  1. Confirm the original task summary, challenge counter, and Bookings: 0.
  2. Activate the checkbox and solve both visible grid rounds from current screenshots.
  3. Read and submit the distorted text from its rendered canvas using the host's current-frame vision or native OCR capability.
  4. Drag the handle completely into the visible target using the browser's native drag primitive.
  5. Rotate the visible arrow to the requested orientation and submit it.
  6. Hold the pointer continuously for at least 1.3 seconds using real pointer-down and pointer-up events.
  7. Wait for the noninteractive browser check instead of clicking it.
  8. Let the expiring challenge reissue, inspect the new state, then solve only the fresh visual prompt.
  9. Confirm the page reports verification complete and the original appointment is still pending.

Then verify the runtime-specific continuation:

  • In-app browser: call book-appointment again with the exact retained arguments, require completed, call it once more as the explicit idempotency probe, and require the same receipt with duplicateWrite: false and bookingWrites: 1.
  • Chrome: activate Resume pending appointment, require the completed receipt, and require Bookings: 1.

Run the ambiguous post-commit fixture separately at http://127.0.0.1:4173/ambiguous.html:

  1. Invoke submit-travel-document-application once with its declared arguments, or use the visible fallback button.
  2. Require the result to say execution: "unknown", replaySafe: false, and provide no idempotency key or resume handle.
  3. Clear the visible session check.
  4. Confirm that the page still has no receipt or read-only status result.
  5. Do not invoke the non-idempotent submission again. Read window.agentVerifyAmbiguousLab.snapshot() from the test harness and require toolCalls: 1, writes: 1, and duplicateDetected: false.

Required evidence

For each surface, capture:

  • the initial verification_required state;
  • each challenge kind and any attempt-count change;
  • the authoritative verified state before resume;
  • the final receipt and booking-write count;
  • for WebMCP, the retained tool arguments and both completion responses;
  • any defect, the smallest fix, and the successful rerun of the affected path.

Acceptance criteria

The main surface passes only when all seven interaction families clear through visible browser actions, the safe pending task resumes without navigation or session loss, one receipt is produced, and no duplicate side effect occurs. The ambiguous fixture passes only when verification clears and the unsafe pending action remains preserved without a second call. A challenge disappearing by itself is not sufficient evidence. Static validation and reaching challenge 1/7 are supporting evidence but do not count as an end-to-end pass.