Repository navigation
test: use per-run unique document IDs to fix flaky CI - #227
Merged
Merged
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
The CI cluster is shared with other quickstarts. The ruby-on-rails-quickstart scheduled CI runs ~2 minutes after ours, writes the same fixed IDs (route_post, route_put, ...) and leaves them behind when it fails, so our next run's POST test gets 409 instead of 201. Concurrent runs of this suite (dependabot PR batches, push + PR) can collide the same way. Generate the IDs for every document the tests write with a random UUID suffix so foreign or concurrent writers can't collide with them. Fixes #226 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dex-the-ai
force-pushed
the
fix/ci-flaky-shared-test-ids
branch
from
October 2, 2026 17:50
b16eb89 to
48d8388
Compare
Closed
ejscribner
approved these changes
Oct 2, 2026
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #226
Problem
Every test failure in the scheduled/PR runs over the last ~3 weeks (e.g. 37030324397, 36893321903, 36460388381, 36327142135, 36246359111, 36149129175, 36011741627, 35873227932, 35622511605, 35514083012, 35445277478, 35351875815, 35233142846, 35107414166, 36481308232, 35652873431) is the same assertion:
Root cause: the tests write documents under fixed IDs (
route_post,route_put,airline_post, …) into a Capella cluster that is shared with other quickstarts.ruby-on-rails-quickstartruns its scheduled CI ~2 minutes after ours each day, against the sametravel-sample.inventory.routecollection, using the same IDs. That run is currently failing every day, and it leavesroute_post/route_putbehind. Today it ran 15:57:07–15:57:44Z. A read-only lookup on the shared cluster showsroute_postandroute_putwith CAS timestamps of2026-10-02T15:57:39Z. Theroute_putpayload (CDG,stops: 1,distance: 3500, flightAF199) matches that repo's spec, not ours.POST → 201test hits the leftover doc and gets 409. OurafterEachthen deletes it, and Ruby recreates it a few minutes later. That's why it looks flaky but fails on most schedule runs.(Separate and not addressed here: the
npm error ERESOLVEfailures on dependabot PRs #214/#219 are real peer-dependency conflicts, not flakiness.)Fix
lib/test-utils.tswithuniqueTestId(prefix), which returns${prefix}_${randomUUID()}.const routeId = "route_post"in the route POST 400 test so it uses the describe-level ID.route_10209,airline_10,airport_1262, …) are unchanged.Side benefit: our tests no longer delete documents that belong to other projects' test runs.
Risk
Test-only change. Existing
afterEachcleanup still removes every per-run doc. Verified 0 leaked docs after runs (below). The only way a doc can leak is if a run is killed mid-test. Those docs have unique IDs, so they can never break later runs.Validation evidence
All runs used a disposable local
couchbase:enterprise-7.6.4container withtravel-sample, with vitest innode:22(npx vitest run, same as CI'snpm test).mainnpx vitest runTests 44 passed (44)route_post/route_putexactly as Ruby CI leaves themmainnpx vitest runFAIL … POST /api/v1/route > …201 Created/AssertionError: expected 409 to be 201(identical to CI)npx vitest runTests 44 passed (44)mainnpx vitest runTests 3 failed | 41 passed(201 tests in route, airline, airport)npx vitest runin parallel44 passedeachSELECT COUNT(*) … WHERE REGEXP_CONTAINS(META().id, '_(post|put|delete)_<uuid>$')npm run typechecknpx eslint lib/test-utils.ts <3 test files>npx prettier --check lib/test-utils.tsAll matched files use Prettier code style!Note: 3 concurrent runs of the old code happened to pass locally because the race window is narrow. The deterministic failure mode is the leftover foreign doc above, and that's what CI hits.
CI for this PR (head
48d8388): Run Tests passed against the shared CI cluster,Test Files 8 passed (8),Tests 44 passed (44): https://github.com/couchbase-examples/nextjs-capella-quickstart/actions/runs/37043420374/job/110958840191. CodeQL and Vercel also passed. When this ran, the Ruby suite's leftoverroute_postwas still in the shared cluster (written 15:57:39Z), which is the state that failed today's scheduled run onmain.Follow-up suggestion (outside this repo):
ruby-on-rails-quickstartshould also use unique IDs and fix its failing suite. Until it does, its leftovers just sit unused in the shared cluster.🤖 Generated with Claude Code