Skip to content

test: use per-run unique document IDs to fix flaky CI - #227

Merged
ejscribner merged 1 commit into
mainfrom
fix/ci-flaky-shared-test-ids
Oct 2, 2026
Merged

ejscribner merged 1 commit into
mainfrom
fix/ci-flaky-shared-test-ids

Conversation

@dex-the-ai

@dex-the-ai dex-the-ai commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #226

Problem

Every test failure in the scheduled/PR runs over the last ~3 weeks (e.g. 37030324397, 36893321903, 36460388381, 36327142135, 36246359111, 36149129175, 36011741627, 35873227932, 35622511605, 35514083012, 35445277478, 35351875815, 35233142846, 35107414166, 36481308232, 35652873431) is the same assertion:

FAIL app/api/v1/route/[routeId]/route.test.ts > POST /api/v1/route > should respond with status code 201 Created
AssertionError: expected 409 to be 201

Root cause: the tests write documents under fixed IDs (route_post, route_put, airline_post, …) into a Capella cluster that is shared with other quickstarts.

  • ruby-on-rails-quickstart runs its scheduled CI ~2 minutes after ours each day, against the same travel-sample.inventory.route collection, using the same IDs. That run is currently failing every day, and it leaves route_post/route_put behind. Today it ran 15:57:07–15:57:44Z. A read-only lookup on the shared cluster shows route_post and route_put with CAS timestamps of 2026-10-02T15:57:39Z. The route_put payload (CDG, stops: 1, distance: 3500, flight AF199) matches that repo's spec, not ours.
  • The next day our POST → 201 test hits the leftover doc and gets 409. Our afterEach then deletes it, and Ruby recreates it a few minutes later. That's why it looks flaky but fails on most schedule runs.
  • The same fixed IDs can also collide between concurrent runs of this suite (dependabot opens several PRs at once; push and PR runs overlap).

(Separate and not addressed here: the npm error ERESOLVE failures on dependabot PRs #214/#219 are real peer-dependency conflicts, not flakiness.)

Fix

  • Add lib/test-utils.ts with uniqueTestId(prefix), which returns ${prefix}_${randomUUID()}.
  • Use it for every document the tests write (POST/PUT/DELETE fixtures) in the route, airline and airport test files. The airline and airport files have the same bug shape, so they're fixed too.
  • Drop a redundant shadowed const routeId = "route_post" in the route POST 400 test so it uses the describe-level ID.
  • Read-only fixtures (route_10209, airline_10, airport_1262, …) are unchanged.

Side benefit: our tests no longer delete documents that belong to other projects' test runs.

Risk

Test-only change. Existing afterEach cleanup still removes every per-run doc. Verified 0 leaked docs after runs (below). The only way a doc can leak is if a run is killed mid-test. Those docs have unique IDs, so they can never break later runs.

Validation evidence

All runs used a disposable local couchbase:enterprise-7.6.4 container with travel-sample, with vitest in node:22 (npx vitest run, same as CI's npm test).

Scenario Code Command Exit Result
Clean cluster main npx vitest run 0 Tests 44 passed (44)
Repro: seed route_post/route_put exactly as Ruby CI leaves them main npx vitest run 1 FAIL … POST /api/v1/route > …201 Created / AssertionError: expected 409 to be 201 (identical to CI)
Seed leftovers for all fixed IDs (route/airline/airport post/put/delete) this PR npx vitest run 0 Tests 44 passed (44)
Sabotage: revert test files, same seeded state main npx vitest run 1 Tests 3 failed | 41 passed (201 tests in route, airline, airport)
2 suites concurrently this PR 2× npx vitest run in parallel 0 / 0 44 passed each
Leak check after runs this PR SELECT COUNT(*) … WHERE REGEXP_CONTAINS(META().id, '_(post|put|delete)_<uuid>$') – route 0, airline 0, airport 0
Typecheck this PR npm run typecheck 0 clean
Lint this PR npx eslint lib/test-utils.ts <3 test files> 0 clean
Format (new file) this PR npx prettier --check lib/test-utils.ts 0 All matched files use Prettier code style!

Note: 3 concurrent runs of the old code happened to pass locally because the race window is narrow. The deterministic failure mode is the leftover foreign doc above, and that's what CI hits.

CI for this PR (head 48d8388): Run Tests passed against the shared CI cluster, Test Files 8 passed (8), Tests 44 passed (44): https://github.com/couchbase-examples/nextjs-capella-quickstart/actions/runs/37043420374/job/110958840191. CodeQL and Vercel also passed. When this ran, the Ruby suite's leftover route_post was still in the shared cluster (written 15:57:39Z), which is the state that failed today's scheduled run on main.

Follow-up suggestion (outside this repo): ruby-on-rails-quickstart should also use unique IDs and fix its failing suite. Until it does, its leftovers just sit unused in the shared cluster.

🤖 Generated with Claude Code

@vercel

vercel Bot commented Oct 2, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
nextjs-capella-quickstart Ready Ready Preview Oct 2, 2026 5:50pm UTC

The CI cluster is shared with other quickstarts. The ruby-on-rails-quickstart
scheduled CI runs ~2 minutes after ours, writes the same fixed IDs
(route_post, route_put, ...) and leaves them behind when it fails, so our next
run's POST test gets 409 instead of 201. Concurrent runs of this suite
(dependabot PR batches, push + PR) can collide the same way.

Generate the IDs for every document the tests write with a random UUID suffix
so foreign or concurrent writers can't collide with them.

Fixes #226

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ejscribner
ejscribner merged commit a4e3a40 into main Oct 2, 2026
6 checks passed
@ejscribner
ejscribner deleted the fix/ci-flaky-shared-test-ids branch October 2, 2026 18:00

This branch was successfully deployed

1 active deployment
Preview — 48d83886 Deployed Oct 2, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Fix CI Flakiness

2 participants