Repository navigation
chore(dev): run local dev, integration and e2e on DocumentDB - #3375
jordan-simonovski wants to merge 5 commits into
Conversation
…of MongoDB Local development and CI now use open-source DocumentDB (Postgres plus a Mongo wire-protocol gateway, MIT) in place of MongoDB 5.0 (SSPL, EOL). The API still talks to it through Mongoose, so this is the first step towards dropping MongoDB from what we ship, tested daily by every developer and CI run before users see it. HDX-5709. - One image, built from source in docker/documentdb and shared by the dev, integration and e2e stacks; it handles root-owned volumes, interrupted first boots and clean shutdown, and pins the upstream commit. - e2e database helpers use the Node driver instead of mongosh, which the image doesn't ship; CI builds the image with the GHA cache and waits on its healthcheck. - Test teardown retries DocumentDB's intermittent dropDatabase failure. - Dev data moves to .volumes/docdb_dev_<slot>; old MongoDB volumes are not migrated.
|
The latest updates on your projects. Learn more about Vercel for GitHub. 2 Skipped Deployments
|
🟡 Tier 3 — StandardIntroduces new logic, modifies core functionality, or touches areas with non-trivial risk. Why this tier:
Additional context: touches background tasks or the delivery pipeline lightly (10 lines, under the 30-line bar for Tier 4) Review process: Full human review — logic, architecture, edge cases. Stats
|
Duplicate code analysisNo new duplicate code introduced by this PR. jscpd compared this branch against |
PR Review
2 finding(s): 🔴 0 critical · 🟠 0 major · 🔵 2 minor None could be anchored to changed lines; they are listed below. Findings outside the changed lines2 minor
Severity is the reviewer's own estimate and is used for ordering, not filtering. No finding blocks merge automatically; a maintainer decides. |
E2E Test Results✅ All tests passed • 460 passed • 1 skipped • 1519s
Tests ran across 4 shards in parallel. |
 [Medium impact] The PR appears safe to merge; no new actionable issues were found. SummaryMoves development, integration tests, and E2E tests to a source-built DocumentDB image. Shipped deployment images remain unchanged.
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Deep Review✅ No critical issues found. This is a dev/test/CI-only migration from MongoDB 5.0 to a source-built DocumentDB image; the API, shipped images, and the root user-facing 🟡 P2 — recommended
🔵 P3 nitpicks (6)
Reviewers (6): correctness, testing, maintainability, security, kieran-typescript, previous-comments. Testing gaps:
No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond |
…s start errors CI e2e setup failed on every shard: `buildx bake` on the e2e compose file resolves its out-of-tree build context as a filesystem entitlement and refuses to run. Build docker/documentdb directly with the same GHA cache. A failed Postgres start now prints the tail of postgresql.log to the container logs, so `docker compose logs db` shows the cause.
|
The integration job's `up --wait` failed before any test ran: the OTel collector's healthcheck needs an OpAMP server the job doesn't run, so it never reports healthy. Start every service as before, then wait on db.
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| ); | ||
| } | ||
| return `mongodb://localhost:${port}/hyperdx`; | ||
| return `mongodb://hyperdx:hyperdx@localhost:${port}/hyperdx?tls=true&tlsAllowInvalidCertificates=true&authMechanism=SCRAM-SHA-256`; |
There was a problem hiding this comment.
🔵 minor — DocumentDB credentials and TLS options are now hand-copied into six places
The hyperdx:hyperdx user and the tls=true&tlsAllowInvalidCertificates=true&authMechanism=SCRAM-SHA-256 options are hardcoded here, in .env.development, .env.test, .env.e2e, .github/actions/e2e-setup/action.yml and e2eMongoUri() in packages/app/tests/e2e/utils/db-helpers.ts. They must all match DOCDB_USER/DOCDB_PASSWORD in the three compose files, so changing the user or the TLS mode in compose leaves yarn seed:alerts and the e2e helpers failing to authenticate. Here, build the URI by substituting the slot port into process.env.MONGO_URI (already loaded from .env.development), or move the auth/TLS suffix into one exported constant.
Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond
There was a problem hiding this comment.
Leaving this as is. The .env.* files cannot import a shared fragment and each environment uses its own port and database, so a single constant would only cover the two TypeScript copies. The values are fixed local-dev credentials that change together with the compose files.
| services: | ||
| db: | ||
| image: mongo:5.0.32-focal | ||
| image: hyperdx/documentdb:local |
There was a problem hiding this comment.
🔵 minor — The e2e README still documents MongoDB, its ports and its log commands after the e2e stack switched to DocumentDB
packages/app/tests/e2e/README.md (lines 17-37, 98-124, 397-465) still describes a MongoDB service on 29998/27017 and how to troubleshoot it. The PR updated AGENTS.md and agent_docs but not this file. Update it to describe DocumentDB on HDX_E2E_MONGO_PORT (container port 10260) and the authenticated TLS connection string, and remove the troubleshooting steps that rely on mongosh running inside the container.
Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond
0006 records DocumentDB 1.0 replacing FerretDB, following ClickHouse/ClickStack#3375. 0001 keeps its text and gains a status line pointing at it, and the index marks it superseded.
Integration shards were rebuilding DocumentDB from source and timing out. The bake shared one GHA cache scope between the collector and db images, and a scope holds a single manifest, so each export overwrote the other. db now uses the documentdb scope the e2e job already fills.
Summary
Local development, integration tests and e2e tests now run against open-source DocumentDB (Postgres extensions plus a Mongo wire-protocol gateway, MIT) instead of MongoDB 5.0 (SSPL, end of life). The API is unchanged and still uses Mongoose. Running DocumentDB everywhere internally first surfaces compatibility problems before anything we ship to users changes.
What changed
yarn dev,make dev-int,make dev-e2eand the CI integration and e2e jobs all use one DocumentDB image built fromdocker/documentdb. It is 176 MB, against about 700 MB formongo:5.0.32-focal.docker exec mongosh, because the image doesn't shipmongosh.dropDatabasefailure (codes 112 and 1, while index builds are running).Key decisions
HDX_*_MONGO_PORT) and thedbservice name are unchanged, to limit churn across scripts, slots and the dev portal.Impact
.volumes/docdb_dev_<slot>. Existing local MongoDB data isn't migrated, so the dev stack starts empty after this lands.yarn devbuilds the image, which takes a few minutes. It is cached after that.mongosh, use the connection string inagent_docs/development.md. Everything works except Compass's Performance tab.docker-compose.ymlare unchanged.Implementation detail
tar, serialmake, and a one-line patch to an upstream symbol check that miscounts musl's_init/_fini.lost+founddocker stopmake dev-intpasses 2273/2273.mainwith MongoDB: 380 passed, 9 failed, 28 flaky.dropDatabaserace: fails in 3 to 5% of drops under a stress test, the same on PG17, PG18 and the Alpine build. HyperDX never drops databases outside tests.How to reproduce
Starting point: local
yarn dev(needs the API and the metadata database).On
main:yarn dev.docker psshows thedbcontainer runningmongo:5.0.32-focal.On this branch:
yarn dev. The first run buildshyperdx/documentdb:local.docker psshows thedbcontainer runninghyperdx/documentdb:localashealthy, and the startup banner lists DocumentDB.make dev-int. All suites pass.How to test on Vercel preview
N/A, non-UI change.
References