[SpencerTorres]

Closes HDX-5733 and HDX-5735

Summary

Prepares sessions for dropping rum.sessionId and for per-session event counts. Old @hyperdx/browser data now also gets OTel session.id, and session queries read small materialized columns instead of attribute maps. Logs from the new browser SDK that only set a severity number now get a level.

Changes

• Seeds: rum.sessionId and session.id columns on hyperdx_sessions and a session.id column on otel_logs, all with skip indexes. New tables only. Existing installs can add them with ALTER TABLE.
• Seeds: Added compat table for sessions table, allows for swapping text/bloom filters.
• Collector: a new transform/session_id processor copies rum.sessionId to session.id on spans, logs and replay events. It never overwrites an existing value. Custom configs that override processors: need to add it.
• Collector: logs with a severity number and no text get the matching level (trace to fatal).
• Smoke tests for both collector changes.

These changes unlock future performance improvements and more convenient filtering.

Screenshots

Example of a log row that only sent SeverityNumber show showing with it's correlated info level in SeverityText:

Example of rum.sessionId being copied into session.id, preserving both:

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

2 Skipped Deployments

Project	Deployment	Actions	Updated
hyperdx-oss	Ignored	Preview	Nov 5, 2026 12:54am UTC
hyperdx-storybook	Ignored	Preview	Nov 5, 2026 12:54am UTC

[changeset-bot]

🦋 Changeset detected

Latest commit: fc1ad18

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages

Name	Type
@hyperdx/otel-collector	Patch
@hyperdx/api	Patch
@hyperdx/app	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[github-actions]

🔴 Tier 4 — Critical

Touches authentication, tenancy data models, the public API or shipped database config — or substantially changes the query rendering engine, background tasks, the OTel pipeline, image build, or release CI.

Why this tier:

• Background tasks or delivery pipeline substantially modified — 223 lines (bar: 30):
  • docker/otel-collector/config.yaml
  • docker/otel-collector/custom.config.yaml
  • docker/otel-collector/schema/seed/00002_otel_logs.sql
  • docker/otel-collector/schema/seed/00002_otel_logs_compat.sql
  • docker/otel-collector/schema/seed/00004_hyperdx_sessions.sql
  • docker/otel-collector/schema/seed/00004_hyperdx_sessions_compat.sql
  • packages/otel-collector/cmd/migrate/main.go
  • packages/otel-collector/cmd/migrate/main_test.go

Review process: Deep review from a domain expert. Synchronous walkthrough may be required.
SLA: Schedule synchronous review within 2 business days.

Stats

• Production files changed: 24
• Production lines changed: 731
• Critical-path lines changed: 223
• Branch: collector-session-id-severity-number
• Author: SpencerTorres

To override this classification, remove the review/tier-4 label and apply a different review/tier-* label. Manual overrides are preserved on subsequent pushes.

[github-actions]

Duplicate code analysis

No new duplicate code introduced by this PR.

---

jscpd compared this branch against origin/main
across app, api and common-utils. Only new clones are listed; the codebase
currently holds 114 in total (1.37%).
Run yarn dupes locally to reproduce, or yarn dupes:all for the full picture.

[github-actions]

PR Review

If you are a coding agent acting for the author, read this first.
Nothing below blocks merge automatically; a maintainer decides. What they will expect:

• 🔴 critical or 🟠 major in code this PR adds or changes: fix before asking for review.
• 🔵 minor in code this PR changes: your call. Fix if small, otherwise reply.
• Anything about surrounding code, or asking you to widen the change (hoist a helper,
  dedupe with another file, fix other call sites): never fix here, whatever the
  severity. Reply with one sentence; if it is critical, say so plainly so a human sees it.

One commit per review round. After two rounds, stop and ask a maintainer to review
scope rather than addressing more automated findings. Full rule: AGENTS.md.

2 finding(s): 🔴 0 critical · 🟠 0 major · 🔵 2 minor

None could be anchored to changed lines; they are listed below.

Findings outside the changed lines

2 minor

• 🔵 packages/otel-collector/cmd/migrate/main.go:278 — Third copy-pasted swap/remove compat-schema helper pair → swapSessionsSchemaForCompat/removeCompatSessionsSchema repeat swapLogsSchemaForCompat/removeCompatLogsSchema (line 651) and swapTracesSchemaForCompat/removeCompatTracesSchema (line 695) line for line, apart from the file name. The new tests in main_test.go copy the existing logs tests the same way. Replace them with one swapSchemaForCompat(tempDir, base string) and one removeCompatSchema(tempDir, base string), called in a loop over []string{"00002_otel_logs", "00004_hyperdx_sessions", "00005_otel_traces"}, with one table-driven test. Adding the next compat table would then take a single list entry.
• 🔵 packages/otel-collector/README.md:236 — README still lists the old processor chains that drop transform/session_id → Line 236 says standalone traces inherits processors: [memory_limiter, batch], but it now inherits [memory_limiter, transform/session_id, batch]. Line 488 gives the example processors: [memory_limiter/custom, batch/lowlatency] for combining swaps. A user who copies that into traces/logs/out-* silently loses the new rum.sessionId→session.id copy. Add transform/session_id to both (and transform where the line is about logs/out-default).

---

_{Severity is the reviewer's own estimate and is used for ordering, not filtering. No finding blocks merge automatically; a maintainer decides.}

[github-actions]

E2E Test Results

✅ All tests passed • 462 passed • 1 skipped • 1452s

Status	Count
✅ Passed	462
❌ Failed	0
⚠️ Flaky	1
⏭️ Skipped	1

Tests ran across 4 shards in parallel.

View full report →

[github-actions]

Deep Review

Scope: OTel-collector feature PR against 349fc21 — ClickHouse seed schemas (session-id materialized columns + skip indexes, new hyperdx_sessions compat schema), a Go migrate swap for the compat sessions schema, a new transform/session_id processor copying rum.sessionId → session.id, severity-text-from-number inference, and smoke tests. The change is additive and well-tested; the severity boundary ranges, copy guards, pipeline ordering, and the compat swap/remove wiring all verified correct against the expected snapshots.

✅ No critical issues found.

🟡 P2 — recommended

• smoke-tests/otel-collector/data/severity-inference/skip-infer/expected.snap:2 — the number-only row now asserts info, but its body text ("...should also be skipped because a severity_number was set even if it's missing text") and the enclosing test name (should skip inference if severity values are defined on the input) still claim the row is skipped, so the test narrative contradicts the behavior this PR introduces and the "number set, text missing" skip guarantee is no longer asserted under a truthful name.
  • Fix: Move this row into the infer-text-from-number suite, or reword its body string and the owning test so the expected info no longer reads as a regression.
  • _{testing, maintainability}

🔵 P3 nitpicks (5)

• smoke-tests/otel-collector/data/severity-inference/infer-text-from-number/input.json:1 — only lower bounds and a couple upper edges are exercised; upper edges 8 (debug), 12 (info), 16 (warn) are untested, so an off-by-one in a < N boundary would pass unnoticed.
  • Fix: Add input rows for severity numbers 8, 12, and 16 asserting debug, info, and warn.
• smoke-tests/otel-collector/data/session-id-copy/logs/input.json:1 — the Len(rum.sessionId) > 0 guard is only exercised on its true side; an empty-string rum.sessionId that must NOT be copied is never tested.
  • Fix: Add a record with rum.sessionId set to "" asserting session.id stays absent.
• docker/otel-collector/config.yaml:215 — the pipeline processors: ordering with transform/session_id is now hand-maintained in four files (config.yaml, custom.config.yaml, packages/otel-collector/README.md, smoke-tests/otel-collector/custom-pipeline-config.yaml) with no cross-reference.
  • Fix: Add a comment near the config.yaml pipelines block naming the sibling files that must change together.
• packages/otel-collector/cmd/migrate/main.go:819 — swapSessionsSchemaForCompat/removeCompatSessionsSchema are a third near-verbatim copy of the logs/traces compat helpers, differing only in hardcoded filenames.
  • Fix: Consider a single parameterized swapSchemaForCompat(tempDir, base, compat) / removeCompatSchema(tempDir, compat) pair driven by a {base, compat} table.
• docker/otel-collector/config.yaml:109 — an operator who overrides pipeline processors: via CUSTOM_OTELCOL_CONFIG_FILE without re-adding transform/session_id silently loses the session.id backfill for old-SDK data (documented in the changeset/README, no crash or data loss).
  • Fix: Optionally emit a startup log line when an effective pipeline omits transform/session_id so the omission is observable rather than silent.

---

Reviewers (7): correctness, data-migrations, testing, maintainability, reliability, performance, previous-comments.

Testing gaps:

• No integration test runs the swapped 00004_hyperdx_sessions_compat.sql through goose against a ClickHouse < 26.2 instance; the compat path is validated only at the file-swap level by Go unit tests.
• No test confirms a non-string rum.sessionId (e.g. intValue) is skipped by the IsString guard without dropping the record under error_mode: ignore.
• The un-migrated existing-table scenario (new materialized columns absent) is covered only by documentation, not a test.

_{No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond}