feat: self-instrument with the ClickStack browser and Node SDKs

.changeset/clickstack-sdks.md

@@ -0,0 +1,45 @@
+---
+'@hyperdx/api': minor
+'@hyperdx/app': minor
+---
+
+feat: self-instrument with `@clickhouse/clickstack-node` and
+`@clickhouse/clickstack-browser` instead of `@hyperdx/node-opentelemetry` and
+`@hyperdx/browser`
+
+HyperDX's own telemetry changes shape. HTTP spans use the stable semantic
+convention keys (`http.request.method`, `http.response.status_code`,
+`url.full`). pino records come from the `@opentelemetry/instrumentation-pino`
+scope with real severity numbers instead of the `node-logger` scope with
+severity number 0. Console capture covers every `console` call in the API, and
+browser console output is recorded as log records instead of spans. Uncaught
+exceptions are recorded as FATAL log records, and API errors as an exception
+event on the Express span instead of a separate error span.
+
+Browser telemetry records the user as `user.id`, `user.email` and `user.name`
+instead of `userEmail` and `userName`, keeps the team name as `teamName`, and no
+longer records a `user navigated` action. Browser fetch and XHR spans no longer
+record request and response headers or bodies. Browser sessions restart once
+after the upgrade. The per-signal browser endpoints and
+`NEXT_PUBLIC_OTEL_RESOURCE_ATTRIBUTES` work as before. Browser traces and logs
+are now sent gzip-compressed every 5 seconds. If the browser sends to a
+collector other than ClickStack's, its CORS `allowed_headers` must be `"*"` or
+include `content-type` and `content-encoding` (and `authorization`), and a proxy
+in front of it must forward `Content-Encoding` unchanged.
+
+The Next.js server now sends its own telemetry in production builds too, not
+only under `next dev`, when `HYPERDX_API_KEY` or `OTEL_EXPORTER_OTLP_HEADERS` is
+set. `HDX_EXPORTER_ENABLED=false` turns it off along with browser telemetry. It
+reports under its own service name, the configured one plus `-next`
+(`hdx-oss-app-next` in the docker-compose `app` service and when no name is
+set), so its spans stay apart from the API's and the browser's. The HyperDX
+images now default `OTEL_SERVICE_NAME` to `hdx-oss-app`. Each `/api` request
+adds Next.js server spans for the proxy hop. Requests for build and `public/`
+assets, such as `/_next/static/*`, `/__ENV.js` and `/pyodide/*`, are not traced.
+
+`HYPERDX_API_KEY` still enables self-instrumentation. Without
+`OTEL_EXPORTER_OTLP_ENDPOINT`, server-side self-telemetry now goes to
+`http://localhost:4318` instead of `https://in-otel.hyperdx.io`, so set the
+endpoint explicitly. `HDX_NODE_*` variables are no longer read. For request
+headers and bodies set `OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_*` and
+`CLICKSTACK_CAPTURE_HTTP_BODY`.

.changeset/collector-cors-max-age.md

@@ -0,0 +1,10 @@
+---
+'@hyperdx/api': patch
+'@hyperdx/otel-collector': patch
+---
+
+fix: let browsers cache the collector's CORS preflights for 10 minutes
+
+The OTLP receiver now answers preflights with `Access-Control-Max-Age: 600`.
+Without it, browsers cached a preflight for only 5 seconds, which is how often
+the browser SDK exports, so about half of its requests were preflights.

CONTRIBUTING.md

@@ -214,6 +214,11 @@ yarn dev:down
 
 To enable self-instrumentation and demo logs, you can set the `HYPERDX_API_KEY`
 to your ingestion key (visit the Team settings page after creating your account).
+The API and the Next.js server pass it to the ClickStack SDK
+(`@clickhouse/clickstack-node`) as `apiKey`, and the browser SDK
+(`@clickhouse/clickstack-browser`) gets it from `/api/config`. The API and the
+alert task report as `hdx-oss-dev-api`, the Next.js server as
+`hdx-oss-dev-app-next`, and the browser as `hdx-oss-dev-app`.
 
 To do this, create a `.env.local` file in the root of the project and add the
 following:

LOCAL.md

@@ -38,7 +38,7 @@ Configuring instrumentation for ClickStack Local is similar to configuring it fo
 
 Most instrumentations can be configured using the `OTEL_EXPORTER_OTLP_ENDPOINT` environment variable. Ex: `OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318`.
 
-If you're using a ClickStack maintained SDK, you may need to give a non-empty `HYPERDX_API_KEY` as well, API keys are not validated in ClickStack Local and therefore can be any non-empty value.
+If you're using a ClickStack SDK, pass the key as the `apiKey` init option. `@clickhouse/clickstack-node` also accepts `OTEL_EXPORTER_OTLP_HEADERS=authorization=<key>`, and `@clickhouse/clickstack-browser` only takes `apiKey`. Older HyperDX SDKs (`@hyperdx/*`) may need a non-empty `HYPERDX_API_KEY` instead. API keys are not validated in ClickStack Local and therefore can be any non-empty value.
 
 ## Customizing Ports
 

agent_docs/observability.md

@@ -47,7 +47,7 @@ All manual instrumentation in `packages/api` goes through
 | Helper                                                    | Purpose                                                                                                      |
 | --------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ |
 | `withSpan(name, fn, opts?)`                               | Run an async unit of work in an active span. Records exceptions, sets OK/ERROR status, always ends the span. |
-| `setBusinessContext({ teamId, userId, email, ...extra })` | Attach standardized incident-remediation context to the trace + active span.                                 |
+| `setBusinessContext({ teamId, userId, email, ...extra })` | Attach standardized incident-remediation context to the active span and the request's HTTP server span.      |
 | `getStaticFeatureFlags()`                                 | Returns the static (env/compile-time) feature-flag states as `feature_flag.*` attributes.                    |
 | `getCounter(name, opts?)` / `getHistogram(name, opts?)`   | Memoized OTel instrument accessors. Always use these instead of `meter.create*`.                             |
 | `recordDuration(histogram, fn, attrs?)`                   | Run `fn`, recording its wall-clock duration on a histogram (even on throw).                                  |
@@ -57,10 +57,10 @@ types, so a single import is usually enough.
 
 ### Tracing
 
-Rely on the ClickStack SDK auto-instrumentation (HTTP, Express, Mongo, etc.) for
-the common path. Add a **manual span only for a meaningful unit of work** that
-auto-instrumentation can't see - a background job step, a fan-out, an expensive
-computation, an external tool invocation.
+Rely on the ClickStack SDK (`@clickhouse/clickstack-node`) auto-instrumentation
+(HTTP, Express, Mongo, etc.) for the common path. Add a **manual span only for a
+meaningful unit of work** that auto-instrumentation can't see - a background job
+step, a fan-out, an expensive computation, an external tool invocation.
 
 ```ts
 import { withSpan } from '@/utils/instrumentation';
@@ -91,8 +91,10 @@ Span rules to follow (from the Instrumentation Score spec):
 ### Business context
 
 Attach team/user context as early as possible (at the auth boundary or the top
-of a job). `setBusinessContext` writes both to the whole trace (via the ClickStack
-SDK, requires `HDX_NODE_BETA_MODE=1`) and to the active span.
+of a job). `setBusinessContext` writes to the active span and, inside an HTTP
+request, to the request's HTTP server span, so the context stays on the request
+even when the active span is a short Express middleware span. Other spans in the
+trace do not get it.
 
 ```ts
 import {

agent_docs/tech_stack.md

@@ -19,7 +19,7 @@
 - **Database**: ClickHouse (telemetry data), MongoDB via Mongoose (metadata)
 - **Authentication**: Passport.js with local strategy
 - **Validation**: Zod schemas
-- **Telemetry**: Self-instrumented with `@hyperdx/node-opentelemetry`
+- **Telemetry**: Self-instrumented with `@clickhouse/clickstack-node` (API and Next.js server) and `@clickhouse/clickstack-browser` (frontend)
 
 ## Common Utilities (`packages/common-utils`)
 

docker/hyperdx/Dockerfile

@@ -86,6 +86,8 @@ COPY --from=app ./pages ./packages/app/pages
 COPY --from=app ./public ./packages/app/public
 COPY --from=app ./styles ./packages/app/styles
 COPY --from=app ./types ./packages/app/types
+# next build finds instrumentation.ts only next to pages/, not under src/
+COPY --from=app ./instrumentation.ts ./instrumentation-client.ts ./packages/app/
 # Build inputs for public/whats-new.json, generated by next.config.mjs for the
 # in-app "What's new" section: the root changelog it parses and the parser
 # itself. WORKDIR is /app, and next.config resolves the changelog as
@@ -116,6 +118,10 @@ ARG CODE_VERSION
 
 ENV CODE_VERSION=$CODE_VERSION
 ENV OTEL_RESOURCE_ATTRIBUTES="service.version=$CODE_VERSION"
+# Service name for HyperDX's own telemetry, sent only when HYPERDX_API_KEY or
+# OTEL_EXPORTER_OTLP_HEADERS is set. The API and alert task use it as is, the
+# Next.js server adds "-next". Override it with -e OTEL_SERVICE_NAME=<name>.
+ENV OTEL_SERVICE_NAME=hdx-oss-app
 ENV NODE_ENV=production
 
 # Remove package managers from production image (not needed at runtime)
@@ -155,6 +161,10 @@ ARG USER_GID=10001
 
 ENV CODE_VERSION=$CODE_VERSION
 ENV OTEL_RESOURCE_ATTRIBUTES="service.version=$CODE_VERSION"
+# Service name for HyperDX's own telemetry, sent only when HYPERDX_API_KEY or
+# OTEL_EXPORTER_OTLP_HEADERS is set. The API and alert task use it as is, the
+# Next.js server adds "-next". Override it with -e OTEL_SERVICE_NAME=<name>.
+ENV OTEL_SERVICE_NAME=hdx-oss-app
 # Copy from otel collector bases
 COPY --from=ocb-builder --chmod=755 /build/output/otelcol-hyperdx /otelcontribcol
 COPY --from=otel_collector_opampsupervisor_base --chmod=755 /usr/local/bin/opampsupervisor /usr/local/bin/opampsupervisor

docker/otel-collector/config.standalone.yaml

@@ -11,6 +11,9 @@ receivers:
         cors:
           allowed_origins: ['*']
           allowed_headers: ['*']
+          # Lets browsers cache preflights for 10 minutes, see
+          # buildOtelCollectorConfig()
+          max_age: 600
         include_metadata: true
         endpoint: '0.0.0.0:4318'
 

packages/api/.env.development

@@ -3,9 +3,10 @@ HYPERDX_LOG_LEVEL=debug
 # Development only: keep sessions valid across nodemon restarts.
 EXPRESS_SESSION_SECRET="local-development-only"
 FRONTEND_URL="http://localhost:${HYPERDX_APP_PORT}"
-HDX_NODE_ADVANCED_NETWORK_CAPTURE=1
-HDX_NODE_BETA_MODE=1
-HDX_NODE_CONSOLE_CAPTURE=1
+# Self-telemetry request and response capture. Booleans must be `true`.
+CLICKSTACK_CAPTURE_HTTP_BODY=true
+OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_REQUEST="user-agent,content-type,x-request-id,referer"
+OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_RESPONSE="content-type,content-length"
 MONGO_URI="mongodb://localhost:${HDX_DEV_MONGO_PORT}/hyperdx"
 NODE_ENV=development
 OTEL_SERVICE_NAME="hdx-oss-dev-api"

packages/api/bin/hyperdx

@@ -4,19 +4,18 @@ set -e
 SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
 API_BUILD_DIR="${SCRIPT_DIR}/../build"
 
-# shellcheck disable=SC2086
-# NODE_REQUIRE_FLAGS is intentionally unquoted for word-splitting into -r <module>
-NODE_REQUIRE_FLAGS="-r @hyperdx/node-opentelemetry/build/src/tracing"
+# Self-telemetry preload, see src/telemetry.ts
+TELEMETRY_PRELOAD="${API_BUILD_DIR}/telemetry.js"
 
 command="$1"
 shift 2>/dev/null || true
 
 case "$command" in
   api)
-    exec node $NODE_REQUIRE_FLAGS "$API_BUILD_DIR/index.js" "$@"
+    exec node -r "$TELEMETRY_PRELOAD" "$API_BUILD_DIR/index.js" "$@"
     ;;
   task)
-    exec node $NODE_REQUIRE_FLAGS "$API_BUILD_DIR/tasks/index.js" "$@"
+    exec node -r "$TELEMETRY_PRELOAD" "$API_BUILD_DIR/tasks/index.js" "$@"
     ;;
   *)
     echo "Usage: hyperdx <command> [args...]"

packages/api/package.json

@@ -20,15 +20,18 @@
     "@ai-sdk/openai": "^4.0.78",
     "@ai-sdk/otel": "^1.0.116",
     "@braintree/sanitize-url": "^7.1.1",
+    "@clickhouse/clickstack-node": "^1.0.2",
     "@clickhouse/client-common": "1.23.0-head.fae5998.1",
     "@esm2cjs/p-queue": "^7.3.0",
     "@hyperdx/common-utils": "^0.33.0",
-    "@hyperdx/node-opentelemetry": "^0.9.0",
     "@hyperdx/passport-local-mongoose": "^9.0.1",
     "@modelcontextprotocol/sdk": "^1.27.1",
-    "@opentelemetry/api": "^1.8.0",
-    "@opentelemetry/host-metrics": "^0.35.5",
-    "@opentelemetry/sdk-metrics": "^1.30.1",
+    "@opentelemetry/api": "~1.9.1",
+    "@opentelemetry/api-logs": "^0.223.0",
+    "@opentelemetry/core": "^2.12.0",
+    "@opentelemetry/otlp-transformer": "^0.223.0",
+    "@opentelemetry/resources": "^2.12.0",
+    "@opentelemetry/sdk-logs": "^0.223.0",
     "@slack/webhook": "^7.0.0",
     "@types/node": "^22.15.18",
     "ai": "^7.0.116",
@@ -98,9 +101,9 @@
   },
   "scripts": {
     "start": "node ./build/index.js",
-    "dev": "DOTENV_CONFIG_PATH=.env.development nodemon --exec 'ts-node' --transpile-only -r tsconfig-paths/register -r dotenv-expand/config -r '@hyperdx/node-opentelemetry/build/src/tracing' ./src/index.ts",
+    "dev": "DOTENV_CONFIG_PATH=.env.development nodemon --exec 'ts-node' --transpile-only -r tsconfig-paths/register -r dotenv-expand/config -r ./src/telemetry.ts ./src/index.ts",
     "dev:mcp": "npx @modelcontextprotocol/inspector",
-    "dev-task": "DOTENV_CONFIG_PATH=.env.development nodemon --exec 'ts-node' --transpile-only -r tsconfig-paths/register -r dotenv-expand/config -r '@hyperdx/node-opentelemetry/build/src/tracing' ./src/tasks/index.ts",
+    "dev-task": "DOTENV_CONFIG_PATH=.env.development nodemon --exec 'ts-node' --transpile-only -r tsconfig-paths/register -r dotenv-expand/config -r ./src/telemetry.ts ./src/tasks/index.ts",
     "build": "rimraf ./build && tsc -p tsconfig.build.json && tsc-alias -p tsconfig.build.json && cp -r ./src/opamp/proto ./build/opamp/",
     "build:vercel": "rimraf ./build && tsc -p tsconfig.vercel.json && tsc-alias -p tsconfig.vercel.json && cp -r ./src/opamp/proto ./build/opamp/",
     "lint": "npx eslint . --ext .ts --max-warnings 302",

packages/api/src/__tests__/telemetry.test.ts

@@ -0,0 +1,82 @@
+const mockInit = jest.fn();
+let mockIsMainThread = true;
+
+jest.mock('@clickhouse/clickstack-node', () => ({
+  init: (...args: unknown[]) => mockInit(...args),
+}));
+jest.mock('node:worker_threads', () => ({
+  get isMainThread() {
+    return mockIsMainThread;
+  },
+}));
+
+describe('telemetry preload', () => {
+  const originalEnv = { ...process.env };
+
+  const runPreload = (env: Record<string, string>) => {
+    delete process.env.HYPERDX_API_KEY;
+    delete process.env.OTEL_EXPORTER_OTLP_HEADERS;
+    delete process.env.NODE_ENV;
+    Object.assign(process.env, env);
+    jest.isolateModules(() => {
+      jest.requireActual('@/telemetry');
+    });
+  };
+
+  beforeEach(() => {
+    mockInit.mockClear();
+    mockIsMainThread = true;
+  });
+
+  afterAll(() => {
+    process.env = originalEnv;
+  });
+
+  it('does not start the SDK without a key or OTLP headers', () => {
+    runPreload({ HYPERDX_API_KEY: ' ', OTEL_EXPORTER_OTLP_HEADERS: '' });
+
+    expect(mockInit).not.toHaveBeenCalled();
+  });
+
+  it('passes HYPERDX_API_KEY as the apiKey', () => {
+    runPreload({ HYPERDX_API_KEY: 'ingest-key', NODE_ENV: 'production' });
+
+    expect(mockInit).toHaveBeenCalledWith({
+      apiKey: 'ingest-key',
+      instrumentations: undefined,
+    });
+  });
+
+  it('appends the key to OTLP headers that have no authorization entry', () => {
+    runPreload({
+      HYPERDX_API_KEY: 'ingest,key',
+      OTEL_EXPORTER_OTLP_HEADERS: 'x-scope-orgid=acme',
+    });
+
+    expect(process.env.OTEL_EXPORTER_OTLP_HEADERS).toBe(
+      'x-scope-orgid=acme,authorization=ingest%2Ckey',
+    );
+    expect(mockInit).toHaveBeenCalledWith(
+      expect.objectContaining({ apiKey: undefined }),
+    );
+  });
+
+  it('enables host metrics only in development', () => {
+    runPreload({ HYPERDX_API_KEY: 'ingest-key', NODE_ENV: 'development' });
+
+    expect(mockInit).toHaveBeenCalledWith({
+      apiKey: 'ingest-key',
+      instrumentations: {
+        '@opentelemetry/instrumentation-host-metrics': { enabled: true },
+      },
+    });
+  });
+
+  it('does not start a second SDK in a worker thread', () => {
+    mockIsMainThread = false;
+
+    runPreload({ HYPERDX_API_KEY: 'ingest-key' });
+
+    expect(mockInit).not.toHaveBeenCalled();
+  });
+});

packages/api/src/index.ts

@@ -1,32 +1,26 @@
-import { getHyperDXMetricReader } from '@hyperdx/node-opentelemetry/build/src/metrics';
-import { HostMetrics } from '@opentelemetry/host-metrics';
-import { MeterProvider, MetricReader } from '@opentelemetry/sdk-metrics';
+import { flush } from '@clickhouse/clickstack-node';
 import { serializeError } from 'serialize-error';
 
 import * as config from '@/config';
 import Server from '@/server';
 import { isOperationalError } from '@/utils/errors';
 import logger from '@/utils/logger';
 
-if (config.IS_DEV) {
-  // Start collecting host metrics
-  const meterProvider = new MeterProvider({
-    // FIXME: missing selectCardinalityLimit property
-    readers: [getHyperDXMetricReader() as unknown as MetricReader],
-  });
-  const hostMetrics = new HostMetrics({ meterProvider });
-  hostMetrics.start();
-}
-
 const server = new Server();
 
+// process.exit() skips the SDK's exit-time flush, so export what is buffered
+// first. Before init (no key set) flush() resolves at once.
+const exitAfterFlush = (code: number) => {
+  void flush().finally(() => process.exit(code));
+};
+
 process.on('uncaughtException', (err: Error) => {
   logger.error({ err: serializeError(err) }, 'Uncaught exception');
 
   // FIXME: disable server restart until
   // we make sure all expected exceptions are handled properly
   if (config.IS_DEV && !isOperationalError(err)) {
-    process.exit(1);
+    exitAfterFlush(1);
   }
 });
 
@@ -41,5 +35,5 @@ server.start().catch(e => {
   // orchestrator restarts us instead of lingering as a zombie that listens
   // but can never serve (https://github.com/ClickHouse/ClickStack/issues/2966).
   logger.error({ err: serializeError(e) }, 'Server start failed, exiting');
-  process.exit(1);
+  exitAfterFlush(1);
 });

packages/api/src/mcp/__tests__/tracing.test.ts

@@ -58,9 +58,10 @@ jest.mock('@opentelemetry/api', () => ({
   },
 }));
 
-jest.mock('@hyperdx/node-opentelemetry', () => ({
-  __esModule: true,
-  setTraceAttributes: jest.fn(),
+// The real package would read helpers off the mocked @opentelemetry/api at
+// load time. Only setBusinessContext uses it, and these tests never call it.
+jest.mock('@opentelemetry/core', () => ({
+  getRPCMetadata: jest.fn(),
 }));
 
 jest.mock('@/config', () => ({