OTel Map Server

OpenTelemetry traces → ClickHouse storage → Service Map API. This project ingests OTLP (HTTP and gRPC) into ClickHouse via the OpenTelemetry Collector, and exposes an API to build a service map and basic SLO-style stats from collected spans.

Architecture

• NGINX reverse proxy routes traffic:
  • /api/* → Go API server (server:8000)
  • /v1/* → OTLP HTTP to collector (otelcollector:4318)
• OTLP gRPC goes direct to the collector (otelcollector:4317, published on the host)
• OpenTelemetry Collector writes to ClickHouse tables
• Go API reads from ClickHouse and returns a service-map DTO

flowchart TD
    %% Nodes
    A[🌐 Internet] --> B[NGINX]

    %% Tracing Flow
    B -->|OTLP /v1/tracing| C[OpenTelemetry Collector]
    C -->|Insert traces| D[(ClickHouse)]

    %% Application Flow
    B -->|Create token / Get service map| E[Server]
    E -->|Store / Query data| D

    %% Endpoints
    D --> F[📊 Persisted Data]

Loading

Prerequisites

• Docker and Docker Compose

Environment variables (.env.production)

Copy .env.production.example to .env.production in the repo root and adjust:

PORT=8000
SERVICE_NAME=otel-map-server
LOG_LEVEL=info
SHUTDOWN_TIMEOUT_SECONDS=10
CLICKHOUSE_DSN=clickhouse://default:default@clickhouse:9000/default?dial_timeout=5s&compress=true
OTLP_HTTP_URL=http://localhost/v1/traces
OTLP_GRPC_URL=http://localhost:4317
CORS_ALLOWED_ORIGINS=*

Run with Docker Compose

docker compose pull
docker compose up -d

Services:

• clickhouse:9000 (native), 8123 (HTTP)
• otelcollector:4317 (gRPC), 4318 (HTTP), 13133 (health)
• server:8000 (internal API)
• nginx:80 (reverse proxy; 443 is published for future TLS termination)

After startup, the endpoints will be available at:

• http://localhost/api/v1/* (API endpoints)
• http://localhost/v1/traces (OTLP HTTP ingest)
• http://localhost:4317 (OTLP gRPC ingest - direct to collector)

API Endpoints

• GET /api/v1/healthz → health check
• GET /api/v1/readyz → readiness check
• POST /api/v1/session-token → returns a session token and example ingest config
• GET /api/v1/session-events?token=<uuid> → SSE endpoint for listening to trace events
• GET /api/v1/service-map/:session-token → get service map

Service Map Response

• Direct, Jaeger-style service dependencies are returned as deduplicated parent→child edges.
• Per-service fields:
  • service_name, total_requests, error_count, error_rate, latency_p50_ms, latency_p90_ms, latency_p95_ms.
• Per-edge fields:
  • source_service_name, target_service_name, target_service_path, total_requests, requests_per_second.
• Time window:
  • Metrics cover the whole session (all spans carrying the session token).
  • Edge requests_per_second is computed over the session's observed time window.

Example

{
  "services": [
    {
      "service_name": "frontend",
      "total_requests": 1234,
      "error_count": 25,
      "error_rate": 0.02,
      "latency_p50_ms": 12.3,
      "latency_p90_ms": 45.6,
      "latency_p95_ms": 78.9
    },
    {
      "service_name": "backend",
      "total_requests": 980,
      "error_count": 10,
      "error_rate": 0.01,
      "latency_p50_ms": 8.1,
      "latency_p90_ms": 30.2,
      "latency_p95_ms": 55.4
    }
  ],
  "edges": [
    {
      "source_service_name": "frontend",
      "target_service_name": "backend",
      "target_service_path": "GET /api/orders",
      "total_requests": 980,
      "requests_per_second": 8.5
    }
  ]
}

Conceptually aligned with Jaeger’s service dependency graph. See: Jaeger repository.

Tracing Ingest

First, create a session token:

curl -X POST http://localhost/api/v1/session-token

The response contains:

• token: your unique session token (UUID)
• ingest.otlp_http_url: from OTLP_HTTP_URL (default http://localhost/v1/traces)
• ingest.otlp_grpc_url: from OTLP_GRPC_URL (default http://localhost:4317)
• ingest.resource_attribute: { key: "otelmap.session_token", value: <token> }

Important: Ensure your tracer sets the resource attribute otelmap.session_token with your session token value so spans are associated with your session.

Context Propagation

• NGINX forwards traceparent, tracestate, and baggage headers
• Echo middleware extracts W3C headers into the request context
• Global propagator is set to TraceContext + Baggage
• All spans created in handlers and MapManager use the incoming context for proper trace continuity

Troubleshooting

• Ingest: curl http://localhost:4318/v1/traces -I should return 405/404 (collector present). Through nginx use the otlp.otelmap.com (or otlp.localhost) host, since the default vhost serves the API.
• Collector health: curl http://localhost:13133/healthz
• ClickHouse connectivity: docker exec -it <clickhouse-container> clickhouse-client --query "SELECT 1"
• Service map empty: ensure spans include the otelmap.session_token resource attribute matching your session token
• Check collector logs: docker logs otel
• Check server logs: docker logs <server-container-name>

Development

Standard Go module project. Key packages:

• cmd/server: main entrypoint
• internal/http: Echo routing and middleware
• internal/handlers: handlers for health, session token, session events, and service map
• internal/mapz: builds the map DTO from ClickHouse rows

Useful targets: make build, make run, make test, make vet, make fmt.