feat(supervisor): export OTLP traces from sandbox supervisors

[krishicks]

Summary

Sandbox supervisors now export OTLP traces to the gateway's collector, so sandbox startup, supervisor-to-gateway calls, and egress handling show up in the same trace as the gateway request that created the sandbox.

Related Issue

Partially addresses #2508

#2508 proposed relaying supervisor spans through the gateway because the supervisor ran inside the sandbox and direct export would have needed an egress policy exception. Since RFC 0012 the supervisor runs outside the workload's egress policy, so it exports directly to the collector.

Changes

• Kubernetes, Docker, Podman, and VM drivers pass the gateway's OTLP endpoint to supervisors as OPENSHELL_OTLP_ENDPOINT, along with TRACEPARENT from the launching operation. Driver TOML cannot set the endpoint.
• The supervisor exports spans as openshell-supervisor with the openshell.sandbox.id resource attribute, and flushes them before exiting.
• supervisor.startup joins the sandbox creation trace and covers image policy discovery, policy load, and boundary attach, confirm, agent start, and access start.
• Supervisor gRPC calls to the gateway carry W3C trace context, so gateway server spans nest under the supervisor's client spans.
• Each egress connection emits supervisor.egress.connect with authorize, resolve, and dial children. These use DEBUG level because every outbound connection starts its own trace.
• The supervisor's OTLP filter follows the sandbox log level for OpenShell targets (never below INFO) and keeps other libraries at INFO.
• Docs: docs/how-it-works/gateways/configuration.mdx notes that the collector must be reachable from sandboxes and how to include DEBUG spans.

Testing

• mise run pre-commit passes
• Unit tests added/updated
• E2E tests added/updated (if applicable)

New tests cover the OTLP span filter directives, egress span levels and parent/child structure, and trace context propagation. Traces were checked manually against a local collector with the Docker driver. The two startup spans at INFO (boundary.confirm, access.start) have no level test; that would need running supervisor startup under a span-capturing subscriber.

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture docs updated (if applicable)

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-3977.docs.buildwithfern.com/openshell

[purp]

Sol found three things:

• The shipped local collector configuration won’t work for some supervisors. Local gateway tasks configure http://127.0.0.1:4317. Before this PR, that address served the host gateway’s exporter. The PR copies it unchanged into Kubernetes Pods and Podman Machine supervisors, where loopback identifies the Pod or VM. Operators would see gateway traces but missing supervisor traces and export failures. The local workflows need a supervisor-reachable collector address. This follows from the configuration and network topology; I haven’t reproduced it live. See [Kubernetes endpoint forwarding](

  OpenShell/crates/openshell-driver-kubernetes/src/driver.rs

  Line 707 in 53a60f5

  fn supervisor_tracing_environment(&self) -> Vec<(&'static str, String)> {

  ) and [the local Podman configuration](

  OpenShell/tasks/scripts/gateway-podman.sh

  Line 115 in 53a60f5

  append_local_otlp_config_if_available() {

  ).
• The positive propagation test doesn’t prove explicit parenting works. It creates the child while the parent is already entered. I temporarily removed the explicit set_parent call in the isolated worktree, and the test still passed. Create the child outside the parent’s scope and assert both trace ID and parent span ID. That strengthens the existing test without adding much code. See [the test](

  OpenShell/crates/openshell-otel/src/propagation.rs

  Line 186 in 53a60f5

  let (parent_span_id, environment) = tracing::subscriber::with_default(subscriber, || {

  ).
• [NIT]: Keep the umbrella issue open. [#2508](feat(observability): OpenTelemetry span emission from the sandbox supervisor #2508) also covers L7 timing, middleware propagation, OCSF correlation, and sampling decisions. This is a useful first slice; I’d change “Closes” to “Partially addresses” and leave the remaining work separately scoped.

Sol's suggestion for the first:

Could we add an optional supervisor_endpoint, defaulting to endpoint? Local gateways use loopback, which identifies the Pod or VM from Kubernetes and Podman Machine supervisors.

Suggested changes:

1. In openshell-server/src/config_file.rs, extend OtlpConfig:

#[serde(default)]
pub supervisor_endpoint: Option<String>,

impl OtlpConfig {
    pub fn supervisor_endpoint(&self) -> &str {
        self.supervisor_endpoint
            .as_deref()
            .unwrap_or(&self.endpoint)
    }
}

2. In the Kubernetes, Docker, and Podman factories in openshell-gateway/src/lib.rs:

config.supervisor_otlp_endpoint = context
    .otlp_config()
    .map(|otlp| otlp.supervisor_endpoint().to_owned());

3. Add this argument to the four driver binaries:

#[arg(long)]
supervisor_otlp_endpoint: Option<String>,

Use it when constructing each driver config:

supervisor_otlp_endpoint: args
    .supervisor_otlp_endpoint
    .clone()
    .or_else(|| args.otlp_endpoint.clone()),

In openshell-gateway/src/vm.rs, forward it alongside the existing exporter argument:

command
    .arg("--supervisor-otlp-endpoint")
    .arg(config.supervisor_endpoint());

4. Have the local Kubernetes task generate:

[openshell.gateway.otlp]
endpoint = "http://127.0.0.1:4317"
supervisor_endpoint = "http://openshell-collector.observability.svc.cluster.local:4317"

For Podman Machine, use http://host.containers.internal:4317 and verify the forwarded listener accepts that route.

Update existing OtlpConfig literals, docs, and Helm rendering; test fallback/override behavior and actual supervisor span delivery in both local workflows.

[purp]

That seems reasonable to me, but I'm only one-code-read's worth of familiar with this, so you might have a more efficient choice in mind. I'd address the first two.

[purp]

gator-agent

Blocked

GitHub reports merge conflicts for this PR, so Gator cannot advance it to code review yet.

Thanks @purp. I checked the current head after your comments: no new commit has been pushed since you asked for a supervisor-reachable collector endpoint and a propagation test that proves explicit parenting. Those requests remain with the author.

Next action: @krishicks, resolve the merge conflicts and address those two maintainer requests in an updated commit. Gator will review the new head.

Gator metadata

• Head SHA: 53a60f5627c5942a41742b10c9a2f522fcb4477c
• Gator payload: 9
• Next state: gator:blocked
• Blocked reason: merge_conflict

[krishicks]

@purp Updated:

1. Collector address. The supported dev setups work. Docker supervisors reach a port-forwarded collector on 127.0.0.1:4317 like the gateway does, since Docker Desktop's host networking bridges loopback to the Mac. Helm/Skaffold deploys (values-skaffold.yaml) point the gateway and supervisor pods at openshell-collector.observability.svc.cluster.local:4317. Podman on macOS was broken because the supervisor's loopback is the Podman Machine VM's. The Podman driver now points a loopback endpoint at host.containers.internal in Machine mode, the same alias it uses for the gateway endpoint. I checked this with mise run gateway:podman on macOS: supervisor spans arrive nested under the gateway's CreateSandbox trace.

   The remaining gap is a host-run gateway (mise run gateway) with the Kubernetes driver, where pods get the gateway's loopback address. I'd rather not add a supervisor_endpoint setting in this PR. feat(observability): supervisor OTLP telemetry relay #3196 will add a supervisor export lane for agent traces with its own endpoint, and I'd like to design the supervisor endpoint configuration once for both lanes. Until then, use the Skaffold deploy for Kubernetes tracing.

2. Propagation test. Fixed. The child is now created outside the parent's scope, and the test asserts both the trace ID and the parent span ID. I also added a test that parents a span from TRACEPARENT via set_parent_from_environment. Both fail with set_parent removed.

[purp]

gator-agent

PR Review Status

Thanks @krishicks. I checked your update: the Podman Machine address conversion and the explicit-parent propagation tests address those two parts of @purp's feedback. Your local Podman trace check is helpful. One supported local Kubernetes path still receives an unreachable supervisor collector endpoint; I left the remaining finding on the changed line. The author can fix that path or ask a maintainer to explicitly accept deferring it to #3196. No other blocking findings arose in the full patch review.

Next state: gator:in-review. The author and maintainer should resolve the Kubernetes tracing scope before Gator dispatches the required runtime tests.

Gator metadata

• Head SHA: f82246ee3bb0f0ee568df57f2846115ca1d0b187
• Base SHA: be7af99ff2eacba807d82b69adb0c3d4eaa32158
• Merge base SHA: be7af99ff2eacba807d82b69adb0c3d4eaa32158
• Patch ID: 459f587dc45ef7dccebdfe356cf4da7ed34dd2aa
• Gator payload: 9
• Next state: gator:in-review

[github-actions]

Label test:e2e applied for f82246e. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

[purp]

Deferral makes sense.

LGTM 🚢

[purp]

gator-agent

Monitoring Complete

Monitoring is complete because this PR has merged.

Final status: Gator's current-head review was preserved, and the PR merged after maintainer approval. I removed the active gator:* label because there is nothing left for Gator to monitor on this PR.