feat(ocsf): add trace_id/span_id correlation fields to OCSF event builders

[rhuss]

User Story

As a security or compliance reviewer investigating a policy violation, I want each OCSF event to carry the ID of the trace that produced it, so that I can jump from the event in my log aggregator straight to the supervisor's trace instead of matching sandbox IDs and timestamps by hand.

Problem Statement

Open Cybersecurity Schema Framework (OCSF) security events and OpenTelemetry (OTel) traces exist in separate systems with no connection between them. A security reviewer filters the OCSF log aggregator (Loki, Splunk) for DENY events in the last hour and finds a network deny for api.suspicious.com from sandbox sb-abc123. To see how the supervisor handled that connection (which policy matched, what L7 enforcement and middleware ran), they search Jaeger for the sandbox ID and hope the timestamps line up. Nothing links the security event to its trace.

Today only the gateway emits OTel spans. Once #3977 lands, the supervisor does too, and most OCSF events and OTel spans then come from the same code paths. The OCSF builders have no way to record the active trace context.

Impact / Why This Matters

Every deny investigation needs manual correlation across two systems. The workaround is searching the trace backend by sandbox ID within a time window. That breaks down quickly: with #3977 every egress connection starts its own trace, so a busy sandbox produces many traces per second and the time window rarely identifies one. SIEM pipelines also have no join key to automate the correlation.

Proposed Design

Record the active trace context in OCSF events using the OCSF 1.8 Trace profile. When a sampled OTel span is active at emission time, the event carries trace.uid and lists trace in metadata.profiles. Without one, the trace object is omitted. The reviewer pastes trace.uid into Jaeger or Tempo and lands on the supervisor trace that produced the event.

Representation

OCSF 1.8 has no top-level trace_id or span_id attributes. It defines a Trace profile whose trace object carries the W3C trace ID in trace.uid, plus an optional span object:

{"class_uid": 4001, "activity_id": 1, "metadata": {"profiles": ["security_control", "network_proxy", "container", "host", "trace"], ...}, "trace": {"uid": "0af7651916cd43dd8448eb211c80319c"}, ...}

The first version populates trace.uid only. The 1.8 span object requires start_time and end_time, which are unknown while the span is still open at emission time. Whether and how to fill trace.span is settled during implementation against the vendored-schema validation tests. The Trace profile schema files get vendored next to the existing ai_operation profile.

Layering

openshell-ocsf stays independent of OpenTelemetry:

1. openshell-ocsf adds a plain-data trace correlation type and a builder setter that renders the trace object and adds the profile.
2. openshell-otel provides a function that extracts the active OTel context and returns it only when the span context is valid and sampled. An unsampled trace ID points to nothing in the trace backend.
3. Binaries that emit OCSF events register this extractor with openshell-ocsf at startup. ocsf_emit! fills in the trace context when the builder has not set one, so individual call sites need no OTel dependency. An explicit setter call overrides the automatic value.

Egress deny spans

The supervisor egress spans from #3977 are DEBUG, and the supervisor's OTLP filter resolves to openshell=info at the default warn log level. Without a change, network denies get no trace ID at default settings. The proposal is an INFO-level span for deny decisions only. Denies are rare and are what reviewers investigate, while allowed connections stay at DEBUG to keep span volume down.

The decision is known only after authorization returns, and every deny OCSF event is emitted after the authorization scope has ended. So the deny span is opened in the deny branch and entered around the OCSF emission at every deny site: L4 policy denials, mapping and SSRF denials, and L7 HTTP denials in the relay and middleware. It carries the decision attributes (policy, reason, destination). A span that wraps only the policy evaluation would leave the emission hook without a trace context.

At the default log level the DEBUG connect span is disabled, so the deny span has no exported parent and becomes a one-span root trace. trace.uid still resolves, but the full connect, authorize, resolve and dial tree appears only at debug level, where the deny span nests under the connect span.

Out of scope: agent trace context

Each egress connection starts its own root trace, so the linked trace shows the supervisor's handling of the connection, not the agent's activity. Linking a deny to the agent's own trace, through the workload's traceparent header relayed via #3196, is a follow-up. The workload controls that header, so it goes in a separate, clearly labelled field and never in the trace object.

Scope

• crates/openshell-ocsf/: trace correlation type, builder setter, trace serialization, metadata.profiles entry, vendored Trace profile schema, emission hook, and downgrade stripping of the trace object and Trace profile for OCSF 1.3 and 1.1 (the profile first appears in 1.4)
• crates/openshell-otel/: sampled trace context extraction
• crates/openshell-supervisor-network/: INFO-level deny span, entered around the OCSF emission at every deny site
• Supervisor and gateway binaries: register the extractor at startup
• No breaking changes (the trace object is optional)

Dependencies

• feat(supervisor): export OTLP traces from sandbox supervisors #3977 (supervisor OTLP span export) must land first.

Acceptance Criteria

• OCSF events emitted inside a sampled OTel span include trace.uid (32 lowercase hex characters) and list trace in metadata.profiles.
• OCSF events emitted without an active sampled span omit the trace object and do not list the profile.
• At the default supervisor log level, OCSF network and HTTP deny events carry a trace.uid that resolves to an exported trace containing the deny span.
• Events downgraded to OCSF 1.3 or 1.1 contain neither the trace object nor trace in metadata.profiles, covered by regression tests next to the existing downgrade tests.
• openshell-ocsf has no OpenTelemetry dependency.
• The Trace profile schema is vendored, and the schema validation tests cover events with and without trace.
• The published observability docs describe the trace field and how to follow it to the trace backend.

Alternatives Considered

Top-level trace_id / span_id fields: Easy to query, but outside the OCSF 1.8 schema. Consumers that validate against OCSF treat them as unknown attributes. The standard Trace profile covers the same need.

The unmapped mechanism: Schema-valid, but unmapped is meant for source data that has no OCSF attribute. Trace context has one, and downstream tooling won't look for it in unmapped.

Per-site .trace_context() calls: Explicit, but every emitting crate would need an OTel dependency, and adoption would drift across the call sites. The emission hook covers every event, and the explicit setter stays available where the current span is the wrong source.

Enrichment in the JSONL layer: Works, but puts the trace context into one output format instead of the event itself, so the shorthand format and any future sink would miss it.

Raising all egress spans to INFO: Gives every OCSF network event a trace ID, but exports one trace per outbound connection at default settings. Scoping the INFO span to denies covers the investigation case at a fraction of the volume.

Agent Investigation

• OCSF builders live in crates/openshell-ocsf/src/builders/. Each builder declares its profiles through ctx.metadata(&[...]), and api_activity.rs already adds ai_operation conditionally.
• The vendored schemas cover OCSF 1.8.0 but include only the ai_operation profile. The Trace profile is available from the OCSF schema server.
• OCSF events are emitted via ocsf_emit!(), which stores the event in a thread-local and emits via tracing::info!(). The shorthand and JSONL layers extract it from there.
• openshell-otel owns OTel context handling via tracing_opentelemetry::OpenTelemetrySpanExt. The supervisor crates have no OTel dependency on main.
• ocsf_emit! call sites exist in openshell-sandbox, openshell-server, openshell-supervisor, openshell-supervisor-network, and openshell-supervisor-process.
• feat(supervisor): export OTLP traces from sandbox supervisors #3977 (closes feat(observability): OpenTelemetry span emission from the sandbox supervisor #2508) adds supervisor span export. Its egress spans (supervisor.egress.connect, authorize, resolve, dial) are DEBUG, and the OTLP filter is info,openshell=<level> with a floor of INFO.

Related: #1055 (Enterprise Observability), #2508 (Supervisor OTel span emission), #3977 (supervisor OTLP span export), #3196 (OTLP relay), #2507 (Gateway OTel export surface)

[kvnloo]

I checked this against the current crate boundary and the vendored OCSF schema.

One wrinkle: the OCSF 1.8 base schema does not define top-level trace_id / span_id fields for these events. It does provide the schema-bounded unmapped escape hatch.

There is also a dependency-boundary question:

• openshell-ocsf currently owns OCSF event construction and depends on tracing;
• openshell-otel already owns OpenTelemetry context extraction through tracing_opentelemetry::OpenTelemetrySpanExt.

I would prefer not to make the OCSF crate itself responsible for discovering the active OpenTelemetry context.

A narrow design could be:

1. let the OCSF builders accept optional trace/span correlation values;
2. represent them through the schema-supported extension/unmapped path;
3. have the OTEL layer extract the active context and pass those values into the event builder.

That keeps OCSF serialization independent of the OpenTelemetry implementation.

If top-level trace_id / span_id fields are intended as an OpenShell-specific extension instead, I think that should be explicit in the contract because it is outside the vendored base schema.

Which representation do you prefer?

[rhuss]

Thanks for checking this against the schema, @kvnloo. You're right on both counts. The dependency point also exposes a bug in my original proposal: tracing::Span::current() only gives you process-local tracing IDs, not W3C trace and span IDs.

Representation. I'd go with neither top-level fields nor unmapped. OCSF 1.8 defines a Trace profile for this case, with a trace object that holds the W3C trace ID in trace.uid. We only vendor the ai_operation profile today, so it's easy to miss. An event would look like this:

{
  "metadata": { "profiles": ["security_control", "network_proxy", "container", "host", "trace"] },
  "trace": { "uid": "0af7651916cd43dd8448eb211c80319c" }
}

One catch: the 1.8 span object requires start_time and end_time, and we emit the OCSF event while the span is still open. So I'd ship trace.uid first, since that's what a security reviewer pastes into Tempo or Jaeger anyway, and settle trace.span in the PR against the vendored-schema validation tests.

Layering. Agreed, openshell-ocsf shouldn't know about OpenTelemetry. What I have in mind:

1. openshell-ocsf gets a plain-data correlation type and a builder setter that renders the trace object and adds the profile.
2. openshell-otel extracts the active context and returns it only when the span context is valid and sampled. An unsampled trace ID sends the reviewer to an empty search result, which is worse than no ID.
3. Binaries that emit OCSF events register that extractor with openshell-ocsf at startup, and ocsf_emit! fills in the context when the builder hasn't set one. The emitting crates don't each need an OTel dependency, and an explicit setter still wins where the current span is the wrong source.

Prerequisite. I also got something wrong in the issue: on main, the supervisor crates have no OpenTelemetry instrumentation yet. #3977 adds it, so this should build on that PR. Even with #3977 there are two gaps for the network-deny case:

• The egress spans are DEBUG, and the supervisor's OTLP filter resolves to openshell=info at the default warn log level, so at default settings a deny would still carry no exported trace ID. I'd lean toward an INFO span around deny decisions only. Denies are rare and they're what a security reviewer investigates, while allowed connections stay at DEBUG to keep the span volume down.
• Each egress connection starts its own root trace. That trace shows how the supervisor handled the connection, not what the agent was doing. The agent's side lives in the workload's own traceparent header (relayed via feat(observability): supervisor OTLP telemetry relay #3196). The workload controls that header, so it needs a separate, clearly labelled field and must not end up in the trace object. I'd handle that as a follow-up.

I'll update the issue body to reflect this. Are you planning to pick up the implementation? If so, building on top of #3977 makes sense.

[HarryMoss]

I checked this against main at 5601d71b0 now that #3977 has landed, and ran a couple of focused tests around the OCSF/OTel path you described.

The correlation gap is still present as expected. With an active sampled OpenTelemetry span, current_trace_context_carrier() gives me a valid W3C traceparent, but an OCSF NetworkActivity emitted in that context still serializes without a trace object / trace.uid.

I also noticed two implementation wrinkles / edge cases that may be worth accounting for when this is implemented.

1. OCSF downgrade path

OpenShell emits OCSF 1.8 internally but supports downgrading JSON output to 1.3 and 1.1. The OCSF Trace profile is not present in those older schemas, while the current downgrade code only strips the newer fields/profiles it already knows about (ai_model, container, ai_operation, etc.).

I tried a 1.8 event containing:

{
  "metadata": {
    "profiles": ["security_control", "trace"]
  },
  "trace": {
    "uid": "0af7651916cd43dd8448eb211c80319c"
  }
}

and downgraded it to 1.3. The trace object survived the downgrade.

So it looks like adding the Trace profile here will also require the downgrade path to strip both the trace object and "trace" from metadata.profiles for 1.3/1.1, with a regression test alongside the existing downgrade tests.

2. Span lifetime around automatic enrichment

In the current egress path the connect span is made current while authorization runs, but some deny OCSF events are emitted afterwards.

I checked the same lifecycle in a focused test:

inside connect span = Some(traceparent ...)
at later OCSF emission = None

So if ocsf_emit! obtains correlation only from the current active context, the new INFO level deny span needs to remain current through the actual OCSF emission.

Wrapping only the policy evaluation would still leave the emission hook with no trace context. The other option would be to capture the correlation while the span is active and carry it explicitly to the event.

A narrow way to implement and test this seems to be:

1. add the plain data Trace correlation and sampled context extractor as proposed;
2. make the INFO deny span cover the OCSF emission point, rather than just the authorization call;
3. add downgrade coverage so trace and the Trace profile are removed for OCSF 1.3/1.1;
4. keep an acceptance test showing that a deny event at the default log level contains a trace.uid that resolves to the exported supervisor trace.

Does that match the intended implementation boundary for the deny span and the older schema downgrade path?

[rhuss]

Thanks @HarryMoss, both points check out and they sharpen the plan.

Downgrade. Confirmed: the Trace profile and the trace object first appear in OCSF 1.4, and ocsf_schema_version only accepts 1.1 and 1.3 as downgrade targets. So the existing <= 1.3 branch in format/downgrade.rs is the right place: strip the trace object and the trace profile there, with a regression test next to the existing ones.

Deny span lifetime. Also confirmed, on all the L4 egress paths: authorization runs inside in_scope(...) and the deny events are emitted after it returns. There's a second reason the span can't wrap only authorization: the decision isn't known until authorization finishes, so a span that exists only for denies has to be opened in the deny branch anyway. The plan is a small helper that every deny site enters around its OCSF emission, carrying the decision attributes (policy, reason, destination). That includes the L7 deny sites in the relay and middleware. The automatic hook then picks up the context without call-site changes, and the explicit setter stays available for emissions outside such a span.

One consequence worth stating: at the default log level the DEBUG connect span is disabled, so the deny span has no exported parent and becomes a one-span trace of its own. trace.uid still resolves and the span carries the decision attributes, but the full connect, authorize, resolve and dial tree only shows up at debug level. I think that's the right trade-off for span volume. The acceptance test should check exactly that: at the default level, a deny event's trace.uid resolves to an exported trace containing the deny span.

So yes, your four steps match the boundary I have in mind. I'll fold both points into the issue body.

On implementation: I have nothing in progress on this. If you'd like to take it on, @HarryMoss (or @kvnloo, since you looked at it first), go ahead. I'm happy to collaborate, answer design questions as they come up, and review the PR. If neither of you wants to pick it up, just let me know and I'll start on it myself.

[HarryMoss]

Thanks @rhuss, I’m happy to take it on and I’ll start looking at it tonight and tomorrow.

@kvnloo, if you’ve already started implementation please let me know, I’m happy to leave it with you rather than duplicate.