Skip to content

feat(mxc): warn when the ETW consumer receives zero provider events - #3499

Open
pkhodade-NV wants to merge 1 commit into
windowsfrom
feat/mxc-etw-zero-events-watchdog
Open

pkhodade-NV wants to merge 1 commit into
windowsfrom
feat/mxc-etw-zero-events-watchdog

Conversation

@pkhodade-NV

Copy link
Copy Markdown
Collaborator

Summary

  • The ETW->OCSF audit pipeline is meant to decode OS "Sandboxing" ETW events during a real sandbox lifecycle into a durable OCSF JSONL audit log. In a real-world repro, EnableTraceEx2/StartTraceW both reported success, the consumer thread ran correctly, but zero OS-sourced events were ever decoded across two full, successful sandbox lifecycles -- with no error, warning, or diagnostic anywhere pointing at why.
  • Root cause (already traced): EnableTraceEx2 success only proves the request to enable the provider succeeded, not that the provider exists on this host/build or will ever actually fire. A provider-identity mismatch (or a non-firing provider) left the audit trail silently empty. This PR is specifically about detection and surfacing, not diagnosing the underlying GUID/provider question, which is an environment/OS-build question outside what static code changes can resolve.

Related Issue

No linked issue -- this is a localized observability fix (a missing diagnostic for an existing, real failure mode) to the MXC ETW consumer.

Changes

  • etw_consumer.rs: CaptureHealth gains an events_matched: AtomicU64 counter, incremented on the consumer thread for every raw event that made it through the callback's provider-GUID filter, regardless of whether TDH decode later succeeds. Exposed via a new EtwSession::events_received() accessor, alongside the existing is_capture_alive().
  • AttributionIndex gains a total_launches counter (never decremented by forget), so the watchdog can tell "real sandbox activity has happened" apart from "the index is currently non-empty" (which forget clears on every normal sandbox completion).
  • Added a zero-events watchdog on the consumer thread's existing 200ms poll loop: once total_launches() > 0 and a generous grace period (30s) elapses with events_matched still at zero, log a tracing::warn! with an actionable message and emit a Detection Finding [2004] (severity: High, is_alert: true) into the OCSF stream itself, so the gap is visible in the audit log the same way the driver's own internal events already are.
  • Gated on actual sandbox activity (not just session uptime), so a gateway configured with etw_audit=true that simply hasn't created any sandboxes yet never warns -- only "activity happened, nothing arrived" does.
  • The watchdog's decision (should_warn_zero_events) is extracted into a small pure function, so it's directly unit-testable without a real ETW session or elevation.

Testing

  • New tests: total_launches_survives_forget, zero_events_watchdog_stays_quiet_without_sandbox_activity, zero_events_watchdog_stays_quiet_once_any_event_matched, zero_events_watchdog_waits_out_the_grace_period, zero_events_watchdog_fires_once_grace_elapses.
  • Full crate suite passes: cargo test -p openshell-driver-mxc --target x86_64-pc-windows-msvc --lib.

Checklist

  • Tests added for the new behavior
  • No unrelated changes bundled in

Originally opened as GitLab MR !117 against our internal mirror; re-opened here against windows for upstream review. Rebased onto windows's independently-evolved etw_consumer.rs (queue-overload reporting, PID-reuse rewrite, and byte-tracking additions that landed there since this MR was authored) -- both sets of changes are preserved side by side.

EnableTraceEx2 succeeding only proves the request to enable the
Sandboxing provider succeeded, not that the provider exists on this
host/build or will ever fire. A provider-identity mismatch or a
non-firing provider left the ETW->OCSF audit trail silently empty
across real, successful sandbox lifecycles, with no error, warning,
or diagnostic anywhere.

Track raw provider-matched events received per session and add a
zero-events watchdog on the consumer thread: once real sandbox
activity has happened (register_launch called at least once) and a
grace period elapses with zero events matched, log a warning and
emit a Detection Finding [2004] naming the gap. Gated on actual
activity (not just session uptime) so an idle gateway with
etw_audit=true and no sandboxes created never warns.

Also exposes EtwSession::events_received() alongside the existing
is_capture_alive(), so a status/diagnostics surface can query capture
health directly, not just infer it from tracing output.

The watchdog's decision logic is extracted into a pure function
(should_warn_zero_events) so it's unit-testable without a real ETW
session or elevation.

Signed-off-by: Prashant Khodade <pkhodade@nvidia.com>
(cherry picked from commit 4fcfa716a808eaa91c6476e9c69366f91702b1fa)
@copy-pr-bot

copy-pr-bot Bot commented Sep 20, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant