feat: otel duration gap additions

[sputti-czi]

Add three OTel metric instruments: recv-direction edge latency, collective duration, inter-collective gap

Summary

This PR extends CoMMA's OTel metrics path with three new Base2-exponential-histogram instruments that fold, at the source, the per-operation signals needed for communication-graph straggler analysis — so they arrive as bounded-cardinality metric aggregates rather than per-event traces:

Instrument	Unit	Keying	What it measures
nccl.net_recv.latency	ns	{nccl.communicator.hash, nccl.source.rank, nccl.destination.rank, nccl.hostname}	recv-direction proxy-step latency per directed edge. Datapoints are re-keyed so source = remote sender, destination = local rank — a recv edge a→b describes the same logical direction as the sender's net_send.latency a→b, giving both-endpoint observations of every edge.
nccl.collective.duration	ns	{nccl.comm.hash, nccl.collective.name, nccl.size.class, nccl.rank, nccl.hostname}	wall time of each collective op per rank, bucketed by size class. Cross-rank comparison of these distributions separates ranks that wait in collectives from ranks that arrive late.
nccl.collective.gap	ns	{nccl.hostname, nccl.pid}	per-process idle intervals — the complement of NCCL engagement. An activity window is an op enqueue, proxy op, or kernel channel; gaps are interval-union-correct under overlap by construction. High gap + low duration on the same rank is the "everyone waits on me" signature.

Together with the existing nccl.net_send.latency and nccl.collective.seq_num, these make source-vs-victim attribution possible from metrics alone: send+recv edge histograms give a walkable directed graph (including the internal ring/tree traffic of collectives, which the span path does not expose as edges), and duration/gap give the compute-side corroboration.

Also fixed (pre-existing bugs found while validating)

1. num_active over-count in HistogramManager::update_priority — the active-set counter drifted upward across grouping intervals, prematurely exhausting the high-fidelity cardinality budget.
2. u32 dur_ns wraparound in step_tracker — step durations above ~4.29 s wrapped; they now saturate, so long/stalled steps record as the ceiling instead of a small bogus value.

Design notes

• Fold at source. Each instrument records into an in-memory exponential histogram (a few atomic ops on the caller's thread); the periodic reader exports aggregates. No per-event I/O, no span volume.
• GapTracker uses a single packed AtomicU64 (low 32 bits: in-flight activity windows; high 32: total begins) plus an idle-since timestamp with fetch_max stamping — lock-free, two atomics per activity begin/end, and a concurrent 0→1 observer can never read a stale idle timestamp from a previous idle period.
• Gating / defaults unchanged. Everything is behind the existing NCCL_PROFILER_OTEL_ENABLE (default false); recv-side recording additionally respects the existing track_recv_steps and p2p_recv_sample_rate knobs. With defaults, this PR changes nothing at runtime.
• Cardinality. The new edge instrument participates in the same HistogramManager Top-K / otel_metrics_max_cardinality governance as net_send.latency; collective.duration is bounded by ranks × size classes × collective names; collective.gap is one series per process. Overflow behavior (aggregated fallback keyed by hostname) is unchanged.
• Copybara markers. New code and the added bench target follow the existing copybara:strip(otel) conventions.

Validation (real hardware)

Two 8-node × 8×H100 (64-GPU) 3-hour runs (FSDP2 workload, ring/SIMPLE), identical except the profiler configuration:

• Throughput: no degradation. Steady-state tokens/s/device delta vs baseline within noise: +0.07% (metrics-only) / −0.11% (metrics + tracing) over 3 h.
• All four histogram families correct at 64 GPUs — edge counts, direction re-keying, size-class bucketing, and gap complement verified against ground truth.
• Memory bounded. Per-node collector RSS flat at ~177 MB (metrics-only); ~205 MB plateau with tracing enabled — a fixed offset, not growth. Zero refused datapoints.
• Recv asymmetry is by design: recv-edge means exceed send-edge means because the recv envelope includes the remote-sender-ready wait — this is the receiver-side wait signal, documented in docs/otel-support.md.
• Cardinality governance under stress: a 448-edge all-to-all at natural fidelity produced 8,728 series; with a Top-K cap of 16 it folded to 3,624 total (overflow correctly aggregated), zero refused.

Micro-benchmarks (benches/otel_utils.rs, criterion)

Hot-path record costs for all three instruments plus the gap tracker's idle transition and nested-activity paths (net_send/net_recv latency record, collective duration record, gap idle transition, gap nested activity) — provided so the per-op overhead claim is reproducible and guarded against regression.

Files

• src/otel_utils.rs — DurationHistogram, GapTracker, recv re-keying in the latency histogram manager
• src/cloud_daemon.rs — recv latency manager wiring; duration/gap recording on the completion path
• src/profiler.rs — activity-window hooks for the gap tracker; num_active fix
• src/step_tracker.rs — u32 saturation fix
• src/lib.rs, Cargo.toml — module/bench registration (copybara-marked)
• benches/otel_utils.rs — criterion micro-benchmarks
• docs/otel-support.md — documents the three instruments, their attribute keying, gating, and the send/recv asymmetry