fix(otel): end recording spans on non-terminal - #696
Conversation
95550ea to
0e1fd55
Compare
0e1fd55 to
51018c3
Compare
51018c3 to
610690e
Compare
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
af13837 to
3bbcad6
Compare
This comment has been minimized.
This comment has been minimized.
| start_time=span_start_time, | ||
| parent_span=parent_span, | ||
| existed=info.attempt != 1 and info.operation_type is not OperationType.STEP, | ||
| existed=existed, |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
There was a problem hiding this comment.
Addressed in 0090a01. InvocationOtelPlugin now reuses an existing recording CONTEXT span during same-invocation re-entry instead of overwriting it. The lifecycle tests now assert identity reuse for single and nested contexts and verify the original span stops recording after the terminal operation hook. Validation: 250 OTel tests passed, OTel type checking passed, and Hatch formatting is clean.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
| existing_span = self._get_span(span_key) | ||
| if ( | ||
| info.operation_type is OperationType.CONTEXT | ||
| and existing_span is not None | ||
| and existing_span.is_recording() | ||
| ): | ||
| # A timed in-process resume can re-enter a CONTEXT before its | ||
| # previous user-function scope reports an end. Continue the same | ||
| # invocation segment instead of overwriting and abandoning it. | ||
| span = existing_span | ||
| else: |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
| end_time = info.end_time | ||
| if end_time is not None and end_time == info.start_time: | ||
| end_time += datetime.timedelta(microseconds=1) | ||
| end_time = ensure_end_after_start(info.start_time, info.end_time) |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
This comment has been minimized.
This comment has been minimized.
Codex AI reviewNo actionable findings. Residual risk is limited to the unexecuted ADOT/OpenTelemetry compatibility matrix because this review was restricted to static inspection. Reviewed commit |
| if ( | ||
| info.operation_type is OperationType.CONTEXT | ||
| and existing_span is not None | ||
| and existing_span.is_recording() | ||
| ): | ||
| # A timed in-process resume can re-enter a CONTEXT before its | ||
| # previous user-function scope reports an end. Continue the same | ||
| # invocation segment instead of overwriting and abandoning it. | ||
| span = existing_span | ||
| else: |
There was a problem hiding this comment.
Claude AI review · Finding arf_v1_dcmwvtn22nrbcgqkeggzqzrdnv
The new reentry guard (existing_span is not None and existing_span.is_recording(): span = existing_span) only fires when info.operation_type is OperationType.CONTEXT. For a STEP attempt, execution always takes the else branch and calls self._start_span(...), whose unchanged body unconditionally does self._operation_spans[registry_key] = span, silently overwriting any span already registered under that attempt's span_key without ending it first.
This is exactly the "timed in-process resume re-enters the same operation" scenario this PR fixes for CONTEXT here, and fixes for every span type (attempts included) in ExecutionOtelPlugin._start_span, which now ends and marks a stale recording span (durable.span.replaced_on_reentry) before replacing it. In InvocationOtelPlugin, a re-entered STEP attempt's previous span is dropped from the registry without .end() ever being called on it, so it is never exported — a real telemetry gap. ExecutionOtelPlugin now has test_reentered_step_attempt_releases_the_previous_scope covering this for attempts; InvocationOtelPlugin has no equivalent test or fix.
Extend the is_recording() reuse check to STEP attempts too, or better, move an equivalent "end the stale recording span before replacing it" check into InvocationOtelPlugin._start_span itself (mirroring the execution plugin's fix) so every caller benefits, and add a regression test mirroring the execution-plugin one.
| end_time = info.end_time | ||
| if end_time is not None and end_time == info.start_time: | ||
| end_time += datetime.timedelta(microseconds=1) | ||
| end_time = ensure_end_after_start(info.start_time, info.end_time) |
There was a problem hiding this comment.
Claude AI review · Finding arf_v1_63zklrnsxmau6x72ea3mxj3ipz
end_time is recomputed via ensure_end_after_start(info.start_time, info.end_time) here, but it was already computed identically a few lines above (right after the matching-start check) and used for the _note_parent_end calls. info is a frozen dataclass, so neither input can have changed in between, making this a redundant recomputation. It's harmless today but is dead code that risks silently diverging from the first computation on a future edit (e.g. someone updates only one of the two call sites). Delete this line and reuse the end_time already in scope.
| end_time = ensure_end_after_start(info.start_time, info.end_time) |
Claude AI reviewReviewed the OTel plugin changes that switch open Workflow/operation spans to non-recording Most of the issues flagged in earlier review rounds on this PR are now fixed in this diff: Two issues from the prior round remain unresolved in this revision, listed below (one moderate telemetry-integrity gap, one minor dead-code nit). No other new correctness, determinism, or thread-safety issues were found in the changed code; test coverage for the new behavior is otherwise thorough. Residual risk: the missing STEP-attempt-reentry test/fix for Reviewed commit |
Issue #, if available:
#642
Description of changes:
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.