Server finalization can silently lose a client's live log before EOF

[YuanTingHsieh]

Description

A client live-log stream can be lost when the server finalizes a job before that client finishes its normal log drain and sends EOF. The job still reports FINISHED:COMPLETED, but the downloaded job result permanently lacks that client's log.json.

This was first observed intermittently in the hello-log-streaming integration test. A controlled A/B test makes the ordering deterministic.

Environment

• NVFlare main: b2e54a3073fa9cebcd0c8433bc1edc280b883c63
• Source tree: 9240a8fb9a336d4583414a2ae08664c6f562fa25
• Two-client POC using the maintained examples/hello-world/hello-log-streaming recipe
• Fresh detached checkout; no NVFlare product source changes

Deterministic reproducer

1. Export the maintained hello-log-streaming recipe for two clients and one round.

2. Replace the generated job's log streamer and receiver FQCNs with test-only subclasses. The receiver subclass only records timestamps. The streamer subclass delays site-2 during ABOUT_TO_END_RUN, immediately before calling the normal JobLogStreamer stop/drain/EOF path:

   def _on_about_to_end_run(self, event_type, fl_ctx):
       if fl_ctx.get_identity_name() == "site-2":
           time.sleep(self.delay_seconds)
       super()._on_about_to_end_run(event_type, fl_ctx)

3. Run controls with delay_seconds=0.0.

4. Run delayed cases with delay_seconds=15.0 for site-2.

5. Submit each job serially, wait for its exact terminal status, download the result immediately, wait 22 seconds, download it again, and check for workspace/site-1/log.json and workspace/site-2/log.json.

The only A/B configuration change is delay_seconds: 0.0 to 15.0.

Results

Arm	Repetitions	Terminal status	Immediate result	Result after 22 seconds
Control	2/2	FINISHED:COMPLETED	Both client logs present	Both client logs present
Delayed site-2 EOF	3/3	FINISHED:COMPLETED	site-2/log.json absent	site-2/log.json still absent

Representative ordering from the three delayed runs:

Run	Server runner finished	Site-2 resumed drain/EOF	Gap	Outcome
1	00:02:53.023Z	00:03:05.626Z	12.603 s later	target_unreachable / timeout; log absent
2	00:03:40.152Z	00:03:52.708Z	12.556 s later	target_unreachable / timeout; log absent
3	00:04:26.215Z	00:04:38.782Z	12.567 s later	target_unreachable / timeout; log absent

The client-local source log.json still existed in all delayed cases. The server-side receiver's stream-done persistence callback was never reached, and preserved server state also lacked the file. Therefore the log was lost, not merely omitted from an early downloaded snapshot.

The natural timing window is intermittent under CI load; the controlled 15-second delay reproduces it deterministically (3/3) against passing controls (2/2).

Expected behavior

Before a job reports successful completion and its server workspace is finalized, every accepted live-log stream should either:

• complete its bounded drain/EOF and be persisted, or
• terminate with an explicit failure/partial-log disposition that is visible to the job outcome and operator.

The server should not report successful completion while silently losing an active client's log.

Likely lifecycle race

At the tested commit:

1. ServerRunner sends ReservedTopic.END_RUN with timeout=0.0.
2. It performs server-local CHECK_END_RUN_READINESS, fires server END_RUN, then shuts down its streamer.
3. JobLogReceiver persists client data only from its stream-done callback and does not participate in CHECK_END_RUN_READINESS.
4. JobLogStreamer performs its client-side drain and EOF asynchronously during the client's end-run sequence.

Consequently, server finalization can overtake a still-active client stream. When that client later sends EOF, the target is already unavailable and the receiver never persists the log.

Possible fix directions include tracking active streams per job in JobLogReceiver and participating in bounded end-run readiness, or otherwise introducing a server/client end-run barrier that guarantees accepted streams reach a terminal persistence state before streamer shutdown and workspace archival.

History

• Live job-log streaming was introduced in PR #4454.
• The recipe integration coverage that exposes the missing-log symptom was added in PR #4915.