Skip to content

fix(relay): answer bytes a failed spool wrote; deflake the memory read - #1970

Open
matt2e wants to merge 1 commit into
mainfrom
flakey-tests-relay
Open

matt2e wants to merge 1 commit into
mainfrom
flakey-tests-relay

Conversation

@matt2e

@matt2e matt2e commented Oct 30, 2026

Copy link
Copy Markdown
Contributor

Summary

Fixes two Rust test flakes from the 23–30 Oct CI flake sweep. The first is a real defect in the media spool, so this PR changes product code. The second is test-only. This is one of six independent PRs split from the flakey-tests branch. Each is cut from main and can merge in any order.

A failed spool still answers the bytes it wrote (media_stream_tests.rs:277)

  • CI: 39970306754 (job 120470903732), 40829345046 (job 123299699319), 40958827040 (job 123745997146). a_failure_mid_stream_ends_the_response_short_of_its_length fails at :300 with 502 where 200 is expected. The case is a 200 response that sends 2 MiB of a 3 MiB Content-Length.
  • Cause: the spool reports progress through a tokio watch channel, which keeps only the latest value. The copy flushes the first MiB as Writing(n), then fails and overwrites that with Failed(502). A reader that polls only after both sends sees just Failed. read() then returned 502, even though the bytes it asked for were already on disk, and the response's first block went out as a 502 head. A player's range request can hit the same race in the app.
  • Change (media_spool.rs): Failed also carries the last byte count the copy reported. read() answers from disk whenever that count covers its range, and otherwise returns the failure status as before. lookup() still drops a failed spool, so later reads fetch the video again. A read past the written bytes still fails, and truncated_whole_videos_fail_and_are_fetched_again is unchanged.

The memory-read deadline test no longer depends on decode speed (agent.rs)

  • CI: 40877681973 (job 123460621816) got 16 entries where 32 were expected. It passed on re-run.
  • Cause: each page took 120 ms against a 300 ms deadline, and page 1 is decoded before page 2 is requested. Page 2 therefore made the deadline only if page 1's decode finished in about 60 ms, and a slow decode on CI pushed it past.
  • Change: read_through answers each page at once and leaves the request after the last page pending. The first case now has 32 events, two full pages, so the deadline always passes while the third page is awaited. The second case asserts the same as before.
  • Hang guard: because the last request now stays pending, a read that stopped honouring its deadline would hang instead of failing its took bound, and CI runs plain cargo test with no per-test timeout. read_through therefore wraps the read in a 5 s tokio::time::timeout that fails with "memory read outlived its deadline". The took assertions still check the real bound.

Verification (local macOS)

  • The new test a_failed_spool_still_answers_the_bytes_it_wrote forces the ordering. It first reads the tail, which can only return after the copy has failed, and then reads the first block, so the second read never sees Writing. Without the media_spool.rs change it fails on the first-block read with called `Result::unwrap()` on an `Err` value: 502. With the change it passes. :277 and truncated_whole_videos_fail_and_are_fetched_again also pass.
  • Memory read: a temporary probe (not committed) added a 100 ms decode stall after page 1. The old test then failed with left: 16, right: 32, the CI signature, and the new test passed. A second probe moved the deadline 1000× later. The test failed after 5.05 s with the hang-guard message instead of hanging.
  • Ran cargo fmt --check and cargo clippy --all-targets -D warnings on the Buzz crate, plus the four tests above by name. After the hang guard was added, only cargo fmt --check and the memory-read test were re-run. Clippy and the media tests were not.

Overlaps

Drafts #1461 and #967 rework relay_agent_memories_read, and #1461 also edits media_stream.rs. Neither changes the code paths fixed here.

Remaining gaps

  • The full Rust suite has not run locally. Hosted CI is its first full run.
  • A green run does not show that these flakes are gone.

🤖 Generated with Claude Code

@matt2e
matt2e requested a review from a team as a code owner October 30, 2026 11:56

@wesbillman wesbillman left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Star Lord’s automated source review via Wes’s account, integrating independent Rocket (systems/lifecycle), Mantis (correctness/security/concurrency), and Groot (product/error paths) reviews.

Reviewed head 1927a0208dd4501b5319ddbb8a16b88fd1d17964 against base 36a283b2386473130430890faa3b06c6e45d4a6b. No actionable findings; no changes requested. The failure state preserves the last completed disk-write boundary, allowing an existing reader to answer a fully written range while missing bytes still fail. Failed spools remain unavailable to new cache lookups. The added regression forces failure before reading the available prefix; the memory test retains its elapsed-time assertions while replacing delayed page delivery with a pending final page and a bounded hang guard. No browser cases or test infrastructure were changed.

Source-only review of the three-file diff, callers, cancellation/file lifetime, network-error recovery, repository guidance, and public description/commit text (no attached images). Hosted CI snapshot: CI required succeeded; Rust job 124048893021 logs show the four relevant tests passed and the Buzz crate reported 702 passed, 2 ignored. Windows native validation and advisory flake lanes were skipped. We ran no local tests or PR code and did not launch the app. Native playback, error/retry focus behavior, cross-platform acceptance, and long-term flake elimination remain unverified; the memory test still uses real-clock deadlines. This COMMENT is not approval or merge authorization.

ravarora2 added a commit that referenced this pull request Nov 3, 2026
## Why

At 200% interface size, a 220px sidebar can leave no width for
“Channels” or “Direct messages”. The header reserves scaled button space
inside a fixed-width sidebar. Fixes
[AIDA-2338](https://linear.app/squareup/issue/AIDA-2338).

## What

Cap the space reserved for actions so headings retain useful text width.
Keep the existing single-row height, adjacent chevron, full collapse
target, and fading for long names.

## How

Normal widths retain the original action clearance. At cramped enlarged
sizes, the padding leaves room for the label and chevron. Hovered or
keyboard-focused actions cover the trailing text with the panel surface,
following the existing Inbox overlay behavior. Shared buttons, menus,
and saved sidebar widths keep their existing owners.

Add one browser case for real text geometry, compact height, chevron
placement, pointer targets, and keyboard focus. Extend the existing
overflow-and-recovery case to cover a long custom name at 200%. No
existing browser cases are removed. These contracts need a browser
layout engine. The geometry matrix runs once; light and dark screenshots
cover paint separately.

## Risk

At the narrowest enlarged sizes, revealed actions cover the end of a
heading. Long names still fade, and the full accessible names remain
available. Default-size overflow and action alignment retain their
existing contracts.

## Testing

Local Chromium and WebKit fixtures exercised 100%, 120%, 150%, 180%, and
200% interface sizing at 220/260/520px sidebar widths. The complete
heading, navigation, group-icon, and section-creation files passed 40
checks, including the previously failing overflow and narrow drag cases.
Screenshots show the actual app UI with fixture messages at 100% and
200%, including both themes and revealed actions.

Mutation checks restore the original CSS at 200%/220px and the reviewed
wrapping layout: the new regression case rejects missing label space and
extra header rows respectively. The checks pass with the revised CSS.

All hosted browser journey shards passed at `3cbc353e1`. CI required is
currently blocked by the native media-stream test failure at
`media_stream_tests.rs:300`, matching the known spool-progress race
tracked by [#1970](#1970). Human
desktop acceptance remains pending. To try it: set Settings → Appearance
→ Interface size to 200%, narrow the sidebar fully, then hover and tab
through section actions. Headers should remain compact; actions should
open without collapsing their section.

Generated with Codex

---------

Signed-off-by: Ravneet Arora <rarora@squareup.com>
Co-authored-by: BuzzPRReviewAssistant1 <a581a7e8115222acdc32ffb14bfce55d48c24478d98e155f8eba126b69b83c7b@buzz.block.builderlab.xyz>
arividar pushed a commit to minniai/minni-buzz-app that referenced this pull request Nov 4, 2026
## Why

At 200% interface size, a 220px sidebar can leave no width for
“Channels” or “Direct messages”. The header reserves scaled button space
inside a fixed-width sidebar. Fixes
[AIDA-2338](https://linear.app/squareup/issue/AIDA-2338).

## What

Cap the space reserved for actions so headings retain useful text width.
Keep the existing single-row height, adjacent chevron, full collapse
target, and fading for long names.

## How

Normal widths retain the original action clearance. At cramped enlarged
sizes, the padding leaves room for the label and chevron. Hovered or
keyboard-focused actions cover the trailing text with the panel surface,
following the existing Inbox overlay behavior. Shared buttons, menus,
and saved sidebar widths keep their existing owners.

Add one browser case for real text geometry, compact height, chevron
placement, pointer targets, and keyboard focus. Extend the existing
overflow-and-recovery case to cover a long custom name at 200%. No
existing browser cases are removed. These contracts need a browser
layout engine. The geometry matrix runs once; light and dark screenshots
cover paint separately.

## Risk

At the narrowest enlarged sizes, revealed actions cover the end of a
heading. Long names still fade, and the full accessible names remain
available. Default-size overflow and action alignment retain their
existing contracts.

## Testing

Local Chromium and WebKit fixtures exercised 100%, 120%, 150%, 180%, and
200% interface sizing at 220/260/520px sidebar widths. The complete
heading, navigation, group-icon, and section-creation files passed 40
checks, including the previously failing overflow and narrow drag cases.
Screenshots show the actual app UI with fixture messages at 100% and
200%, including both themes and revealed actions.

Mutation checks restore the original CSS at 200%/220px and the reviewed
wrapping layout: the new regression case rejects missing label space and
extra header rows respectively. The checks pass with the revised CSS.

All hosted browser journey shards passed at `3cbc353e1`. CI required is
currently blocked by the native media-stream test failure at
`media_stream_tests.rs:300`, matching the known spool-progress race
tracked by [block#1970](block#1970). Human
desktop acceptance remains pending. To try it: set Settings → Appearance
→ Interface size to 200%, narrow the sidebar fully, then hover and tab
through section actions. Headers should remain compact; actions should
open without collapsing their section.

Generated with Codex

---------

Signed-off-by: Ravneet Arora <rarora@squareup.com>
Co-authored-by: BuzzPRReviewAssistant1 <a581a7e8115222acdc32ffb14bfce55d48c24478d98e155f8eba126b69b83c7b@buzz.block.builderlab.xyz>
@matt2e
matt2e force-pushed the flakey-tests-relay branch from 1927a02 to ec3a326 Compare November 5, 2026 02:46

@wesbillman wesbillman left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Star Lord’s automated source follow-up via Wes’s account, integrating independent Rocket, Mantis and Groot review lanes.

Reviewed head ec3a326ae8d7da9c0d37cdd46dcc162c0a505c4f against base f4805605f56a2f60bd25b966464f07f0c3292a41. No new actionable findings; no changes requested. The spool, block reader and loopback stream implementation are unchanged from the earlier reviewed revision. I checked the test’s integration with the rebased pipelined memory reader and its larger pages, along with media failure/retry consumers. Optional description clarification: the current first memory-test case uses 256 events (two 128-event pages); the historical 16/32 failure figures can stay identified as historical.

Hosted evidence at this head: the four relevant tests passed, and the Buzz crate reported 848 passed, 2 ignored in 38.12s. The Rust job nevertheless failed in crates/agent-command/tests/discovery.rs:438 with ExecutableFileBusy (outside this patch). Browser jobs were still running in the status snapshot; Windows validation was skipped. No green-CI claim.

Source-only review; no local tests, builds or app execution. Public description and feature commit text inspected; no attached images. Native playback, error/retry focus, cross-platform behavior and lasting flake elimination remain unverified. The memory test still uses real-clock deadlines; hosted success does not prove deterministic timing. This COMMENT is not approval or merge authorization.

These are §8 and the agent.rs memory-read row of §13 from the
23-30 Oct CI flake sweep. Before editing I checked origin/main
(f3446e8) and the open PRs. Neither unit was fixed since the sweep:
no commit on main touches media_spool.rs, media_stream_tests.rs or
the agent test, and no open PR touches media_spool.rs or either test.
Drafts #1461 and #967 rework relay_agent_memories_read, and #1461
also edits media_stream.rs. Neither changes the code paths fixed here.

media_stream_tests.rs:277
a_failure_mid_stream_ends_the_response_short_of_its_length
- CI runs: 39970306754 (job 120470903732), 40829345046 (job
  123299699319) and 40958827040 (job 123745997146). All three fail at
  :300 with 502 where 200 is expected. That is the case of a whole 200
  that sends 2 MiB of a 3 MiB Content-Length.
- Cause: the spool publishes its progress through a tokio watch
  channel, which keeps only the latest value. The copy flushes the
  first MiB as Writing(n), then fails and replaces that value with
  Failed(502). A reader that polls only after both sends sees just
  Failed. read() then returned 502, although the bytes it asked for
  were already on disk, and the response's first block went out as a
  502 head. A player's range request can hit the same race in the app.
- Change: Failed now also carries the last count the copy reported,
  and read() answers from disk whenever that count covers its range.
  Otherwise it returns the failure status, as before. lookup() still
  drops a failed spool, so later reads fetch the video again, and a
  read past the written bytes still fails
  (truncated_whole_videos_fail_and_are_fetched_again is unchanged).
- Verified: a new test, a_failed_spool_still_answers_the_bytes_it_wrote,
  opens a spool on that truncated 200. It first reads the tail, which
  can only return once the copy has failed, and then reads the first
  block. That second read never sees the Writing progress, so the
  ordering is forced rather than timed. Without the media_spool.rs
  change it fails on the first-block read with "called
  `Result::unwrap()` on an `Err` value: 502". With the change it
  passes. :277 and truncated_whole_videos_fail_and_are_fetched_again
  also pass.

agent.rs memory_read_keeps_what_it_decoded_when_its_deadline_passes
- CI run: 40877681973 (job 123460621816), which passed on re-run. It
  failed with 16 entries where 32 are expected.
- Cause: the first case relied on real-clock slack. Each page took
  120 ms against a 300 ms deadline, and page 1 is decoded before
  page 2 is requested, so page 2 only made the deadline if page 1's
  decode finished in about 60 ms. A slow decode on CI pushed page 2
  past the deadline.
- Change: read_through answers each page at once and leaves the
  request after the last page pending, so its fetch parameter is gone.
  The first case now has 32 events, two full pages, so the deadline
  always passes while the third page is awaited. The second case
  asserts the same as before. Its trailing request now pends instead
  of returning an empty page, but by then its deadline has already
  passed.
- Verified: a temporary probe, not committed, injected a 100 ms decode
  stall after page 1 through the helper's stall parameter. The old
  test then failed with "left: 16, right: 32", the CI signature. The
  new test passed. Without the probe, the new test passes.
- Hang guard: with the last request pending, a read that stopped
  honouring its deadline would hang this test rather than fail its
  `took` bound, and CI runs plain cargo test with no per-test timeout.
  read_through therefore bounds the read with a 5 s
  tokio::time::timeout that fails with "memory read outlived its
  deadline". This is a hang guard, not a flake fix; the `took`
  assertions still check the real bound. A temporary probe, not
  committed, set the read's deadline 1000 times later. The test then
  failed after 5.05 s with that message instead of hanging.

Local checks: cargo fmt --check and cargo clippy --all-targets
-D warnings on the Buzz crate, plus the four tests above, run by name.
After the hang guard was added, cargo fmt --check and the memory-read
test were re-run on this snapshot; clippy and the media tests were not.
Deferred: the full Rust suite and hosted CI.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Matt Toohey <contact@matttoohey.com>
@matt2e
matt2e force-pushed the flakey-tests-relay branch from ec3a326 to 19f79eb Compare November 5, 2026 06:55

@wesbillman wesbillman left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Star Lord’s automated source follow-up via Wes’s account, integrating independent Rocket, Mantis and Groot review lanes.

Reviewed head 19f79ebca0e9b18f4c116c661427bdd2dda95de7 against base 56729773f48d3f922a043ac83b7faa727bcfc9c2. No new actionable findings; no changes requested. The native relay subtree, media entry point and playback/recovery consumers are unchanged from reviewed head ec3a326ae8d7da9c0d37cdd46dcc162c0a505c4f. This follow-up checked the rebased integration; it does not reopen unrelated main-branch changes. The failed-spool byte boundary, retry/cache eviction and memory deadline-test behavior remain intact in source.

Source-only: no local tests, builds or app execution. Public description and feature commit text inspected; no attached media. Native playback, error/retry focus, performance, cross-platform behavior and lasting flake elimination remain unverified. The current-head CI snapshot still had Rust, JavaScript and browser jobs running; Windows and advisory flake lanes were skipped. Final test counts and timings were unavailable at that snapshot. This COMMENT is not approval or merge authorization.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants