Skip to content

ulmshare: stream raw data in large blocks instead of one request per frame - #43

Merged
tristan-deep merged 2 commits into
mainfrom
ulmshare-quiet-streaming
Sep 25, 2026
Merged

tristan-deep merged 2 commits into
mainfrom
ulmshare-quiet-streaming

Conversation

@tristan-deep

@tristan-deep tristan-deep commented Sep 25, 2026 •

Copy link
Copy Markdown
Owner

Follow-up to #42: make streaming raw data from the Hub work for long runs.

raw_data is stored one frame per chunk, and zea 0.1.6 sends one HTTP range request per chunk. A full acquisition therefore meant ~76000 requests, which hits Hugging Face's rate limit (HTTP 429). zea's retries don't wait long enough to get past it. Each request also drew its own "streaming …" progress bar.

beamform_stack now reads raw_data through a new open_raw_data. For hf:// paths this opens the file with h5py through an HfFileSystem block cache, so consecutive frame chunks arrive in a few large requests. Local paths are opened directly. The block size is about 1/16 of the bytes the run needs, capped at 64 MiB (STREAM_BLOCK_BYTES), so a short run fetches little more than the frames it uses. zea still reads parameters and the single B-mode frame. Outputs are unchanged, and the raw frames read are identical to zea's. The log line and progress bar now say the frames are being streamed ("Beamforming N frames (streaming from the Hub)", streaming + beamforming), or read from disk for a local file.

Measured on mouse_40_acquisition_1:

Frames Requests Fetched Chunk bytes used
1 8 2.1 MB 0.8 MB
8 15 8.7 MB 6.3 MB
400 14 406 MB 314 MB
1600 (default) 22 1476 MB 1258 MB

With zea's per-chunk reads, the default 1600 frames took 1600 requests. A full acquisition is now roughly 900 requests instead of ~76000.

The Hub doesn't host the scripts, so there's no HF counterpart.

Files changed: 1.

~ datasets/ulmshare/reconstruct.py

@tristan-deep tristan-deep changed the title ulmshare: silence per-batch streaming bars while beamforming ulmshare: stream raw data in large blocks instead of one request per frame Sep 25, 2026
@tristan-deep
tristan-deep force-pushed the ulmshare-quiet-streaming branch from 532147a to 58976b3 Compare September 25, 2026 08:29
@tristan-deep
tristan-deep merged commit 00cc91f into main Sep 25, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant