Repository navigation
ulmshare: stream raw data in large blocks instead of one request per frame - #43
Merged
Merged
Conversation
tristan-deep
force-pushed
the
ulmshare-quiet-streaming
branch
from
September 25, 2026 08:29
532147a to
58976b3
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #42: make streaming raw data from the Hub work for long runs.
raw_datais stored one frame per chunk, and zea 0.1.6 sends one HTTP range request per chunk. A full acquisition therefore meant ~76000 requests, which hits Hugging Face's rate limit (HTTP 429). zea's retries don't wait long enough to get past it. Each request also drew its own "streaming …" progress bar.beamform_stacknow readsraw_datathrough a newopen_raw_data. Forhf://paths this opens the file with h5py through anHfFileSystemblock cache, so consecutive frame chunks arrive in a few large requests. Local paths are opened directly. The block size is about 1/16 of the bytes the run needs, capped at 64 MiB (STREAM_BLOCK_BYTES), so a short run fetches little more than the frames it uses. zea still reads parameters and the single B-mode frame. Outputs are unchanged, and the raw frames read are identical to zea's. The log line and progress bar now say the frames are being streamed ("Beamforming N frames (streaming from the Hub)",streaming + beamforming), or read from disk for a local file.Measured on
mouse_40_acquisition_1:With zea's per-chunk reads, the default 1600 frames took 1600 requests. A full acquisition is now roughly 900 requests instead of ~76000.
The Hub doesn't host the scripts, so there's no HF counterpart.
Files changed: 1.