Repository navigation
fix: stream uploads to disk instead of buffering them in memory - #9
Merged
Merged
Conversation
ParseMultipartForm was given the upload size limit as its maxMemory argument, so uploads up to MAX_UPLOAD_SIZE_MB were held entirely in the Go heap. Concurrent large uploads could exhaust the container memory limit. Read the file part with MultipartReader and copy it straight to the temp file, capping the body with http.MaxBytesReader. Oversized uploads now return 413. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
handleScan writes uploads to /tmp/scans, which only exists in the container image, so TestHandleScanTooLarge failed on a clean machine. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
io.Copy errors from the temp file (disk full, I/O error) were wrapped as a malformed request and returned as 400. Classify only failures reading the request body; write failures return 500 again. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This was referenced Oct 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
handleScancalledr.ParseMultipartForm(config.MaxUploadSize). That argument ismaxMemory, not a size cap, so withMAX_UPLOAD_SIZE_MB=1024any upload up to 1 GiB was held entirely in the Go heap (with extra transient allocation from buffer growth) before being copied to disk. Several concurrent large extensions can exhaust the container memory limit and get the pod OOM-killed.Change
filepart withr.MultipartReader()andio.Copyit directly into the temp file; memory use no longer depends on upload size.http.MaxBytesReader(MaxUploadSize), which the old code never enforced. Oversized uploads now return413with a JSON error.Behavior is otherwise unchanged: same status codes for non-multipart (400) and missing-file (400) requests. Non-file fields in the form are skipped rather than parsed.
Tests
TestSaveUploadStreamsToDisk: streams a 64 MiB upload and asserts allocations stay below a quarter of its size. With the upload buffered (io.ReadAll) the same test fails (about 165 MB allocated).TestSaveUploadTooLarge,TestHandleScanTooLarge: 413 and no temp file left behind.yrbinary are skipped locally, as before.Not included
No concurrency limit, request-context cancellation or
GOMEMLIMITyet; those are separate follow-ups.clamav-rest-newhas the sameParseMultipartFormpattern and would need the same change.🤖 Generated with Claude Code