Skip to content

fix(storage): send explicit Content-Length on every S3 request - #255

Merged
ArnabChatterjee20k merged 4 commits into
mainfrom
fix/s3-content-length
Sep 10, 2026
Merged

fix(storage): send explicit Content-Length on every S3 request#255
ArnabChatterjee20k merged 4 commits into
mainfrom
fix/s3-content-length

Conversation

@ArnabChatterjee20k

Copy link
Copy Markdown
Contributor

Problem

Uploading large (chunked) files to S3-compatible storage — reported against GCS — fails with:

HTTP 411 — POST requests require a Content-length header

Reported downstream in appwrite/appwrite#13548 (self-hosted 2.0, files 15 MB–550 MB).

Cause

S3::call() handed a streamed body to the client transport without a Content-Length header. The cURL adapter then:

  • streams a body of unknown size (StreamInterface::getSize() returns null) with Transfer-Encoding: chunked, and
  • emits no length header at all for an empty-body POST (e.g. createMultipartUpload).

Many S3-compatible services (GCS in particular) reject both with 411.

Fix

hashBody() already reads the whole (seekable) body once to sign it, so it now also counts the bytes and returns the length. call() sets an explicit, signed content-length header on every request (0 for empty bodies). The value is the full body size, so it matches what the transport already sends for a size-known stream — no behavioural change there, correct headers for the streamed / empty-body cases. No new reads of the body are introduced.

Ports the fix from the archived standalone utopia-php/storage PR #171 into the monorepo (packages/storage).

Testing

  • php -l clean; single hashBody caller updated.
  • Vendor deps not installed in this checkout, so PHPUnit/PHPStan not run locally — CI will cover.

Fixes appwrite/appwrite#13548

Uploading large (chunked) files to S3-compatible storage such as GCS
failed with HTTP 411 "POST requests require a Content-length header".

S3::call() handed a streamed body to the transport without a
Content-Length header. The cURL adapter then streams a body of unknown
size with Transfer-Encoding: chunked, and emits no length at all for an
empty-body POST (createMultipartUpload) — both of which S3-compatible
services reject with 411.

hashBody() already reads the whole (seekable) body once to sign it, so
count the bytes there and set an explicit, signed content-length header
for every request (0 for empty bodies). The value equals the full body
size, matching what the transport sends for a size-known stream.

Ports the fix from the archived utopia-php/storage PR #171 into the
monorepo.

Fixes appwrite/appwrite#13548
@greptile-apps

greptile-apps Bot commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 5/5

The PR appears safe to merge; the S3 request-length behavior is implemented consistently and covered through observable public behavior.

Summary

  • Prevents unknown-size streams from falling back to chunked transfer encoding.
  • Covers exact streamed-body length through the public write() API.
  • Covers empty multipart-initiation requests through the public prepare() API.

Reviews (3) · Last reviewed commit: "test(storage): reach empty-body multipar..."

Comment thread packages/storage/src/Storage/Device/S3.php
Regression coverage for the HTTP 411 fix, driving the real S3::call()
with a capturing PSR-18 client:

- a streamed write sends Content-Length equal to the exact body size
  (never Transfer-Encoding: chunked)
- an empty-body multipart POST (createMultipartUpload) sends
  Content-Length: 0

Both requests are what GCS rejected before the fix.
Comment thread packages/storage/tests/Storage/Device/S3/ContentLengthTest.php Outdated
ArnabChatterjee20k and others added 2 commits September 10, 2026 18:03
CI's storage check failed on two lint rules in the new regression test:
pint's single_line_empty_body (collapse the empty CapturingClient
constructor body) and rector's AssertEmptyNullableObjectToAssertInstanceof
(assertNotNull on a nullable object -> assertInstanceOf).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UQ1CyiMrnx7CNQDi85gxKa
The empty-body Content-Length test drove the protected
createMultipartUpload() through a test-only subclass, coupling it to an
implementation helper. Multi-chunk prepare() reaches the same
multipart-initiation POST through the public API, so drive it that way
and drop the ContentLengthS3 subclass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UQ1CyiMrnx7CNQDi85gxKa
@ArnabChatterjee20k
ArnabChatterjee20k merged commit 3afc0d2 into main Sep 10, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2.0 selfhost: Files fail to upload to s3 compatible storages due to missing Content-length header

2 participants