fix: match Content-Type by media type, not exact string, in body parsers - #2
Open
ertakbuga wants to merge 1 commit into
Open
Conversation
The json, msgpack and protobuf middlewares each installed their request body
unmarshaller only when Content-Type was exactly equal to the media type:
contentType := ctx.RequestHeaders().Get("Content-Type")
if contentType != "application/json" {
return
}
RFC 9110 section 8.3.1 makes the type and subtype case-insensitive and carries
parameters separately from the type/subtype pair that identifies the media
type. RFC 9110 does caution that a parameter may be significant depending on
the media type's own registration; for these three it is not, because RFC 8259
section 11 registers no parameters at all for application/json. So
"application/json; charset=utf-8" names application/json, and under exact
comparison it did not match: no unmarshaller was installed, and the request
reached the handler where UnmarshalRequestBody failed with navaros' own "no
request body unmarshaller set" message about a request that was well formed.
OkHttp appends "; charset=utf-8" to any string request body, so a client using
it could not send a body at all. This hit us in production, where an Android
fleet could not fetch advertising for a week.
internal/mediatype.Is now performs the comparison and all three call sites use
it. It cuts the header at the first ";" -- RFC 9110 defines a media type as
token "/" token, so no quoted string can precede it and parsing parameters was
never necessary to find the media type -- then trims, folds case, and applies
four guards, each of which changes an answer when removed:
- Both halves must be tokens. Otherwise a value that is not a single media
type gets matched on its tail: "application/pdf, junk+json" reads as JSON,
and "application/protobuf, evil+msgpack" installs the msgpack unmarshaller
for a header naming protobuf.
- No wildcards. "*" is a valid tchar, so without an explicit rejection
"application/*+json" reaches the suffix rule and matches. A wildcard names
a range rather than one type and belongs in Accept.
- The type must match, not only the suffix, or "text/x+json" names
application/json.
- At most 255 characters, checked before folding case. RFC 6838 section 4.2
caps a type or subtype name at 127 characters, so nothing longer names a
media type. Content-Type is client-supplied and net/http allows 1 MB of
headers by default; without the limit a 1 MB value cost ~2.3 ms and a 1 MB
allocation per request, against ~1 ns for the compare this replaces.
The structured syntax suffix convention (RFC 6838 section 4.2.8) also matches,
so "application/problem+json" and "application/vnd.api+json" name
application/json. This is deliberately wider than the IANA structured syntax
suffix registry: of the three suffixes at issue here only "+json" is
registered, and "+msgpack" and "+protobuf" are honoured anyway because the
suffix is how a client says its type is structured as that format. That is a
choice rather than a requirement, and the pull request offers to narrow it.
Behaviour for genuinely non-matching content types is unchanged; the
pre-existing text/plain test still passes. Aliases some clients send --
application/x-msgpack, application/x-protobuf, text/json,
application/vnd.google.protobuf -- are explicitly tested as not matching,
since which of those to accept is a decision about what the library accepts
rather than a spec correction.
Tests cover the helper and, per middleware, both a parameterized-header
positive case and a non-matching-Content-Type negative case. The negative
direction was worth adding explicitly: hardcoding the comparison to true
passed all three middleware suites beforehand, because the existing
TestMiddleware_NonJSONContentType reads the raw body without calling
UnmarshalRequestBody and so cannot observe a wrongly-installed unmarshaller.
With these tests that mutation fails all four packages, and so does removing
any one of the four guards above.
The three README middleware sections are updated; they documented exact-string
handling.
ertakbuga
force-pushed
the
fix/content-type-media-type-matching
branch
from
July 30, 2026 16:49
01b10e1 to
6d6d74d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The json, msgpack and protobuf middlewares install their request body unmarshaller only when
Content-Typeis exactly equal to the media type:RFC 9110 §8.3.1 makes the type and subtype case-insensitive, and parameters are carried separately from the type/subtype pair that identifies the media type. RFC 9110 does caution that a parameter can be significant depending on the media type's own registration — for
application/jsonit isn't, since RFC 8259 §11 registers no parameters at all — soapplication/json; charset=utf-8namesapplication/json. Under exact comparison it doesn't match, no unmarshaller is installed, and the request reaches the handler whereUnmarshalRequestBodyfails withno request body unmarshaller set. use SetRequestBodyUnmarshaller() or add body parser middleware— a developer-facing message about a request that was in fact well formed.OkHttp appends
; charset=utf-8to any string request body, so a client using it is silently unable to send a body at all. We hit this in production: an Android fleet couldn't fetch advertising for a week, 400ing on every request.The change
internal/mediatype.Iscompares a header against a media type per RFC 9110 §8.3.1 — parameters and case ignored — and all three call sites use it.RFC 9110 defines a media type as
token "/" token, so no quoted string can precede the first;.Istherefore cuts at the first;, trims, folds case, and requires both halves to be tokens. Four guards there are each load-bearing, and each is present because removing it changes an answer:application/pdf, junk+jsonreads as JSON, and — worse —application/protobuf, evil+msgpackinstalls the msgpack unmarshaller for a header naming protobuf.*is a validtchar, so without an explicit rejectionapplication/*+jsonreaches the suffix rule and matches. A wildcard names a range rather than one type, and belongs inAccept.text/x+jsonnamesapplication/json.Content-Typeis client-supplied andnet/httpallows 1 MB of headers by default, which measured ~2.3 ms and a 1 MB allocation per request, against ~1 ns for the exact compare this replaces.The structured syntax suffix convention (RFC 6838 §4.2.8) also matches, so
application/problem+jsonandapplication/vnd.api+jsonnameapplication/json. Flagging this as a choice rather than a spec requirement: it's deliberately wider than the IANA structured syntax suffix registry, which lists+jsonbut neither+msgpacknor+protobuf. The reasoning is that the suffix is how a client says its type is structured as that format, so honouring it uniformly across the three parsers seemed more useful than honouring it for JSON alone — but I'm happy to narrow it to+json, or drop suffix matching from this PR entirely, if you'd rather keep it to the parameter fix.Behaviour for genuinely non-matching content types is unchanged — the existing
text/plaintest still passes.On tests: each middleware gains a non-matching-
Content-Typetest asserting that no unmarshaller is installed. That direction was worth adding explicitly, because hardcoding the comparison totruepassed all three middleware suites — the existingTestMiddleware_NonJSONContentTypereads the raw body without callingUnmarshalRequestBody, so it can't observe a wrongly-installed unmarshaller. With the new tests, the same mutation fails all four packages, and so does removing any one of the four guards above.go test ./...,-race,vetandgofmtare green. I also updated the three README middleware sections, which still documented exact-string handling.Two things I left out deliberately, happy to follow up on either
Aliases some clients send in the wild:
application/x-msgpack,application/x-protobuf,text/json. That felt like your call about what to accept rather than a spec fix, so they're explicitly tested as not matching.The failure mode itself, which I think is the more valuable change. A missing unmarshaller is a silent no-op, and it surfaces as an internal message from whichever handler happens to read the body. A 415, or even just an exported sentinel error so handlers can distinguish "wrong content type" from "malformed JSON", would have saved us most of a week — the error said decoding failed, so we spent six rounds examining a payload the router had never attempted to decode.
One note on CI
The
Buildandgolangci-lintjobs are already failing onmasteratfaaf5fe, independently of this PR — this branch touches nogo.mod,go.sumor workflow file. The workflow pinsgo-version: '1.24'whilego.modrequiresgo 1.25.0, which looks like the cause, though the run logs are past retention so I couldn't confirm it. Glad to send that as a separate one-line PR if it'd be useful.