Skip to content

fix: match Content-Type by media type, not exact string, in body parsers - #2

Open
ertakbuga wants to merge 1 commit into
RobertWHurst:masterfrom
ertakbuga:fix/content-type-media-type-matching
Open

fix: match Content-Type by media type, not exact string, in body parsers#2
ertakbuga wants to merge 1 commit into
RobertWHurst:masterfrom
ertakbuga:fix/content-type-media-type-matching

Conversation

@ertakbuga

@ertakbuga ertakbuga commented Jul 30, 2026

Copy link
Copy Markdown

The json, msgpack and protobuf middlewares install their request body unmarshaller only when Content-Type is exactly equal to the media type:

contentType := ctx.RequestHeaders().Get("Content-Type")
if contentType != "application/json" {
    return
}

RFC 9110 §8.3.1 makes the type and subtype case-insensitive, and parameters are carried separately from the type/subtype pair that identifies the media type. RFC 9110 does caution that a parameter can be significant depending on the media type's own registration — for application/json it isn't, since RFC 8259 §11 registers no parameters at all — so application/json; charset=utf-8 names application/json. Under exact comparison it doesn't match, no unmarshaller is installed, and the request reaches the handler where UnmarshalRequestBody fails with no request body unmarshaller set. use SetRequestBodyUnmarshaller() or add body parser middleware — a developer-facing message about a request that was in fact well formed.

OkHttp appends ; charset=utf-8 to any string request body, so a client using it is silently unable to send a body at all. We hit this in production: an Android fleet couldn't fetch advertising for a week, 400ing on every request.

The change

internal/mediatype.Is compares a header against a media type per RFC 9110 §8.3.1 — parameters and case ignored — and all three call sites use it.

RFC 9110 defines a media type as token "/" token, so no quoted string can precede the first ;. Is therefore cuts at the first ;, trims, folds case, and requires both halves to be tokens. Four guards there are each load-bearing, and each is present because removing it changes an answer:

  • Both halves must be tokens. Otherwise a value that isn't a single media type gets matched on its tail: application/pdf, junk+json reads as JSON, and — worse — application/protobuf, evil+msgpack installs the msgpack unmarshaller for a header naming protobuf.
  • No wildcards. * is a valid tchar, so without an explicit rejection application/*+json reaches the suffix rule and matches. A wildcard names a range rather than one type, and belongs in Accept.
  • The type must match, not just the suffix. Otherwise text/x+json names application/json.
  • A 255-character limit, checked before folding case. RFC 6838 §4.2 caps a type or subtype name at 127 characters, so nothing longer names a media type. Without the cap, cost is proportional to the header rather than constant — Content-Type is client-supplied and net/http allows 1 MB of headers by default, which measured ~2.3 ms and a 1 MB allocation per request, against ~1 ns for the exact compare this replaces.

The structured syntax suffix convention (RFC 6838 §4.2.8) also matches, so application/problem+json and application/vnd.api+json name application/json. Flagging this as a choice rather than a spec requirement: it's deliberately wider than the IANA structured syntax suffix registry, which lists +json but neither +msgpack nor +protobuf. The reasoning is that the suffix is how a client says its type is structured as that format, so honouring it uniformly across the three parsers seemed more useful than honouring it for JSON alone — but I'm happy to narrow it to +json, or drop suffix matching from this PR entirely, if you'd rather keep it to the parameter fix.

Behaviour for genuinely non-matching content types is unchanged — the existing text/plain test still passes.

On tests: each middleware gains a non-matching-Content-Type test asserting that no unmarshaller is installed. That direction was worth adding explicitly, because hardcoding the comparison to true passed all three middleware suites — the existing TestMiddleware_NonJSONContentType reads the raw body without calling UnmarshalRequestBody, so it can't observe a wrongly-installed unmarshaller. With the new tests, the same mutation fails all four packages, and so does removing any one of the four guards above.

go test ./..., -race, vet and gofmt are green. I also updated the three README middleware sections, which still documented exact-string handling.

Two things I left out deliberately, happy to follow up on either

  1. Aliases some clients send in the wild: application/x-msgpack, application/x-protobuf, text/json. That felt like your call about what to accept rather than a spec fix, so they're explicitly tested as not matching.

  2. The failure mode itself, which I think is the more valuable change. A missing unmarshaller is a silent no-op, and it surfaces as an internal message from whichever handler happens to read the body. A 415, or even just an exported sentinel error so handlers can distinguish "wrong content type" from "malformed JSON", would have saved us most of a week — the error said decoding failed, so we spent six rounds examining a payload the router had never attempted to decode.

One note on CI

The Build and golangci-lint jobs are already failing on master at faaf5fe, independently of this PR — this branch touches no go.mod, go.sum or workflow file. The workflow pins go-version: '1.24' while go.mod requires go 1.25.0, which looks like the cause, though the run logs are past retention so I couldn't confirm it. Glad to send that as a separate one-line PR if it'd be useful.

The json, msgpack and protobuf middlewares each installed their request body
unmarshaller only when Content-Type was exactly equal to the media type:

    contentType := ctx.RequestHeaders().Get("Content-Type")
    if contentType != "application/json" {
        return
    }

RFC 9110 section 8.3.1 makes the type and subtype case-insensitive and carries
parameters separately from the type/subtype pair that identifies the media
type. RFC 9110 does caution that a parameter may be significant depending on
the media type's own registration; for these three it is not, because RFC 8259
section 11 registers no parameters at all for application/json. So
"application/json; charset=utf-8" names application/json, and under exact
comparison it did not match: no unmarshaller was installed, and the request
reached the handler where UnmarshalRequestBody failed with navaros' own "no
request body unmarshaller set" message about a request that was well formed.

OkHttp appends "; charset=utf-8" to any string request body, so a client using
it could not send a body at all. This hit us in production, where an Android
fleet could not fetch advertising for a week.

internal/mediatype.Is now performs the comparison and all three call sites use
it. It cuts the header at the first ";" -- RFC 9110 defines a media type as
token "/" token, so no quoted string can precede it and parsing parameters was
never necessary to find the media type -- then trims, folds case, and applies
four guards, each of which changes an answer when removed:

  - Both halves must be tokens. Otherwise a value that is not a single media
    type gets matched on its tail: "application/pdf, junk+json" reads as JSON,
    and "application/protobuf, evil+msgpack" installs the msgpack unmarshaller
    for a header naming protobuf.
  - No wildcards. "*" is a valid tchar, so without an explicit rejection
    "application/*+json" reaches the suffix rule and matches. A wildcard names
    a range rather than one type and belongs in Accept.
  - The type must match, not only the suffix, or "text/x+json" names
    application/json.
  - At most 255 characters, checked before folding case. RFC 6838 section 4.2
    caps a type or subtype name at 127 characters, so nothing longer names a
    media type. Content-Type is client-supplied and net/http allows 1 MB of
    headers by default; without the limit a 1 MB value cost ~2.3 ms and a 1 MB
    allocation per request, against ~1 ns for the compare this replaces.

The structured syntax suffix convention (RFC 6838 section 4.2.8) also matches,
so "application/problem+json" and "application/vnd.api+json" name
application/json. This is deliberately wider than the IANA structured syntax
suffix registry: of the three suffixes at issue here only "+json" is
registered, and "+msgpack" and "+protobuf" are honoured anyway because the
suffix is how a client says its type is structured as that format. That is a
choice rather than a requirement, and the pull request offers to narrow it.

Behaviour for genuinely non-matching content types is unchanged; the
pre-existing text/plain test still passes. Aliases some clients send --
application/x-msgpack, application/x-protobuf, text/json,
application/vnd.google.protobuf -- are explicitly tested as not matching,
since which of those to accept is a decision about what the library accepts
rather than a spec correction.

Tests cover the helper and, per middleware, both a parameterized-header
positive case and a non-matching-Content-Type negative case. The negative
direction was worth adding explicitly: hardcoding the comparison to true
passed all three middleware suites beforehand, because the existing
TestMiddleware_NonJSONContentType reads the raw body without calling
UnmarshalRequestBody and so cannot observe a wrongly-installed unmarshaller.
With these tests that mutation fails all four packages, and so does removing
any one of the four guards above.

The three README middleware sections are updated; they documented exact-string
handling.
@ertakbuga
ertakbuga force-pushed the fix/content-type-media-type-matching branch from 01b10e1 to 6d6d74d Compare July 30, 2026 16:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant