Skip to content

ConfigStd.Unmarshal silently accepts invalid JSON string escapes where encoding/json returns an error #938

Description

@brualan

Describe the bug

sonic.ConfigStd.Unmarshal accepts JSON strings containing invalid escape sequences
(e.g. \0, \x, \v, \1) without returning an error. encoding/json correctly
rejects the same input with, for \0:

invalid character '0' in string escape code

This is a strict violation of RFC 8259 §7. The ABNF grammar defines the only valid characters
after an escape (\, %x5C) as:

Sequence Meaning
\" quotation mark
\\ reverse solidus
\/ solidus
\b backspace
\f form feed
\n line feed
\r carriage return
\t tab
\uXXXX Unicode code point

None of \0, \x, \v, \1, etc. matches any of these productions.
There is no ambiguity, no implementation-defined behaviour, and no optional leniency in the spec, because they are hard parse errors.

sonic.ConfigStd is documented as a drop-in replacement for encoding/json and is configured with ValidateString: true, making silent acceptance here doubly unexpected.

The bug is specific to the skip-value path: when unmarshaling into a struct and the JSON key has no matching field, sonic skips the value with a SIMD-accelerated string scanner that locates the closing quote but does not validate the content of escape sequences.
The escape-content check only runs in the scalar fallback used for short strings, so the bug surfaces once the string is long enough for SIMD to take over. Empirically, ~29 bytes of remaining string content, and it persists at all greater lengths.
Matched struct fields and interface{} targets go through different paths and are handled correctly.

The relevant code is advance_string_validate in native/scanning.h: the 32-byte and 64-byte SIMD chunks return as soon as the closing quote is found, while escape-sequence validation (advance_escape_validate) is only invoked from the scalar tail loop.

To Reproduce

package main

import (
        stdjson "encoding/json"
        "fmt"

        "github.com/bytedance/sonic"
)

// S has no field matching key "" — sonic takes its skip-value fast path.
type S struct{}

// Each input contains a different invalid escape (\0, \1, \v) followed by enough
// padding to push the string past sonic's SIMD threshold (~29 bytes after `\`).
// The padding is what triggers the bug: shorter strings fall back to the scalar
// validator, which correctly rejects the escape.
var inputs = [][]byte{
        []byte(`{"":"\0` + strings.Repeat("p", 28) + `"}`), // \0
        []byte(`{"":"\1` + strings.Repeat("p", 28) + `"}`), // \1
        []byte(`{"":"\v` + strings.Repeat("p", 28) + `"}`), // \v
}

func main() {
        for _, input := range inputs {
                var std, son S
                fmt.Printf("input=%s\n", input)
                fmt.Printf("  encoding/json:  %v\n", stdjson.Unmarshal(input, &std))
                fmt.Printf("  sonic.ConfigStd: %v\n", sonic.ConfigStd.Unmarshal(input, &son))
        }
}

Output:

input={"":"\0pppppppppppppppppppppppppppp"}
  encoding/json:  invalid character '0' in string escape code
  sonic.ConfigStd: <nil>
input={"":"\1pppppppppppppppppppppppppppp"}
  encoding/json:  invalid character '1' in string escape code
  sonic.ConfigStd: <nil>
input={"":"\vpppppppppppppppppppppppppppp"}
  encoding/json:  invalid character 'v' in string escape code
  sonic.ConfigStd: <nil>

Expected behavior

sonic.ConfigStd.Unmarshal returns a non-nil error for any input containing an invalid
escape sequence, matching encoding/json.

Actual behaviour

sonic returns nil and silently ignores the malformed escape, as shown in the output above.

Sonic version:

v1.15.1

Environment:

AR='ar'
CC='gcc'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_ENABLED='1'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
CXX='g++'
GCCGO='gccgo'
GO111MODULE=''
GOAMD64='v1'
GOARCH='amd64'
GOAUTH='netrc'
GOBIN=''
GOCACHE='/home/brualan/.cache/go-build'
GOCACHEPROG=''
GODEBUG=''
GOENV='/home/brualan/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFIPS140='off'
GOFLAGS=''
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/nix-shell-66243-0/go-build1474066616=/tmp/go-build -gno-record-gcc-switches'
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMOD='/tmp/tmp.59XRHDnlbj/go.mod'
GOMODCACHE='/home/brualan/go/pkg/mod'
GOOS='linux'
GOPATH='/home/brualan/go'
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/nix/store/0hfiiapvcwhdxc30m6606qx79nrik7jz-go-1.25.5/share/go'
GOSUMDB='sum.golang.org'
GOTELEMETRY='local'
GOTELEMETRYDIR='/home/brualan/.config/go/telemetry'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/nix/store/0hfiiapvcwhdxc30m6606qx79nrik7jz-go-1.25.5/share/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.25.5'
GOWORK=''
PKG_CONFIG='pkg-config'

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions