Skip to content

Go SDK: ExtractExtraProperties decodes every object three times #18118

Open

Description

@yershov-oleksandr

Summary

In SDKs from fern-go-sdk, every generated UnmarshalJSON calls internal.ExtractExtraProperties with the bytes it has just decoded into the struct. As a result, each object in a response is decoded three times:

  1. into the struct, by UnmarshalJSON itself;
  2. by json.Unmarshal(bytes, &value) inside ExtractExtraProperties. value is a struct held in an interface{}, so encoding/json replaces it with a fresh map[string]any, and that result is never used;
  3. into map[string]any once more, to find the extra properties.

Nested objects repeat this at every level, so one page of a listing is decoded several times over. The declared keys are also recomputed by reflection on every call.

Impact

We profiled one call to a listing endpoint through a generated SDK (fern-go-sdk 1.58.1): a page of 50 objects, each with four nested objects. ExtractExtraProperties accounts for 83.5% of the bytes allocated by the whole call. Decoding that page costs about 1.2 ms, 960 KiB and 20.9k allocations on Go 1.27.

Proposal

Validate the object once with json.Valid, which does not allocate, then scan it in place. Compare each property name as it appears in the input, skip a declared property's value without reading it, and decode only the extra properties. Cache the declared keys per type. The extra properties returned stay exactly the same. This uses encoding/json only, so it keeps working on every Go version the SDKs support. The implementation and benchmarks are in #18119.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions