Keep bytes past an embedded NUL in the cffi backend

[youdie006]

The yajl2_cffi backend silently truncates every string and key at an embedded NUL. The other three
parsers keep it.

import io, json
import ijson.backends.python as pyb, ijson.backends.yajl2_c as cb, ijson.backends.yajl2_cffi as cffib
load = lambda m, d: next(m.items(io.BytesIO(d), ''))

input                          python(ref)            yajl2_c                yajl2_cffi         stdlib json
{"k": "a\u0000b"}              {'k': 'a\x00b'}        {'k': 'a\x00b'}        {'k': 'a'}         {'k': 'a\x00b'}   MISMATCH
{"a\u0000b": 1}                {'a\x00b': 1}          {'a\x00b': 1}          {'a': 1}           {'a\x00b': 1}     MISMATCH
{"k": "\u0000"}                {'k': '\x00'}          {'k': '\x00'}          {'k': ''}          {'k': '\x00'}     MISMATCH
{"k": "pass\u0000word"}        {'k': 'pass\x00word'}  {'k': 'pass\x00word'}  {'k': 'pass'}      {'k': 'pass\x00word'} MISMATCH
{"k": "ok"}                    {'k': 'ok'}            {'k': 'ok'}            {'k': 'ok'}        {'k': 'ok'}       (control)

\u0000 is legal JSON (RFC 8259 section 7), so this is silent data loss rather than a rejected
input. It truncates keys too, so {"a\u0000b": 1} and {"a\u0000c": 2} collapse onto the same key.

Cause

yajl hands its string callbacks an explicit (pointer, length) pair, but
src/ijson/backends/yajl2_cffi.py:120 discards the length semantics:

def string(val, length):
    return ffi.string(val, maxlen=length).decode('utf-8')

ffi.string is NUL-terminated-C-string semantics - maxlen is a cap, not the length - so it stops
at the first zero byte. Same at map_key (:132).

The change

ffi.buffer(val, length)[:], which uses the length yajl actually gave. Two sites, four lines.

number() (:114) makes the same call, but a JSON number cannot contain a NUL, so I left it alone
rather than widen the diff.

Observable change: yajl2_cffi now returns strings containing \x00 where it previously
returned a prefix. Only documents containing a NUL are affected, and those were losing data.

Tests

EMBEDDED_NUL_JSON in tests/test_base.py plus test_embedded_nul in tests/test_basic_parse.py,
following the existing test_surrogate_pairs pattern so it runs across every backend and adaptor.
No existing test used \u0000 - I grepped tests/ and src/.

Reverting only yajl2_cffi.py and rebuilding:

FAILED tests/test_basic_parse.py::test_embedded_nul[yajl2_cffi-file]
FAILED tests/test_basic_parse.py::test_embedded_nul[yajl2_cffi-iterable]
FAILED tests/test_basic_parse.py::test_embedded_nul[yajl2_cffi-sendable]
6 failed, 18 passed

exactly the six yajl2_cffi variants; python and yajl2_c pass either way. Restored: 24 passed.

I also mutation-checked the length itself - ffi.buffer(val, length + 1) and
ffi.buffer(val, length - 1) both fail those six - so the test pins the length rather than just the
absence of truncation.

Both CI commands pass: pytest -vv gives 2161 passed, 64 skipped, and
pytest --doctest-modules --doctest-ignore-import-errors src gives 1 passed, 1 skipped. Built with
IJSON_EMBED_YAJL=1 and a forced --reinstall-package ijson before every measurement, so none of
the numbers above came from a stale extension.

Unrelated, noting rather than bundling

The python backend disagrees with itself on float overflow with use_float=True: [-1e309] gives
-inf but [1e309] raises UnexpectedSymbol, while yajl2_c/yajl2_cffi raise
IncompleteJSONError for both. Which of the three is intended isn't obvious to me, so I left it out

• happy to open an issue if it's useful.

---

Disclosure: found and prepared with AI assistance (Claude). Every figure above is from a run on this
branch, and the root cause is one I traced and verified by hand.

Summary by Sourcery

Preserve complete JSON strings and map keys containing embedded NUL bytes in the yajl2_cffi backend.

Bug Fixes:

• Preserve embedded NUL bytes in strings and map keys parsed by the yajl2_cffi backend instead of silently truncating them.

Tests:

• Add cross-backend coverage verifying that embedded NUL bytes are retained in JSON keys and string values.

[sourcery-ai]

Reviewer's guide (collapsed on small PRs)

Reviewer's Guide

Fix the YAJL2 CFFI backend’s use of NUL-terminated decoding by honoring YAJL’s explicit string lengths, preventing silent truncation of values and keys; add regression coverage that exercises both cases across the parser matrix.

Sequence diagram for preserving embedded NUL bytes

sequenceDiagram
    participant YAJL
    participant CFFI as yajl2_cffi
    participant Parser
    YAJL->>CFFI: string(val, length)
    CFFI->>CFFI: ffi.buffer(val, length)[:]
    CFFI->>Parser: append_event_to_ctx(string)
    YAJL->>CFFI: map_key(key, length)
    CFFI->>CFFI: ffi.buffer(key, length)[:]
    CFFI->>Parser: append_event_to_ctx(map_key)

Loading

File-Level Changes

Change	Details	Files
Preserve embedded NUL bytes when decoding YAJL string and map-key callbacks.	Replace NUL-terminated ffi.string decoding with length-bounded ffi.buffer decoding for values. Apply the same length-preserving conversion to object keys while leaving number handling unchanged.	src/ijson/backends/yajl2_cffi.py
Add cross-backend regression coverage for NUL-containing keys and string values.	Define a JSON fixture containing embedded NULs in both a map key and string value. Assert parsed events retain the complete strings, including \x00, across adaptors and backends.	tests/test_base.py tests/test_basic_parse.py

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[sourcery-ai]

Hey - I've reviewed your changes and they look great!

Sourcery assessment

Approved.

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}