Skip to content

Use int64_t for row limit/offset on all platforms - #411

Open
hadley wants to merge 1 commit into
WizardMac:devfrom
hadley:int64-row-offset
Open

hadley wants to merge 1 commit into
WizardMac:devfrom
hadley:int64-row-offset

Conversation

@hadley

@hadley hadley commented Oct 19, 2026

Copy link
Copy Markdown
Contributor

🤖 Written by AI 🤖

readstat_set_row_limit() and readstat_set_row_offset() take long, which is 64-bit on Unix but 32-bit on Windows (LLP64). On Windows, this caps row offsets at ~2.15 billion, which blocks chunked reading of very large files (see tidyverse/haven#194 for a real-world report).

This PR widens the row limit/offset to int64_t throughout:

  • readstat_parser_t fields and the two setter signatures in readstat.h / readstat_parser.c
  • The internal context fields in the sas7bdat (uint32_t), xport (int), sav (int), and por (int) readers, so the full range survives the handoff from parser to reader
  • The row counters in the txt reader that are compared against row_limit (int k could overflow for limits above INT_MAX)
  • One snprintf format updated to PRId64 accordingly

The Stata reader already used int64_t for these fields, so this also makes the readers consistent.

Note this changes the public API signatures of the two setters. On Windows, callers passing long will still compile (implicit widening), but the ABI changes.

make check passes (4/4, including the test_readstat corpus suite).

`long` is 32-bit on Windows (LLP64), so `readstat_set_row_limit()` and
`readstat_set_row_offset()` overflow for files with more than ~2.15
billion rows there. Widen the parser fields and setter arguments to
`int64_t`, along with the internal `row_limit`/`row_offset` context
fields in the sas7bdat, xport, sav, and por readers (previously
`uint32_t`/`int`) so the full range is preserved end-to-end. The Stata
reader already used `int64_t`.
@hadley

hadley commented Oct 19, 2026

Copy link
Copy Markdown
Contributor Author

🤖 Written by AI 🤖

One deliberate scope boundary: the per-row counters that are compared against row_limit (current_row in sav, obs_count in por, parsed_row_count in xport/sas7bdat) are still int/uint32_t. That is safe because they only count emitted rows — skipped rows are handled by seeking (sav) or decrementing row_offset before counting begins — and a single parse call is already capped at INT_MAX emitted rows by the public value handler signature (int obs_index). Widening that handler signature would be a much larger API break and is out of scope here; this PR only fixes the offset/limit truncation, which is what blocks chunked reading of >2^31-row files on Windows.

@shun2wang shun2wang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This make sense.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants