Skip to content

Plain-text reader: reject schema entries outside the observation - #413

Open
anabobillo3 wants to merge 1 commit into
WizardMac:devfrom
anabobillo3:fix/txt-row-bounds
Open

anabobillo3 wants to merge 1 commit into
WizardMac:devfrom
anabobillo3:fix/txt-row-bounds

Conversation

@anabobillo3

Copy link
Copy Markdown

readstat_parse_txt trusts the row, column and width of every schema entry. line_lens holds rows_per_observation entries but is indexed by entry->row, and each field is copied out of a line buffer sized from col + len. Nothing checks these against each other, so a schema file can make the reader access memory out of bounds:

  • a Stata dictionary that uses _line(n) or _newline past the default single line without _lines(), or _line(0) or _lines(0). Unlike submit_columns for SAS and SPSS commands, the dictionary parser never raises rows_per_observation, so line_lens[entry->row] at readstat_txt_read.c:203 runs past the array;
  • SAS INPUT #0 and SPSS DATA LIST /0: the row wraps to UINT32_MAX, so row + 1 in submit_columns wraps to 0 and rows_per_observation stays 1;
  • a column of 0 (_column(0), SAS 0-1): col + len wraps in 32 bits, so the line buffer is sized too small and the field is copied from far past its end.

Fix

txt_check_schema() runs at the start of readstat_parse_txt, before the schema is used, so schemas from all three parsers and from API callers are covered. It returns READSTAT_ERROR_PARSE, with a message naming the variable through the error handler, for:

  • an entry whose row is not below rows_per_observation, or whose col + len does not fit in 32 bits;
  • an observation of zero lines, or of more lines than the int loop counters hold.

Two size computations also no longer wrap on 32-bit builds: line_lens is allocated with calloc, and adding 2 for CRLF to the line buffer length is checked first.

Behaviour changes, all for schemas that could not be read correctly before:

  • A dictionary that uses _line() past _lines() (or past the default of one line) now fails with e.g. Variable y lies outside the bounds of an observation (line 2 of 1, column 1, width 1), instead of overflowing line_lens and reading each physical line as its own observation without the later lines' variables. Stata infers _lines() when it is omitted; the dictionary parser could do the same, as submit_columns does for SAS and SPSS. I can send that as a separate change if you'd like it.
  • Reversed SAS column ranges (3-1), which were silently skipped, are now rejected by the same check.

Tests

test_readstat now starts with test_parse_txt_schema_bounds() (src/test/test_txt.c). It parses Stata, SAS and SPSS schemas and runs each through readstat_parse_txt. The valid ones (multi-line, delimited and single-line) must read their two values. Each case above must be rejected with an error message. The file is added to Makefile.am and to the VS17 test project.

make check passes with gcc and clang, and with gcc and -fsanitize=address,undefined. With the new test and the old reader, test_readstat fails with an AddressSanitizer heap-buffer-overflow at readstat_txt_read.c:203.

readstat_parse_txt sizes line_lens by rows_per_observation and indexes it
by each entry's row, and copies len bytes at col out of a line buffer
sized from col + len. None of these were checked, so a schema file could
make the reader access memory out of bounds:

- A Stata dictionary that uses _line(n) or _newline past the default
  single line without _lines(), or _line(0) or _lines(0). The dictionary
  parser never raises rows_per_observation the way submit_columns does
  for SAS and SPSS commands.
- SAS INPUT #0 and SPSS DATA LIST /0: the row wraps to UINT32_MAX, so
  row + 1 in submit_columns wraps to 0 and rows_per_observation stays 1.
- A column of 0 (_column(0), SAS 0-1): col + len wraps in 32 bits, and
  the field is copied from far past the end of the line buffer.

Check every entry at the start of readstat_parse_txt, so schemas from all
three parsers and from API callers are covered, and name the offending
variable through the error handler. Reversed SAS column ranges (3-1),
which were silently skipped, are rejected by the same check. Also reject
an observation of zero lines, or of more lines than the int loop
counters hold, and keep the line_lens and line buffer sizes from wrapping
on 32-bit builds.

test_readstat now runs Stata, SAS and SPSS schemas through
readstat_parse_txt, both valid ones and ones it must reject.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant