Repository navigation
Plain-text reader: reject schema entries outside the observation - #413
Open
anabobillo3 wants to merge 1 commit into
Open
anabobillo3 wants to merge 1 commit into
anabobillo3 wants to merge 1 commit into
Conversation
readstat_parse_txt sizes line_lens by rows_per_observation and indexes it by each entry's row, and copies len bytes at col out of a line buffer sized from col + len. None of these were checked, so a schema file could make the reader access memory out of bounds: - A Stata dictionary that uses _line(n) or _newline past the default single line without _lines(), or _line(0) or _lines(0). The dictionary parser never raises rows_per_observation the way submit_columns does for SAS and SPSS commands. - SAS INPUT #0 and SPSS DATA LIST /0: the row wraps to UINT32_MAX, so row + 1 in submit_columns wraps to 0 and rows_per_observation stays 1. - A column of 0 (_column(0), SAS 0-1): col + len wraps in 32 bits, and the field is copied from far past the end of the line buffer. Check every entry at the start of readstat_parse_txt, so schemas from all three parsers and from API callers are covered, and name the offending variable through the error handler. Reversed SAS column ranges (3-1), which were silently skipped, are rejected by the same check. Also reject an observation of zero lines, or of more lines than the int loop counters hold, and keep the line_lens and line buffer sizes from wrapping on 32-bit builds. test_readstat now runs Stata, SAS and SPSS schemas through readstat_parse_txt, both valid ones and ones it must reject.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
readstat_parse_txttrusts the row, column and width of every schema entry.line_lensholdsrows_per_observationentries but is indexed byentry->row, and each field is copied out of a line buffer sized fromcol + len. Nothing checks these against each other, so a schema file can make the reader access memory out of bounds:_line(n)or_newlinepast the default single line without_lines(), or_line(0)or_lines(0). Unlikesubmit_columnsfor SAS and SPSS commands, the dictionary parser never raisesrows_per_observation, soline_lens[entry->row]atreadstat_txt_read.c:203runs past the array;INPUT #0and SPSSDATA LIST /0: the row wraps toUINT32_MAX, sorow + 1insubmit_columnswraps to 0 androws_per_observationstays 1;_column(0), SAS0-1):col + lenwraps in 32 bits, so the line buffer is sized too small and the field is copied from far past its end.Fix
txt_check_schema()runs at the start ofreadstat_parse_txt, before the schema is used, so schemas from all three parsers and from API callers are covered. It returnsREADSTAT_ERROR_PARSE, with a message naming the variable through the error handler, for:rows_per_observation, or whosecol + lendoes not fit in 32 bits;intloop counters hold.Two size computations also no longer wrap on 32-bit builds:
line_lensis allocated withcalloc, and adding 2 for CRLF to the line buffer length is checked first.Behaviour changes, all for schemas that could not be read correctly before:
_line()past_lines()(or past the default of one line) now fails with e.g.Variable y lies outside the bounds of an observation (line 2 of 1, column 1, width 1), instead of overflowingline_lensand reading each physical line as its own observation without the later lines' variables. Stata infers_lines()when it is omitted; the dictionary parser could do the same, assubmit_columnsdoes for SAS and SPSS. I can send that as a separate change if you'd like it.3-1), which were silently skipped, are now rejected by the same check.Tests
test_readstatnow starts withtest_parse_txt_schema_bounds()(src/test/test_txt.c). It parses Stata, SAS and SPSS schemas and runs each throughreadstat_parse_txt. The valid ones (multi-line, delimited and single-line) must read their two values. Each case above must be rejected with an error message. The file is added toMakefile.amand to the VS17 test project.make checkpasses with gcc and clang, and with gcc and-fsanitize=address,undefined. With the new test and the old reader,test_readstatfails with an AddressSanitizer heap-buffer-overflow atreadstat_txt_read.c:203.