Parquet: skip reading constant column chunks via statistics - #2266
Open
UnamedRus wants to merge 12 commits into
Open
Parquet: skip reading constant column chunks via statistics#2266UnamedRus wants to merge 12 commits into
UnamedRus wants to merge 12 commits into
Conversation
UnamedRus
force-pushed
the
parquet-v3-constant-column-opt
branch
from
August 26, 2026 08:29
d291a4d to
4f8d600
Compare
UnamedRus
changed the title
UnamedRus
force-pushed
the
parquet-v3-constant-column-opt
branch
from
August 26, 2026 08:44
4f8d600 to
467503c
Compare
The Parquet v3 reader now detects column chunks that provably hold a single value in every row and materializes that value without fetching or decoding any of the chunk's data pages. On wide tables with low-cardinality or defaulted columns this removes both the I/O and the decode cost of those columns. A chunk is treated as constant when its column statistics prove it: - min == max with no nulls: materialize the decoded min value; - null_count == num_values (physically-nullable leaf): the whole chunk is null, materialized as Null for a Nullable output or the column default under input_format_null_as_default (a non-nullable output without null substitution is left to the normal decode path). The result is emitted as a ColumnConst rather than an expanded column: O(1) memory instead of O(rows), and the const-ness propagates downstream so a PREWHERE/WHERE predicate is evaluated from the single value and GROUP BY / aggregation get a const key. An all-null chunk additionally records every row in block_missing_values so input_format_null_as_default still applies. Correctness of min == max is gated on the physical type. Only fixed-width numeric physical types (BOOLEAN, INT32, INT64, INT96, FLOAT, DOUBLE), whose statistics are never truncated, are trusted unconditionally; BYTE_ARRAY, FIXED_LEN_BYTE_ARRAY and any future physical type require the writer's is_min_value_exact / is_max_value_exact flags, because a truncated min/max could make two distinct values compare equal. This is an allowlist, so an unrecognized type fails closed (treated as possibly-truncated). Date, Time and Timestamp logical types ride on the trusted integer physical types; large Decimals stored as FIXED_LEN_BYTE_ARRAY correctly fall to the flag-gated path. Controlled by input_format_parquet_use_constant_column_optimization (default on) and counted by the ParquetConstantColumnChunks profile event. Only flat, top-level primitive columns are eligible (a parquet value maps 1:1 to an output row); arrays, physically-nullable structs, and leaves nested in Tuple/Map/Array outputs are excluded. Signed-off-by: UnamedRus <dtitmoav@gmail.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
UnamedRus
force-pushed
the
parquet-v3-constant-column-opt
branch
from
August 26, 2026 09:00
467503c to
3b7ce77
Compare
`min == max` in column chunk statistics does not prove a single value for floating point columns: parquet.thrift says NaN values must not be written to min/max, and both arrow and our own writer drop NaN when computing them, so a chunk like `[1.0, NaN, 1.0]` gets `min == max == 1.0` with `null_count == 0`. The spec also allows `min = +0.0` to hide `-0.0` rows, which are distinct `GROUP BY` keys in ClickHouse. Remove `FLOAT` and `DOUBLE` from the never-truncated allowlist in `detectConstantColumn`; they cannot fall back to the `is_*_value_exact` path either, because exactness says nothing about NaN. Revisit when `Statistics::nan_count` (parquet-format 2.11) is available. Extend `04811_parquet_constant_column_optimization` with a chunk mixing `1.0` with `nan` and `0.0` with `-0.0`. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: UnamedRus <dtitmoav@gmail.com>
…nk statistics Replace the `is_constant` / `is_all_null` flags with `Reader::ConstantKind`, chosen once per column chunk by the pure `chooseConstantKind` from the chunk statistics and the final output type: - `Const`: `null_count == 0`, `min == max`. `ColumnConst`, data pages not read. A constant equal to the type default stays `Const` (const propagation). - `AllDefault`: `null_count == num_values`. `ColumnSparse` with no non-default rows, data pages not read. Sparse rather than const because `ColumnConst` silently drops inserts, which broke `AddingDefaultsTransform` for `DEFAULT` columns (`File` engine, `INSERT ... FROM INFILE`) when `block_missing_values` was set. `block_missing_values` is now set only under `null_as_default`; a `Nullable` output holds real NULLs that must not be replaced by defaults. - `SparseNulls`: `min == max` plus nulls, null ratio >= the new setting `input_format_parquet_constant_column_sparse_ratio` (default 0.9375). `ColumnSparse` with the value at the non-null rows; the data pages are read for definition levels only, values are neither decompressed (`DATA_PAGE_V2`) nor decoded. - `DenseNulls`: `min == max` plus few nulls. `ColumnNullable` filled with the value plus the decoded null map; same page handling as `SparseNulls`. Fix a domain bug in the constant path: `decodeField` yields the value in the final output type's domain (the statistics converter is chosen from the type hint), not in `decoded_type`, so `TIMESTAMP_MILLIS` read as `DateTime` failed with `Bad get: has UInt64, requested Decimal64`. All kinds are now materialized directly in the final output type without `castColumn`. Reject `FLOAT` / `DOUBLE` before the `is_*_value_exact` fallback: our own writer marks every type exact, which let floats bypass the NaN exclusion. `estimateColumnMemoryBytesPerRow` returns 0 for kinds that skip the pages and O(non-null rows) for `SparseNulls`. New profile event `ParquetConstantColumnChunksWithNulls`. Test `04812_parquet_constant_column_kinds` covers each kind, on/off equality, `null_as_default`, `DEFAULT` columns over a `File` table, `LowCardinality` fallback and the profile events. The tests now take `user_files_path` from `system.server_settings`, since parsing the exception text of a `send_logs_level`-enabled client picks up the echoed log line. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: UnamedRus <dtitmoav@gmail.com>
Input formats may now produce sparse columns (the Parquet reader for chunks that are all null or mostly null), and they survive the Native protocol when the client parses `INSERT ... FORMAT Parquet`. `insertNullAsDefaultIfNeeded` `assert_cast`s a Nullable-typed input column to `ColumnNullable`, which on a `ColumnSparse` raised the logical error `Bad cast from type DB::IColumn const* to DB::ColumnNullable const*` (caught by `04140_parquet_types_roundtrip`). Convert sparse input to a full column first, as `AddingDefaultsTransform` already does with `removeSpecialRepresentations`. Extend `04812_parquet_constant_column_kinds` with an `INSERT ... FORMAT Parquet` through the client under `input_format_null_as_default`. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: UnamedRus <dtitmoav@gmail.com>
…ed with insertRangeFrom Input formats may now return `ColumnConst` / `ColumnSparse` (the Parquet reader for column chunks whose statistics prove a single value or all nulls). Pipelines handle those, but a few places accumulate format output by hand with `insertRangeFrom` into a full column, which `assert_cast`s the source to the concrete column type. `01429_empty_arrow_and_parquet` hit this in `StreamingFormatExecutor::insertChunk` (asynchronous inserts) with the logical error `Bad cast from type DB::ColumnConst to DB::ColumnVector<Int8>`. Convert const/sparse columns to full first there, in `AsynchronousInsertQueue` for pre-parsed blocks, and in the Iceberg equality-delete file reader. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: UnamedRus <dtitmoav@gmail.com>
…volved `decodeField` converts a statistics value according to the requested type so that it can be compared against key conditions; that conversion is not guaranteed to match decoding the page and `castColumn`-ing it to the requested type. `00900_long_parquet` (`FIXED_LEN_BYTE_ARRAY` read as `String` keeps its zero padding in the statistics but not in the decoded column) and `04006_parquet_date_to_enum_insert` (`Date32` read as `Enum8` is accepted for statistics but rejected by the cast with `CANNOT_CONVERT_TYPE`) showed the difference. Require `!needs_cast` in `isConstantColumnCandidate`, so the decoded type is the output type and the domains coincide. A skipped data page cannot be checksum-verified, like pages pruned by the page index; `03408_parquet_checksums` builds a single-row (hence provably constant) column chunk, so it disables the shortcut for its corrupt-page probe. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: UnamedRus <dtitmoav@gmail.com>
…hold On object storage the reader's concurrency came out of how ranges happened to coalesce rather than from anything chosen. Measured on a 271 MB single-file scan (59-column projection, 23 row groups of ~11.8 MB), the default settings produced twelve 22.6 MB reads - fewer than one read per row group - and a 4-thread pool, which left ~3 reads in flight and most of each read's latency on the critical path. Anything that perturbed coalescing swung the result by a factor of two. Changes: * `bytes_per_read_task` now bounds the task. It was compared against the distance from the seed range in each direction independently, so a task could reach `seed + 2 * bytes_per_read_task`; setting it below the natural coalescing width did nothing at all. Compare against the resulting span instead. * A read never spans a row group. `getRangeData` waits for a whole task - there is no partial completion - so a read covering the tail of one row group and the head of the next made the earlier one wait for the later one's bytes, serializing in-order delivery. `Reader` now hands the Prefetcher the row group boundaries; if the metadata is unusable or row groups are not in ascending order, coalescing is left unconstrained. * Read size adapts to how busy the IO pool is. While the pool has spare capacity, smaller reads fill it faster; once it is busy, larger reads amortize the round trip. Hysteresis keeps the size from flapping at the threshold. * The IO pool is sized from the query. `max_download_threads` (default 4) was picked for the URL engine and is usually too small here, so decoding threads end up running reads themselves or waiting for them. * New settings, all defaulting to the previous behaviour except the pool size: `input_format_parquet_bytes_per_read_task`, `input_format_parquet_max_io_threads`, and `input_format_parquet_max_active_files`, which bounds how many files read ahead at once so each active one runs at a useful depth instead of every file crawling. A file without a slot still reads the row group it must deliver next, so a query cannot stall on it. * Profile events to make this visible without a profiler: `ParquetReadTasks`, `ParquetReadTaskBytes` and `ParquetPrefetchStarvation`. Dividing read tasks by `ParquetReadRowGroups` gives reads per row group; below 1 means reads span row groups. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: UnamedRus <dtitmoav@gmail.com>
UnamedRus
force-pushed
the
parquet-v3-constant-column-opt
branch
from
August 27, 2026 07:43
b7f6cd2 to
36783c7
Compare
Fixes the 02995_new_settings_history failure in Fast test: every new setting has to appear in SettingsChangesHistory.cpp. All three default to 0, so the settings themselves change no behavior; the note on max_io_threads records that its derived value is larger than the previous hard-coded max_download_threads. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: UnamedRus <dtitmoav@gmail.com>
v3 is the default reader now, matching 6ba63a1 which already removed it from the other reader setting descriptions. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: UnamedRus <dtitmoav@gmail.com>
Keep only what the code cannot say: why the task budget is compared against the span rather than per direction, why a read must not cross a row group, the writer quirk around dictionary_page_offset, why Task::owner exists, and the exactly-once accounting of tasks_in_flight. Drop the restatements. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: UnamedRus <dtitmoav@gmail.com>
Two changes to how data-page reads get issued. Issuance is one unit of work per subgroup instead of one task per column. The work already ran on the scheduling thread (scheduleTask) and the task body was empty, so a task per column bought a thread-pool round trip each and an N-wide barrier before decoding could start. New setting `input_format_parquet_read_ahead_subgroups` (default 0, previous behavior) issues the next subgroup's reads while the current one decodes. Until now a subgroup's reads were issued only after its predecessor was decoded and delivered, so the storage's response time was paid again on every subgroup instead of overlapping with work. Read-ahead skips filtered-out subgroups and claims a subgroup with a CAS, so losing the race to the normal path is harmless. That CAS exposed a latent defect in the sequential-admission loop: it took the expected value from its own load, so it succeeded whatever the current stage was and guarded nothing. It only worked because nothing else moved a subgroup out of NotStarted. It now compares against NotStarted explicitly, which is required for read-ahead not to admit a subgroup twice. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: UnamedRus <dtitmoav@gmail.com>
…instead of admitting the next subgroup early The previous read-ahead admitted subgroup N+1 into the stage machine while N was still decoding. `finishRowSubgroupStage` then issued N+1's `ColumnData` tasks as soon as its `ColumnDataPrefetch` finished, so two subgroups of one row group decoded concurrently. `Reader::ColumnChunk` has a single sequential page cursor (`page`, `next_page_offset`, `data_pages_idx`) and a lazily initialised dictionary shared by all subgroups of the row group, so this corrupted decoding; it also collided on the one-slot-per-(stage, row group) task queue in `setTasksToSchedule` and could regress `read_ptr`, calling `clearColumnChunk` while a subgroup was still being decoded. Now the `ColumnDataPrefetch` task of subgroup N also issues the first-step data-page reads of subgroups N+1..N+k. Nothing about admission or decode order changes: the next subgroup is admitted by the normal path once N's main step is done, and skips its own `ColumnDataPrefetch` stage (`reads_issued_ahead`) because there is nothing left to issue. This is idempotent by construction: `determinePagesToPrefetch` advances `data_pages_prefetch_idx` past the pages it handed out, and `startPrefetch` skips handles that already have a task. Own reads are issued before read-ahead reads so N keeps priority in coalescing. `input_format_parquet_read_ahead_subgroups` values above 1 now work. Read-ahead bytes are charged to a new accounting-only stage `ColumnDataReadAhead` with its own share of the prefetch budget (`input_format_parquet_read_ahead_memory_fraction`, default 0.25 of the prefetch share), so read-ahead cannot eat the `ColumnDataPrefetch` budget that keeps other row groups moving. `flushMemoryUsageDiff` skips scheduling on it. New profile event `ParquetReadAheadSubgroups`. Test `04813_parquet_read_ahead_subgroups` reads a file with many subgroups per row group and several pages per subgroup under a low decode watermark, with read-ahead 0/1/3, filtered and single-threaded, and checks the profile event. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: UnamedRus <dtitmoav@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Parquet reader (v3) now uses column chunk statistics to materialize chunks whose content they prove, without decoding the values from the data pages. The representation is chosen once per chunk, purely from the statistics and the output type (
Reader::chooseConstantKind):null_count == 0,min == maxConstColumnConst— const-ness propagates toWHERE/GROUP BYnull_count == num_valuesAllDefaultColumnSparsewith no non-default rows (NULL, or the type default underinput_format_null_as_default)min == max, nulls ≥input_format_parquet_constant_column_sparse_ratioSparseNullsColumnSparse: the value at the non-null rows, O(non-null rows) memoryDATA_PAGE_V2) nor decodedmin == max, fewer nullsDenseNullsColumnNullablefilled with the value plus the decoded null mapAll-null chunks use a sparse column rather than
ColumnConstbecause a sparse column stays writable:ColumnConst::insertis a no-op, which would silently drop the defaultsAddingDefaultsTransformmixes in forDEFAULTcolumns (Fileengine,INSERT ... FROM INFILE). A non-null constant equal to the type default is stillConst, since const propagation beats sparse there and there is no null map to mix.Correctness of
min == maxis gated on the physical type. Only fixed-width integer physical types (BOOLEAN,INT32,INT64,INT96), whose statistics are never truncated, are trusted unconditionally.FLOATandDOUBLEare excluded entirely: NaN values are not written to min/max (per parquet.thrift, arrow and our writer), so[1.0, NaN, 1.0]hasmin == max == 1.0, andmin = +0.0may hide-0.0rows; no available statistic proves the absence of NaN (nan_countneeds parquet-format 2.11).BYTE_ARRAY,FIXED_LEN_BYTE_ARRAYand any future physical type require the writer'sis_min_value_exact/is_max_value_exactflags, because a truncated min/max could make two distinct values compare equal. Date, Time and Timestamp logical types ride on the trusted integer physical types; large Decimals stored asFIXED_LEN_BYTE_ARRAYfall to the flag-gated path. A physically nullable leaf must statenull_count; aREQUIREDleaf cannot hold nulls.The shortcut applies only when the requested type equals the decoded type (no cast):
decodeFieldconversions are chosen from the requested type for statistics comparison and are not guaranteed to match decode-then-castColumn(e.g.FIXED_LEN_BYTE_ARRAYread asStringkeeps its zero padding in statistics;Date32read asEnum8is accepted for statistics but rejected by the cast). Skipped data pages are, like pages pruned by the page index, not checksum-verified;03408_parquet_checksumsdisables the shortcut for its corrupt-page probe.Consumers that accumulate format output by hand with
insertRangeFrom(StreamingFormatExecutorfor asynchronous inserts,AsynchronousInsertQueue, the Iceberg equality-delete reader) andinsertNullAsDefaultIfNeedednow materialize const/sparse columns first; pipelines already handle them.Controlled by
input_format_parquet_use_constant_column_optimization(default on) andinput_format_parquet_constant_column_sparse_ratio(default 0.9375, same asratio_of_defaults_for_sparse_serialization;1disables sparse materialization of mixed chunks). Counted by theParquetConstantColumnChunks(pages skipped) andParquetConstantColumnChunksWithNulls(values skipped) profile events. Only flat, top-level primitive columns are eligible (a parquet value maps 1:1 to an output row); arrays, physically-nullable structs, and leaves nested in Tuple/Map/Array outputs are excluded. Types that cannot be inside a sparse column (LowCardinality) fall back toDenseNulls/ normal decode.Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
...
Documentation entry for user-facing changes
...
CI/CD Options
Exclude tests:
Regression jobs to run: