storage: Flesh out testing for string primary key probes - #38030
Draft
peterdukelarsen wants to merge 23 commits into
Draft
storage: Flesh out testing for string primary key probes#38030peterdukelarsen wants to merge 23 commits into
peterdukelarsen wants to merge 23 commits into
Conversation
Add a probe module exposing KeyProber over a string key column of a table: optimizer row count estimates for half-open key ranges via EXPLAIN index dives, and first/next key prefix discovery, with all key ordering done server-side under the column's own collation. Includes a LIKE-pattern escaping helper so prefixes containing wildcard characters match literally. Covered by unit tests plus a live-MySQL test (opt-in via MZ_TEST_MYSQL_URL) that exercises EXPLAIN estimates, LIKE escaping against metacharacter keys, and range bounds on both probes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add the MySql service to the cargo-test composition and hand its URL to nextest as MZ_TEST_MYSQL_URL, following the pattern POSTGRES_URL and METADATA_BACKEND_URL already use. Tests gated on the variable skip locally when it is unset but panic when CI is set, matching the timestamp oracle's tripwire so a wiring regression cannot silently retire them. Tests sharing the server run concurrently under nextest, so each must confine itself to a uniquely named scratch database it creates itself. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Cover ULID keys (long shared timestamp prefix), UUID keys, keys built from LIKE metacharacters, case-insensitive vs binary collations, and stale table statistics. The stale statistics test pins down that range estimates come from index dives on the real B-tree, so they stay accurate even while information_schema.tables reports 0 rows. The metacharacter test asserts the partition property of a prefix walk, since a key shorter than the prefix length subsumes longer keys sharing it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A key shorter than the prefix length names an exact key, and the LIKE anchor skipped every key extending it, leaving such ranges unsplittable at any depth. next_prefix now steps just past the exact key so its extensions become prefixes of their own. Tests: pin charset and collation explicitly everywhere, share table setup through helpers, and reorganize so behavior-explaining tests lead and helpers trail. New coverage: basic and case-insensitive traversal, multibyte keys, EXPLAIN estimate sizing, and sargability asserted via session handler counters, with a self-check that a non-sargable query trips them. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Move the broader live-MySQL test suite (case sensitivity, wildcards, multibyte data, ULID/UUID keys, LIKE metacharacters, collations, stale statistics, sargability) to a stacked follow-up branch so this PR stays focused on the probing interface itself.
Nothing outside the probe module uses the LIKE escaping, keeping it private leaves the NO_BACKSLASH_ESCAPES caveat an internal note rather than a public contract.
peterdukelarsen
force-pushed
the
pl/mysql-key-probes-tests
branch
from
August 4, 2026 00:41
1d1b417 to
253eb1d
Compare
estimate_range_rows returns the optimizer estimate as an Option instead of defaulting to 0, and its doc records observed accuracy on a large static table. The next-prefix LIKE anchor declares an explicit ESCAPE so the pattern no longer depends on the sql_mode default escape character, making NO_BACKSLASH_ESCAPES sessions behave identically. Comments document the charset conversion reasoning and the connection charset assumption, and range end-bound SQL assembly is deduplicated into a helper.
peterdukelarsen
force-pushed
the
pl/mysql-key-probes-tests
branch
from
August 4, 2026 21:49
7915d9a to
6c072b6
Compare
Take a concrete mysql_async::Conn instead of a Queryable generic. Replace the inclusive lower bound with an exclusive one throughout, a key exactly equal to a bound is skipped as a split point and its extensions surface through the exclusive bound on re-splits. Decompose the next-prefix probe into max_key_with_prefix plus prefix_of_first_key_in_range, drop the client-side character counting entirely, and rename the probes to say what they return. Privatize explain_row_estimate, estimates are reached through KeyProber.
The anchor and seek probes read separate snapshots outside a transaction, and an insert matching the prefix between them makes the walk see the same prefix again instead of advancing.
Both prefix probes take max_prefix_length, replacing the mismatched prefix_len and len, and docs say "up to" to match the shorter-key behavior.
peterdukelarsen
force-pushed
the
pl/mysql-key-probes-tests
branch
from
August 5, 2026 20:16
6c072b6 to
8bbcb53
Compare
upper_bound becomes upper_bound_exclusive to match lower_bound_exclusive, and estimate_range_rows takes the same names. Docs state that both bounds are exclusive.
peterdukelarsen
force-pushed
the
pl/mysql-key-probes-tests
branch
from
August 5, 2026 20:23
8bbcb53 to
c7acbf8
Compare
first becomes first_key_in_range and next becomes first_row_not_matching_prefix, the probe method names minus the shared prefix_of_ stem, short enough that every assertion stays a one-liner.
max_key_with_prefix splices range_filter output after its LIKE condition instead of using a bespoke append helper. The TRUE fallback makes that composition uniform, the clause stays a valid predicate after WHERE or AND even with no bounds.
peterdukelarsen
force-pushed
the
pl/mysql-key-probes-tests
branch
from
August 5, 2026 20:35
c7acbf8 to
3872dc7
Compare
The test wrappers take the full probe method names, trading one-line assertions for grep-identical naming, and the range_filter doc is condensed.
peterdukelarsen
force-pushed
the
pl/mysql-key-probes-tests
branch
from
August 5, 2026 20:49
3872dc7 to
301be6a
Compare
estimate_range_rows accepts an open lower bound like the prefix probes, and callers pass None for the start of the key space instead of an empty string sentinel.
peterdukelarsen
force-pushed
the
pl/mysql-key-probes-tests
branch
3 times, most recently
from
August 5, 2026 22:36
50812a8 to
061a44c
Compare
query_string mapped values that fail UTF-8 decoding to None, silently ending prefix walks early. It now returns MySqlError::NonUtf8KeyValue so callers can log the condition and fall back explicitly instead of mistaking it for an exhausted range.
Restores the broader live test suite on top of the probe interface PR: case-insensitive traversal, LIKE wildcard and metacharacter data, multibyte and emoji keys, ULID and UUID primary keys, collation behavior, stale table statistics, and sargability via Handler_read session counters.
A latin1 table exercises the column-to-connection charset conversion, including the exact-key step on a key that is one character but two UTF-8 bytes. A binary key column pins the defensive behavior for invalid UTF-8: decode failures read as "no next prefix" and end the walk early instead of erroring. setup_table now derives the charset from the collation name instead of hardcoding utf8mb4.
peterdukelarsen
force-pushed
the
pl/mysql-key-probes-tests
branch
from
August 6, 2026 14:12
061a44c to
3e56f33
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Split out some testing from #38022 to keep that PR manageable.
Description
Adds testing for more diverse character types, collations, and for asserting searches use th B-tree.
Verification