fix: preserve sorts for non-monotonic temporal casts - #25526
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #25526 +/- ##
========================================
Coverage 82.39% 82.39%
========================================
Files 1138 1138
Lines 434722 434863 +141
Branches 434722 434863 +141
========================================
+ Hits 358186 358316 +130
- Misses 54853 54862 +9
- Partials 21683 21685 +2 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
a3c7589 to
da300c9
Compare
jayzhan211
left a comment
There was a problem hiding this comment.
Thanks @Toby1009 , a small suggestion
| match (source_type, target_type) { | ||
| (Date32 | Date64, Date32 | Date64) | ||
| | (Date32 | Date64, Timestamp(_, None)) | ||
| | (Timestamp(_, None), Date32) => true, |
There was a problem hiding this comment.
UTC / fixed-offset timezones have no transitions, so Timestamp(_, Some("UTC" | "+08:00")) → Date32 and Timestamp(_, None) → Timestamp(_, Some(fixed)) are monotonic, but now return Unordered. Extra SortExec that the old rule elided:
EXPLAIN SELECT CAST(ts AS DATE) AS d
FROM (
SELECT arrow_cast(column1, 'Timestamp(Second, Some("UTC"))') AS ts
FROM (VALUES (562129259::bigint), (562129260::bigint))
ORDER BY ts LIMIT 2
)
ORDER BY d;
-- SortExec: expr=[d@0 ASC NULLS LAST] <-- unnecessary
-- ProjectionExec: expr=[CAST(ts@0 AS Date32) as d]
-- SortExec: TopK(fetch=2), expr=[ts@0 ASC NULLS LAST]Fix (fine as a follow-up):
/// UTC and fixed offsets have no transitions, so local date/time is
/// monotonic in the epoch value.
fn is_fixed_offset(tz: &str) -> bool {
tz == "UTC" || tz.starts_with(['+', '-'])
} | (Timestamp(_, None), Date32) => true,
+ (Timestamp(_, Some(tz)), Date32) => is_fixed_offset(tz),
@@
- from_tz.is_some() || to_tz.is_none()
+ from_tz.is_some() || to_tz.as_deref().is_none_or(is_fixed_offset)Add (Timestamp(Second, Some("UTC")), Date32, true), (Timestamp(Second, Some("+08:00")), Date32, true), and (Timestamp(Second, None), Timestamp(Second, Some("+08:00")), true) to test_temporal_cast_ordering.
There was a problem hiding this comment.
Thanks, applied this suggestion. UTC and fixed-offset timezones now preserve ordering for timestamp-to-Date32 and naive-to-fixed-offset timestamp casts. I added the three requested unit cases and verified the EXPLAIN example no longer contains the outer SortExec. The full required local test suite also passes.
da300c9 to
06c0cab
Compare
|
Thanks @Toby1009 |
Which issue does this PR close?
ORDER BY CAST(ts AS TIME)can return incorrectly ordered results across midnight #25506.Rationale for this change
ORDER BY CAST(ts AS TIME)can return incorrectly ordered results when analready ordered timestamp input crosses midnight. A named timezone rollback
can similarly make a timestamp-to-date conversion non-monotonic. In both
cases, DataFusion may incorrectly reuse the input ordering and remove the sort
required by the cast result.
What changes are included in this PR?
Replace the broad temporal-to-temporal ordering rule with an explicit list of
order-preserving temporal casts. Keep the existing lossless cast check
separate because it provides the stronger guarantees used for strict ordering
and statistics propagation.
Cast expressions that may turn conversion failures into NULL no longer
propagate ordering unless the conversion is known to be lossless, since a new
NULL can violate
NULLS FIRSTorNULLS LAST.What is the testing strategy for this PR?
The unit tests cover safe and unsafe temporal cast combinations, both sort
directions, both NULL placements, unknown input types, and NULL-on-failure
casts. The cases added to
cast.sltcover timestamp-to-time conversion acrossmidnight, timestamp-to-date conversion across a named timezone rollback, and
verify that reducing timestamp precision still avoids an unnecessary sort.
Are there any user-facing changes?
Queries that order by a non-monotonic temporal cast now retain the required
sort and return correctly ordered results. There are no public API changes.