Skip to content

fix: preserve sorts for non-monotonic temporal casts - #25526

Merged
jayzhan211 merged 2 commits into
apache:mainfrom
Toby1009:fix/temporal-cast-ordering
Sep 20, 2026
Merged

jayzhan211 merged 2 commits into
apache:mainfrom
Toby1009:fix/temporal-cast-ordering

Conversation

@Toby1009

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

ORDER BY CAST(ts AS TIME) can return incorrectly ordered results when an
already ordered timestamp input crosses midnight. A named timezone rollback
can similarly make a timestamp-to-date conversion non-monotonic. In both
cases, DataFusion may incorrectly reuse the input ordering and remove the sort
required by the cast result.

What changes are included in this PR?

Replace the broad temporal-to-temporal ordering rule with an explicit list of
order-preserving temporal casts. Keep the existing lossless cast check
separate because it provides the stronger guarantees used for strict ordering
and statistics propagation.

Cast expressions that may turn conversion failures into NULL no longer
propagate ordering unless the conversion is known to be lossless, since a new
NULL can violate NULLS FIRST or NULLS LAST.

What is the testing strategy for this PR?

The unit tests cover safe and unsafe temporal cast combinations, both sort
directions, both NULL placements, unknown input types, and NULL-on-failure
casts. The cases added to cast.slt cover timestamp-to-time conversion across
midnight, timestamp-to-date conversion across a named timezone rollback, and
verify that reducing timestamp precision still avoids an unnecessary sort.

Are there any user-facing changes?

Queries that order by a non-monotonic temporal cast now retain the required
sort and return correctly ordered results. There are no public API changes.

@github-actions github-actions Bot added physical-expr Changes to the physical-expr crates sqllogictest SQL Logic Tests (.slt) labels Sep 20, 2026
@codecov-commenter

codecov-commenter commented Sep 20, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.39%. Comparing base (4e603e1) to head (da3118a).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff            @@
##             main   #25526    +/-   ##
========================================
  Coverage   82.39%   82.39%            
========================================
  Files        1138     1138            
  Lines      434722   434863   +141     
  Branches   434722   434863   +141     
========================================
+ Hits       358186   358316   +130     
- Misses      54853    54862     +9     
- Partials    21683    21685     +2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@Toby1009
Toby1009 force-pushed the fix/temporal-cast-ordering branch from a3c7589 to da300c9 Compare September 20, 2026 07:19

@jayzhan211 jayzhan211 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @Toby1009 , a small suggestion

match (source_type, target_type) {
(Date32 | Date64, Date32 | Date64)
| (Date32 | Date64, Timestamp(_, None))
| (Timestamp(_, None), Date32) => true,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

UTC / fixed-offset timezones have no transitions, so Timestamp(_, Some("UTC" | "+08:00")) → Date32 and Timestamp(_, None) → Timestamp(_, Some(fixed)) are monotonic, but now return Unordered. Extra SortExec that the old rule elided:

EXPLAIN SELECT CAST(ts AS DATE) AS d
FROM (
  SELECT arrow_cast(column1, 'Timestamp(Second, Some("UTC"))') AS ts
  FROM (VALUES (562129259::bigint), (562129260::bigint))
  ORDER BY ts LIMIT 2
)
ORDER BY d;
-- SortExec: expr=[d@0 ASC NULLS LAST]   <-- unnecessary
--   ProjectionExec: expr=[CAST(ts@0 AS Date32) as d]
--     SortExec: TopK(fetch=2), expr=[ts@0 ASC NULLS LAST]

Fix (fine as a follow-up):

/// UTC and fixed offsets have no transitions, so local date/time is
/// monotonic in the epoch value.
fn is_fixed_offset(tz: &str) -> bool {
    tz == "UTC" || tz.starts_with(['+', '-'])
}
         | (Timestamp(_, None), Date32) => true,
+        (Timestamp(_, Some(tz)), Date32) => is_fixed_offset(tz),
@@
-            from_tz.is_some() || to_tz.is_none()
+            from_tz.is_some() || to_tz.as_deref().is_none_or(is_fixed_offset)

Add (Timestamp(Second, Some("UTC")), Date32, true), (Timestamp(Second, Some("+08:00")), Date32, true), and (Timestamp(Second, None), Timestamp(Second, Some("+08:00")), true) to test_temporal_cast_ordering.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, applied this suggestion. UTC and fixed-offset timezones now preserve ordering for timestamp-to-Date32 and naive-to-fixed-offset timestamp casts. I added the three requested unit cases and verified the EXPLAIN example no longer contains the outer SortExec. The full required local test suite also passes.

@Toby1009
Toby1009 force-pushed the fix/temporal-cast-ordering branch from da300c9 to 06c0cab Compare September 20, 2026 12:50
@jayzhan211
jayzhan211 added this pull request to the merge queue Sep 20, 2026
@jayzhan211

Copy link
Copy Markdown
Contributor

Thanks @Toby1009

Merged via the queue into apache:main with commit 918ab2c Sep 20, 2026
41 checks passed
@Toby1009
Toby1009 deleted the fix/temporal-cast-ordering branch September 20, 2026 14:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

physical-expr Changes to the physical-expr crates sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ORDER BY CAST(ts AS TIME) can return incorrectly ordered results across midnight

3 participants