feat: Add TimestampNTZType support for unix_timestamp#4039
Open
parthchandra wants to merge 4 commits intoapache:mainfrom
Open
feat: Add TimestampNTZType support for unix_timestamp#4039parthchandra wants to merge 4 commits intoapache:mainfrom
parthchandra wants to merge 4 commits intoapache:mainfrom
Conversation
parthchandra
force-pushed
the
complete_ntz
branch
from
2775b5e to
e2df118
Compare
3 tasks
andygrove
reviewed
Apr 22, 2026
| // without timezone, so no conversion should be applied. | ||
| expr.children.head.dataType match { | ||
| case TimestampType | DateType => true | ||
| case dt if dt.typeName == "timestamp_ntz" => true |
Member
There was a problem hiding this comment.
We can just just match on TimestampNTZType rather than use typeName. I know we had to do this in the past, but Spark 3.4 and later have TimestampNTZType
Contributor
Author
There was a problem hiding this comment.
Good suggestion. Changed.
andygrove
reviewed
Apr 23, 2026
| // calculate DST transitions, which can result in incorrect offsets. See the compatibility | ||
| // guide for more information. | ||
| #[inline] | ||
| fn as_micros_from_unix_epoch_utc(dt: Option<DateTime<Tz>>) -> i64 { |
Member
There was a problem hiding this comment.
I think the changes here may introduce a regression in DST handling.
Could we merge #4040 first to add new regression test and make sure they still pass with this PR?
Contributor
Author
There was a problem hiding this comment.
#4040 is merged. Rebased to make sure tests all pass
parthchandra
force-pushed
the
complete_ntz
branch
from
830874b to
85c31e8
Compare
Contributor
Author
|
@andygrove rebased, updated, and ready for review. (Original PR: #3253) |
parthchandra
force-pushed
the
complete_ntz
branch
from
85c31e8 to
aebca12
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #378
Summary
Add TimestampNTZ support for temporal expressions and fix DST offset calculation in
date_trunc.Key changes:
unix_timestamp(no timezone conversion — simply divides microseconds by 1,000,000)as_micros_from_unix_epoch_utcsodate_truncproduces correct results when truncation crosses DST boundaries (e.g., December/PST to October/PDT)hour/minute/second,date_trunc, anddate_formatNot included in this PR: TimestampNTZ cast support (to/from String, Date, Timestamp, Long) — that will come in a separate PR.
Test plan
mvn test -pl spark -Pspark-4.0 -Dsuites="org.apache.comet.CometTemporalExpressionSuite"cargo check -p datafusion-comet-spark-expr