Skip to content

Commit beecc2d

Browse files
authored
feat: Cast string to timestamp_ntz (#4034)
* feat: cast string to timestamp_ntz
1 parent a64c6ab commit beecc2d

7 files changed

Lines changed: 509 additions & 36 deletions

File tree

docs/source/user-guide/latest/compatibility/expressions/cast.md

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -58,6 +58,14 @@ suffixes (e.g. `Europe/Moscow`), and the full Spark timestamp year range
5858
(-290308 to 294247). Note that `CAST(string AS DATE)` is only compatible for years between
5959
262143 BC and 262142 AD due to an underlying library limitation.
6060

61+
## String to TimestampNTZ
62+
63+
Comet's native `CAST(string AS TIMESTAMP_NTZ)` implementation matches Apache Spark's behavior.
64+
Unlike `CAST(string AS TIMESTAMP)`, this cast is timezone-independent: any timezone offset in
65+
the input string (e.g. `+08:00`, `Z`, `UTC`) is silently discarded, and the local date-time
66+
components are preserved as-is. Time-only strings (e.g. `T12:34:56`, `12:34`) produce `NULL`.
67+
The result is always a wall-clock timestamp with no timezone conversion or DST adjustment.
68+
6169
## Decimal with Negative Scale to String
6270

6371
Casting a `DecimalType` with a negative scale to `StringType` is marked as incompatible when

native/spark-expr/src/conversion_funcs/cast.rs

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,8 @@ use crate::conversion_funcs::numeric::{
2828
};
2929
use crate::conversion_funcs::string::{
3030
cast_string_to_date, cast_string_to_decimal, cast_string_to_float, cast_string_to_int,
31-
cast_string_to_timestamp, is_df_cast_from_string_spark_compatible, spark_cast_utf8_to_boolean,
31+
cast_string_to_timestamp, cast_string_to_timestamp_ntz,
32+
is_df_cast_from_string_spark_compatible, spark_cast_utf8_to_boolean,
3233
};
3334
use crate::conversion_funcs::temporal::{
3435
cast_date_to_timestamp, is_df_cast_from_date_spark_compatible,
@@ -316,6 +317,9 @@ pub(crate) fn cast_array(
316317
(Null, _) => Ok(cast_with_options(&array, to_type, &native_cast_options)?),
317318
(Utf8, Boolean) => spark_cast_utf8_to_boolean::<i32>(&array, eval_mode),
318319
(LargeUtf8, Boolean) => spark_cast_utf8_to_boolean::<i64>(&array, eval_mode),
320+
(Utf8, Timestamp(_, None)) => {
321+
cast_string_to_timestamp_ntz(&array, eval_mode, true, cast_options.is_spark4_plus)
322+
}
319323
(Utf8, Timestamp(_, _)) => cast_string_to_timestamp(
320324
&array,
321325
to_type,

0 commit comments

Comments
 (0)