Skip to content

datetime.md

Latest commit

 

History

History
47 lines (36 loc) · 2.29 KB

datetime.md

File metadata and controls

47 lines (36 loc) · 2.29 KB

Date/Time Expressions

  • Hour, Minute, Second: Incorrectly apply timezone conversion to TimestampNTZ inputs. TimestampNTZ stores local time without timezone, so no conversion should be applied. These expressions work correctly with Timestamp inputs. #3180
  • TruncTimestamp (date_trunc): Produces incorrect results when used with non-UTC timezones. Compatible when timezone is UTC. TimestampNTZ inputs are handled correctly (timezone-independent truncation). #2649

Date and Time Functions

Comet's native implementation of date and time functions may produce different results than Spark for dates far in the future (approximately beyond year 2100). This is because Comet uses the chrono-tz library for timezone calculations, which has limited support for Daylight Saving Time (DST) rules beyond the IANA time zone database's explicit transitions.

For dates within a reasonable range (approximately 1970-2100), Comet's date and time functions are compatible with Spark. For dates beyond this range, functions that involve timezone-aware calculations (such as date_trunc with timezone-aware timestamps) may produce results with incorrect DST offsets.

If you need to process dates far in the future with accurate timezone handling, consider:

  • Using timezone-naive types (timestamp_ntz) when timezone conversion is not required
  • Falling back to Spark for these specific operations