[SQL] Support for TIMESTAMP WITH TIME ZONE by mihaibudiu · Pull Request #6116 · feldera/feldera

mihaibudiu · 2026-04-24T06:07:35Z

This PR finally adds support for the last important missing SQL type TIMESTAMP WITH TIME ZONE
There are multiple ways to implement this; I have chosen a version which provides similar capabilities with most databases: timestamps with time zones are stored at runtime as simply UTC timestamps. Thus they are comparable to each other. No time zone information is preserved at runtime, though.

I will mark this PR as a draft for two reasons:

I need to file an issue and a fix for Calcite to properly support literals: [CALCITE-7491] Literals of type TIMESTAMP WITH TIME ZONE cause crashes apache/calcite#4910
I need to implement many more tests - at least one for each function. Essentially all functions which work for TIMESTAMP will work for the new type as well. I plan to copy-and-paste the python tests we have for TIMESTAMP

The Rust serialization code will need to be audited for compatibility with other standard formats (e.g., Debezium).

mythical-fred

Several blockers; please mark this as draft as the description suggests, or address before merge.

mythical-fred · 2026-04-25T05:06:14Z

            JsonFlavor::DebeziumMySql => Self {
                time_format: TimeFormat::Micros,
                date_format: DateFormat::DaysSinceEpoch,
+                // Why is this missing fractions of second?


Stray // Why is this missing fractions of second? comment with a question to nobody. Either answer it (Debezium MySQL really does emit second-precision timestamps in this layout) or open an issue and reference it. Don't ship a TODO disguised as a question.

mythical-fred

Re-review on tip 7edb1df (force-pushed since the prior review on 8b53059). Several of my prior blockers were silently fixed without a reply (good); a few are not yet addressed; and the new force-push introduced regressions in cast_to_TimestampTz_s and a Python test file that won't even parse. PR is now draft, which is the right call.

Prior blockers — status

timestamp.rs SerializeWithContext using timestamp_format instead of timestamp_tz_format — fixed.
serialize_table_record!(TestStruct[3] {...}) field count — fixed (now [4]).
DBSPTypeTimestampTz getMaxValue 8 nines — fixed (6 nines).
CastTests Americas/New_York — fixed.
TypeCompiler TZ branch warning text + precision constant — fixed (note: timestampPrecisionsWarned is still shared between the two branches; minor).
TimestampTz // Is this right for negative timestamps? TODO + Rustdoc copy-paste — gone (the methods were removed; if/when they come back, please don't reintroduce the bug).
TZ section docs typos / heading — fixed.
Non-actionable cast error — only partially: type name corrected to TIMESTAMP WITH TIME ZONE, but still no format hint as PR_REVIEW_RULES.md asks (see inline).
Magic-number Objects.hash(..., 14) — number changed to 16, still a bare magic constant (see inline).

Still open / not addressed

Test coverage. Same two integration tests as before (issue4146, issue4146a). Your own description still says "I need to implement many more tests - at least one for each function". This is the headline blocker for the PR.
Plain-TIMESTAMP doc typos data time / asssumed (datetime.md L196-197) — flagged previously to fix in the same PR; still present.
Stray rhetorical // Why is this missing fractions of second? comment in serde_config.rs:204.

New issues introduced by this force-push

cast_to_TimestampTz_s now uses only DateTime::parse_from_rfc3339. That's a regression: the previous version at least accepted YYYY-MM-DD HH:MM:SS[.f] and YYYY-MM-DD. RFC3339 also doesn't accept IANA zone names, even though both the docs and DBSPTimestampTzLiteral advertise IANA support. Net effect: literals accept '2020-01-01 10:10:10 America/New_York' but CAST('2020-01-01 10:10:10 America/New_York' AS TIMESTAMP WITH TIME ZONE) will now fail at runtime.
python/tests/runtime_aggtest/aggregate_tests2/test_timestamp_tbl.py — the new TZ test data is broken in three ways (syntax error + two malformed strings). See inline; this file won't even import.
DBSPTimestampTzLiteral.getMicros has a negative-modulo footgun (see inline) — pre-1970 TZ literals will throw from LocalDateTime.ofEpochSecond because nanoOfSecond must be non-negative.

Keeping verdict at REQUEST_CHANGES until the test coverage gap and the regressed cast/python paths are addressed.

mythical-fred · 2026-04-30T01:45:28Z

+
+#[doc(hidden)]
+pub fn cast_to_TimestampTz_s(value: SqlString) -> SqlResult<TimestampTz> {
+    if let Ok(v) = DateTime::parse_from_rfc3339(value.str()) {


Two regressions vs. the previous implementation, both visible to users:

No IANA zone names. DateTime::parse_from_rfc3339 only accepts numeric offsets. But DBSPTimestampTzLiteral and docs/sql/datetime.md both promise IANA names (America/New_York, etc.). So TIMESTAMP WITH TIME ZONE '2020-01-01 10:10:10 America/New_York' works as a literal but CAST('2020-01-01 10:10:10 America/New_York' AS TIMESTAMP WITH TIME ZONE) fails. Please use chrono_tz (or equivalent) to handle IANA names, then fall back to parse_from_rfc3339 for numeric offsets.

Date-only and space-separated forms dropped. The previous version of this function accepted YYYY-MM-DD HH:MM:SS[.f] (no offset, treated as UTC) and YYYY-MM-DD. RFC3339 requires the T separator and an offset, so both of those casts now fail. At minimum, please keep the %Y-%m-%d %H:%M:%S%.f and %Y-%m-%d fallbacks (UTC-assumed, matching the literal grammar's "assumed to be in UTC").

A test that exercises each accepted format end-to-end (numeric offset, IANA name, date-only, no-offset) belongs with this fix.

mythical-fred · 2026-04-30T01:45:28Z

            JsonFlavor::DebeziumMySql => Self {
                time_format: TimeFormat::Micros,
                date_format: DateFormat::DaysSinceEpoch,
+                // Why is this missing fractions of second?


Still here from prior review. Either answer the question (does Debezium MySQL really emit second-precision timestamps in this layout? — if so, say so) or open an issue and reference it. Don't ship a TODO disguised as a question.

mihaibudiu · 2026-05-13T22:41:44Z

Most of the Python tests are validated because they have the same expected shape and results as tests for TIMESTAMP.

mihaibudiu · 2026-05-13T22:44:48Z

@snkas this is a big PR, but I think most scrutiny is needed in the Rust code dealing with connectors and pipeline manager, which is probably < 100 lines of code. Most of the code is in tests; see the above comment why the Python tests are probably correct.

snkas

From pipeline-manager perspective, it's a new type added to the OpenAPI schema, that part looks good.

For the connectors: the input and output format of this new data type is still MicrosSinceEpoch, so an integer. When would one use TIMESTAMP instead of TIMESTAMP WITH TIME ZONE, and vice versa? It seems they are effectively the same in terms of the queries you can do on them (e.g., it's no possible to query "give me all the tuples with the timestamp in time zone +01:00" or so)?

mihaibudiu · 2026-05-15T16:37:59Z

Our implementation with TIMESTAMP WITH TIME ZONE aligns with most major databases. For one, some datasets already include timestamps with time zones, so this will enable us to load them. Then there are functions to convert timezones. CONVERT_TIMEZONE is an example we already have, but there is a function (not yet implemented) AT TIME ZONE, which takes a timestamp at a given time zone and converts it to UDT. Indeed, we do not carry the actual time zone; time zones are relatively complex objects, which do not make sense without a TIMESTAMP, and whose definition evolves every year, they are not simply numeric offsets.

mihaibudiu · 2026-05-15T20:58:19Z

@snkas any actionable feedback? Can we merge this and refine?
There's documentation about the semantics of the new type in the PR.

mythical-fred

Big blockers from my prior two reviews are addressed: IANA zones now parse, the error message is actionable with the expected format, getMaxValue() is back to 6 fractional digits, DBSPTimestampTzLiteral.getMicros uses Math.floorDiv/Math.floorMod (no more negative-modulo footgun), the broken test_timestamp_tbl.py row is now valid Python with a clean +05:00 offset, and test coverage has expanded substantially (issue4146a with ten in-line cases, testCaseTz, testCompareTz, plus the new test_interval_*, test_cmp_operators.py, test_date_time_fn.py and TZ rows in test_timestamp_tbl.py). Approving. Two tiny leftovers inline.

mythical-fred · 2026-05-16T20:53:54Z

+
+    @Override
+    public int hashCode() {
+        return Objects.hash(this.mayBeNull, 16);


Still a bare magic number — 14 -> 16 doesn't really address the original concern; the literal carries no meaning at the call site. A one-line // Discriminator for hash-disambiguation across DBSPType subclasses (or whatever the convention here is) would make this self-explanatory. Not a blocker.

mythical-fred · 2026-05-16T20:53:54Z

            JsonFlavor::DebeziumMySql => Self {
                time_format: TimeFormat::Micros,
                date_format: DateFormat::DaysSinceEpoch,
+                // Why is this missing fractions of second?


This rhetorical TODO comment is still here from my first review. Either answer it (does Debezium MySQL truly emit second-precision timestamps in this flavor? — confirm and replace with a single-sentence explanation citing the upstream behaviour) or delete it. Floating questions to nobody are noise. Not a blocker.

snkas

I see, so this PR has to benefits:

(1) It allows us to ingest data from sources that provide a column that has timestamp+timezone as its type. Would this not already be possible by adding an extra mapping to Timestamp? Given we don't store the timezone in any case.
(2) It opens up the possibility for SQL timezone-related functions. But if we don't store the timezone in this new type, wouldn't CHAR just still be the output type for both AT TIMESTAMP and CONVERT_TIMESTAMP? Documentation: https://docs.feldera.com/sql/datetime/#convert_timezone

As the PR description is not "Add type TIMESTAMP WITH TIMEZONE as an alias for TIMESTAMP", which would be my expectation so far based on our back-and-forth. How does the type differ?

snkas · 2026-05-18T08:42:37Z

            // hardwire that for now.  We can make it configurable in the future.
            // FIXME: Also, the timezone should be `None`, but that gets compiled into `timezone_ntz`
            // in the JSON schema, which Databricks doesn't fully support yet.
            SqlType::Timestamp => DataType::Timestamp(


There is this special case for Timestamp, wouldn't this also apply to TimestampTz? @ryzhyk added the comment here.

I hope that this will work - the special case was None not being supported

mihaibudiu · 2026-05-18T15:57:45Z

In other databases an important difference between TIMESTAMP and TIMESTAMP WITH TIME ZONE is that the latter is usually adjusted for I/O based on the local timezone of the database, whereas the former is absolute. Now, Feldera databases always run in the UTC time zone, which has the nice benefit of making things deterministic. So these two do look very similar.

Think of the former as a NaiveDateTime and the latter as DateTime<Utc>. Both these types exist in Rust, and although they both have the same representation in Rust, they typecheck differently, and they do have different methods. The set of functions in Rust overlaps to a significant degree, but we will add more functions which are specific to only one of the two types.

Signed-off-by: Mihai Budiu <mbudiu@feldera.com>

mihaibudiu · 2026-05-20T04:12:49Z

Merging this is blocked on #6133 - without this it requires a platform upgrade

mihaibudiu force-pushed the issue4146 branch from bd992a1 to 8b53059 Compare April 24, 2026 16:24

mythical-fred suggested changes Apr 25, 2026

View reviewed changes

mihaibudiu force-pushed the issue4146 branch 2 times, most recently from 48d602b to 4e74e03 Compare April 25, 2026 05:46

mihaibudiu mentioned this pull request Apr 25, 2026

[SQLLIB] Adapt the storage tests to handle multiple VARIANT values #6132

Closed

mihaibudiu marked this pull request as draft April 28, 2026 05:28

mihaibudiu force-pushed the issue4146 branch 2 times, most recently from add429a to 7edb1df Compare April 29, 2026 04:24

mythical-fred suggested changes Apr 30, 2026

View reviewed changes

mihaibudiu force-pushed the issue4146 branch from 7edb1df to a5f32ec Compare May 13, 2026 22:15

mihaibudiu marked this pull request as ready for review May 13, 2026 22:16

mihaibudiu requested a review from snkas May 13, 2026 22:43

mihaibudiu force-pushed the issue4146 branch 6 times, most recently from dbdd9a6 to 8091d80 Compare May 14, 2026 22:30

snkas reviewed May 15, 2026

View reviewed changes

mihaibudiu force-pushed the issue4146 branch from 8091d80 to 6d8137c Compare May 15, 2026 20:52

mihaibudiu mentioned this pull request May 15, 2026

Iceberg input connector fails to deserialize timestamptz columns from Parquet: Unsupported timezone: +00:00 #5959

Open

mythical-fred approved these changes May 16, 2026

View reviewed changes

snkas approved these changes May 18, 2026

View reviewed changes

mihaibudiu force-pushed the issue4146 branch from 6d8137c to 37e534f Compare May 19, 2026 01:42

mihaibudiu force-pushed the issue4146 branch 2 times, most recently from 18a7640 to ae3223e Compare May 19, 2026 03:58

mihaibudiu enabled auto-merge May 19, 2026 03:59

mihaibudiu disabled auto-merge May 19, 2026 03:59

swanandx added this pull request to the merge queue May 19, 2026

github-merge-queue Bot removed this pull request from the merge queue due to a conflict with the base branch May 19, 2026

mihaibudiu force-pushed the issue4146 branch from ae3223e to 1192c22 Compare May 19, 2026 15:59

mihaibudiu enabled auto-merge May 19, 2026 15:59

mihaibudiu added this pull request to the merge queue May 19, 2026

github-merge-queue Bot removed this pull request from the merge queue due to failed status checks May 19, 2026

mihaibudiu added 2 commits May 19, 2026 13:08

[SQL] Advance Calcite version

70d1371

Signed-off-by: Mihai Budiu <mbudiu@feldera.com>

[SQL] Support for TIMESTAMP WITH TIME ZONE

0a3ea0a

Signed-off-by: Mihai Budiu <mbudiu@feldera.com>

mihaibudiu force-pushed the issue4146 branch from 1192c22 to 0a3ea0a Compare May 19, 2026 20:17

mihaibudiu enabled auto-merge May 19, 2026 20:17

mihaibudiu added this pull request to the merge queue May 19, 2026

github-merge-queue Bot removed this pull request from the merge queue due to failed status checks May 19, 2026

Conversation

mihaibudiu commented Apr 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mythical-fred left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mythical-fred Apr 25, 2026

Choose a reason for hiding this comment

Uh oh!

mythical-fred left a comment

Choose a reason for hiding this comment

Uh oh!

mythical-fred Apr 30, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mythical-fred Apr 30, 2026

Choose a reason for hiding this comment

Uh oh!

mihaibudiu commented May 13, 2026

Uh oh!

mihaibudiu commented May 13, 2026

Uh oh!

snkas left a comment

Choose a reason for hiding this comment

Uh oh!

mihaibudiu commented May 15, 2026

Uh oh!

mihaibudiu commented May 15, 2026

Uh oh!

mythical-fred left a comment

Choose a reason for hiding this comment

Uh oh!

mythical-fred May 16, 2026

Choose a reason for hiding this comment

Uh oh!

mythical-fred May 16, 2026

Choose a reason for hiding this comment

Uh oh!

snkas left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

snkas May 18, 2026

Choose a reason for hiding this comment

Uh oh!

mihaibudiu May 19, 2026

Choose a reason for hiding this comment

Uh oh!

mihaibudiu commented May 18, 2026

Uh oh!

Uh oh!

Uh oh!

mihaibudiu commented May 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mihaibudiu commented Apr 24, 2026 •

edited

Loading

snkas left a comment •

edited

Loading