Skip to content

Add TimestampWithOffset canonical extension type #8743

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
serramatutu wants to merge 6 commits into
base: main
Choose a base branch
from
Open

Add TimestampWithOffset canonical extension type #8743

serramatutu wants to merge 6 commits into from
+548 −0

Conversation

@serramatutu
Copy link

@serramatutu serramatutu commented Oct 30, 2025 •
edited
Loading

Which issue does this PR close?

Implement arrow.timestamp_with_offset canonical extension type.

Rationale for this change

Be compliant with Arrow spec: apache/arrow#48002

What changes are included in this PR?

This commit adds a new TimestampWithOffset extension type. This type represents a timestamp column that stores potentially different timezone offsets per value. The timestamp is stored in UTC alongside the original timezone offset in minutes.

Are these changes tested?

Yes.

Are there any user-facing changes?

Yes, this is a new canonical extension type.

@github-actions github-actions bot added the arrow Changes to the arrow crate label Oct 30, 2025
@serramatutu serramatutu mentioned this pull request Oct 30, 2025
Merged
@felipecrv felipecrv self-requested a review November 1, 2025 01:54
westonpace
westonpace approved these changes Nov 4, 2025
View reviewed changes
Copy link
Member

@westonpace westonpace left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems pretty straightforward and reasonable

@felipecrv felipecrv marked this pull request as ready for review November 5, 2025 03:52
@alamb alamb changed the title [DRAFT] Add TimestampWithOffset extension type Add TimestampWithOffset extension type Nov 5, 2025
@alamb
Copy link
Contributor

alamb commented Nov 5, 2025

There are several TODOs in this PRs description. Is that intended?

@serramatutu
Copy link
Author

serramatutu commented Nov 6, 2025 •
edited
Loading

@alamb We're currently still discussing the FORMAT in the arrow repo and in the mailing list before talking too much about implementation. Not sure why @felipecrv marked this as ready for review?

For arrow-rs, I still need to

  • Add run-end and dict encoding to the offset column
  • Figure how to do JSON de/encoding using RFC3339
  • Test more

Realistically, even after the FORMAT PR goes into the spec I think this will take me a few more weeks to fully flesh out.

@alamb
Copy link
Contributor

alamb commented Nov 7, 2025

Thanks for the clarification @serramatutu -- I'll mark it as a draft again then

@alamb alamb marked this pull request as draft November 7, 2025 20:01
felipecrv added a commit to apache/arrow that referenced this pull request Dec 5, 2025
@serramatutu @lidavidm
@jorisvandenbossche @felipecrv
GH-44248: [Format] Add TimestampWithOffset canonical extension type (#…
b10386e 
…48002)

### Rationale for this change

Closes #44248

Arrow has no built-in canonical way of representing the `TIMESTAMP WITH
TIME ZONE` SQL type, which is present across multiple different database
systems. Not having a native way to represent this forces users to
either convert to UTC and drop the time zone, which may have correctness
implications, or use bespoke workarounds. A new
`arrow.timestamp_with_offset` extension type would introduce a standard
canonical way of representing that information.

Rust implementation: apache/arrow-rs#8743
Go implementation: apache/arrow-go#558

[DISCUSS] [thread in the mailing
list](https://lists.apache.org/thread/yhbr3rj9l59yoxv92o2s6dqlop16sfnk).

### What changes are included in this PR?

Proposal and documentation for `arrow.timestamp_with_offset` canonical
extension type.

### Are these changes tested?

N/A

### Are there any user-facing changes?

Yes, this is an extension to the arrow format.

* GitHub Issue: #44248

---------

Co-authored-by: David Li <li.davidm96@gmail.com>
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Co-authored-by: Felipe Oliveira Carvalho <felipekde@gmail.com>
@serramatutu serramatutu force-pushed the serramatutu/TimestampWithOffset/rs branch from d187bb9 to f6cda58 Compare December 16, 2025 16:07
@serramatutu serramatutu marked this pull request as ready for review December 16, 2025 16:07
alamb
alamb approved these changes Dec 18, 2025
View reviewed changes
Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me -- thank you @serramatutu and @westonpace

arrow-schema/src/extension/canonical/mod.rs
/// The extension type for `TimestampWithOffset`.
///
/// <https://arrow.apache.org/docs/format/CanonicalExtensions.html#timestamp-with-offset>
TimestampWithOffset(TimestampWithOffset),
Copy link
Contributor

@alamb alamb Dec 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this is a pub enum, unfortunately this will need to wait until the next major arrow release (in Jan):

@alamb alamb changed the title Add TimestampWithOffset extension type Add TimestampWithOffset canonical extension type Dec 18, 2025
@serramatutu
Copy link
Author

serramatutu commented Dec 22, 2025

Looks like we got a CI flake and need to rerun:

The hosted runner lost communication with the server. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error.

felipecrv
felipecrv approved these changes Dec 29, 2025
View reviewed changes
@serramatutu
Add TimestampWithOffset extension type
8b76386 
This commit adds a new `TimestampWithOffset` extension type.

This type represents a timestamp column that stores potentially
different timezone offsets per value. The timestamp is stored in
UTC alongside the original timezone offset in minutes.
@serramatutu
Update description to match new format
46ee65b
@serramatutu
Allow offsets to be dictionary encoded
aae4d37
@serramatutu
Allow offsets to be run end encoded
9e9075b
@serramatutu
Minor doc changes to comply with spec
3d48b36
@serramatutu
paragraph formatting
3efea23
@serramatutu serramatutu force-pushed the serramatutu/TimestampWithOffset/rs branch from ab316a4 to 3efea23 Compare January 8, 2026 11:35
@serramatutu
Copy link
Author

serramatutu commented Jan 8, 2026

Rebased onto main.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@felipecrv felipecrv felipecrv approved these changes

@alamb alamb alamb approved these changes

@westonpace westonpace westonpace approved these changes

Assignees

No one assigned

Labels

arrow Changes to the arrow crate

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@serramatutu @alamb @felipecrv @westonpace