Skip to content

Conversation

@nord-erik
Copy link
Contributor

@nord-erik nord-erik commented Dec 10, 2025

adds support to natively write these spark types to neo4j:

  • Byte: will convert to Integer, therefore cannot read-back as Byte
  • Short: will convert to Integer, therefore cannot read-back as Short
  • Decimal: will convert to String, therefore cannot read-back as Decimal
  • Binary: will convert to ByteArray and will be read-back as ByteArray
  • TimestampNTZType: will convert to LocalDateTime

BREAKING CHANGE:

  • already existing Timestamp now converts to ZonedDateTime, not LocalDateTime.

Before, writing a Timestamp would remove the timezone information.
Now it will be normalized to UTC-time instead. If you want to retain old behaviour,
please use TimestampNTZType or be mindful when you read the stamp.

Assuming you interpret time zone when reading and writing, you should not notice.
But if you indeed intend to use LocalDateTime, this is moved and maps to TimestampNTZType

@nord-erik nord-erik force-pushed the conn-341-support-all-built-in-spark-types-for-reads-and-writes branch from e851e20 to 96ecdba Compare December 10, 2025 15:47
@neo4j neo4j deleted a comment from neo4j-connectors Dec 10, 2025
@nord-erik nord-erik changed the title feat: introduce byte, short, timestampNTZ types feat: introduce byte, short, decimal, timestampNTZ types Dec 15, 2025
@nord-erik nord-erik force-pushed the conn-341-support-all-built-in-spark-types-for-reads-and-writes branch from e51437a to 898c4be Compare December 15, 2025 14:35
@nord-erik nord-erik marked this pull request as ready for review December 16, 2025 10:50
@nord-erik nord-erik requested a review from a team as a code owner December 16, 2025 10:50
@neo4j neo4j deleted a comment from neo4j-connectors Dec 16, 2025
@neo4j neo4j deleted a comment from neo4j-connectors Dec 16, 2025
@neo4j neo4j deleted a comment from neo4j-connectors Dec 16, 2025
@neo4j neo4j deleted a comment from neo4j-connectors Dec 16, 2025
@nord-erik nord-erik force-pushed the conn-341-support-all-built-in-spark-types-for-reads-and-writes branch from 5472292 to 5bcadee Compare December 16, 2025 20:38
@neo4j neo4j deleted a comment from neo4j-connectors Dec 16, 2025
@nord-erik nord-erik force-pushed the conn-341-support-all-built-in-spark-types-for-reads-and-writes branch 2 times, most recently from 789742e to b81a02a Compare December 17, 2025 21:08
DataTypes.createArrayType(DataTypes.TimestampType, false) -> "LIST<LOCAL DATETIME NOT NULL>",
DataTypes.createArrayType(DataTypes.TimestampType, true) -> "LIST<LOCAL DATETIME NOT NULL>",
DataTypes.createArrayType(DataTypes.TimestampType, false) -> "LIST<ZONED DATETIME NOT NULL>",
DataTypes.createArrayType(DataTypes.TimestampType, true) -> "LIST<ZONED DATETIME NOT NULL>",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question: we cannot really have null zoned datetimes in lists, can we?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd assume so as well. No idea what that would look like / represent like if it was the case

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in that case, we cannot call createArray with true. Afaik, Cypher does not support nulls in lists.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm... confused about this to be honest. when I did durations I added both {true, false} because timestamps already had it.

what's the consequences here to remove? I'll remove and see if all tests pass in the meantime.

@nord-erik nord-erik force-pushed the conn-341-support-all-built-in-spark-types-for-reads-and-writes branch from 17f0c3c to aaddd82 Compare December 18, 2025 16:37
@nord-erik nord-erik enabled auto-merge (squash) December 19, 2025 14:24
@nord-erik nord-erik disabled auto-merge December 19, 2025 14:24
@nord-erik nord-erik merged commit 7a77773 into 5.0 Dec 19, 2025
125 checks passed
@nord-erik nord-erik deleted the conn-341-support-all-built-in-spark-types-for-reads-and-writes branch December 19, 2025 14:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants