Fix: Serialize interval columns as PostgreSQL strings during backfill #2095
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fix: Serialize interval columns as HH:MM:SS strings during backfill
Problem
During backfill, Sequin uses
SELECTqueries and processes results through a loader that normalizes values for JSON serialization. ThePostgrex.Intervalstruct returned by Postgrex lacked aJason.Encoderimplementation, causing the loader to emitnullfor interval columns. In contrast, live CDC decodes WAL tuples as text strings, so interval values were emitted correctly in streaming mode.This discrepancy meant:
nullbefore transform executionSolution
Added a
Jason.Encoderimplementation forPostgrex.Intervalthat serializes intervals as simpleHH:MM:SStime strings. All interval components (months, days, seconds) are converted to a total time representation.Conversion rules:
.5for 500000 microsecs)Examples:
"00:00:01""01:01:01""24:00:00""25:00:00""720:00:00""769:01:01""-24:00:00""100:00:00""00:00:01.5"Changes
lib/sequin/postgrex/encoders.ex: AddedJason.Encoderimplementation forPostgrex.Intervalthat converts intervals toHH:MM:SSformattest/sequin/postgrex/encoders_test.exs: Added comprehensive tests for interval encoding including edge cases (zero values, negative values, microseconds, large intervals, days, months)Related Issues
Similar to the approach used for
daterangein GitHub Issue #816.Testing
mix test test/sequin/postgrex/encoders_test.exsAll 15 tests pass, covering: