Row lineage fields for v3 #2129
Conversation
kevinjqliu
left a comment
kevinjqliu
left a comment
There was a problem hiding this comment.
Thanks for adding this feature!
I think it would be great to include a E2E test with spark. I believe the feature will be available in the next java iceberg release, 1.10
|
@kevinjqliu happy to add that e2e test! Would you prefer I wait for the release + add the test or just file an issue? I know v3 writing is blocked on this feature. |
rambleraptor
force-pushed
the
row-lineage
branch
from
d052b55 to
d52cef7
Compare
rambleraptor
force-pushed
the
row-lineage
branch
2 times, most recently
from
11262a6 to
351a1ca
Compare
|
@Fokko Could not agree more! Addressed your comments and added a test on Spark! |
rambleraptor
force-pushed
the
row-lineage
branch
from
351a1ca to
a62d36b
Compare
|
|
||
|
|
||
| @pytest.mark.integration | ||
| def test_v3_write_and_read_row_lineage(spark: SparkSession, session_catalog: Catalog) -> None: |
There was a problem hiding this comment.
This is the test I would want to write for testing with Spark. Unfortunately, it fails because we don't have v3 writer support (which is needed to append new rows)
Fokko
left a comment
Fokko
left a comment
There was a problem hiding this comment.
Thanks @rambleraptor for the quick turnaround! Could you pull in main once more? We had a failing main earlier today 👍
rambleraptor
force-pushed
the
row-lineage
branch
from
456d2ce to
bc11053
Compare
|
@Fokko rebased! |
|
Thanks @rambleraptor for working on this 🙌 |
3 participants
Closes #1821
Rationale for this change
This adds the proper row-lineage fields for v3. Row-lineage is enforced for v3, so all of this is done by default.
Are these changes tested?
Tests are included. I'd like to include an integration test, but we don't currently allow writing v3 manifest files.
Are there any user-facing changes?
Adds row-lineage fields.