Rewrite manifests #1661

amitgilad3 · 2025-02-13T20:21:24Z

This is an initial implementation of rewrite manifests, aiming to mimic the Java implementation as closely as possible. I’ve tried to follow the same structure and logic, but there are still some areas that might need refinement.

I’m looking for feedback and suggestions on:
• Whether the approach aligns well with the existing design.
• Any gaps or optimizations that could improve performance.
• How best to proceed with completing this feature.

Would love any insights or guidance on the next steps! Thanks in advance for the review! 🙌

Fokko

Sorry for the many comments, most of them are easy to resolve. Thanks @amitgilad3 for working on this, I like it how you reuse the _ManifestMergeManager 👍

pyiceberg/manifest.py

pyiceberg/table/update/snapshot.py

tests/integration/test_writes/test_rewrite_manifests.py

Fokko · 2025-04-17T13:39:59Z

@amitgilad3 gentle ping, are you still interested in working on this?

amitgilad3 · 2025-04-17T14:04:39Z

Hey @Fokko , yes i am very interested in finishing this, must of missed this (sorry) , will look at this later today :)

2. write tests for rewrite manifests

Fokko · 2025-05-16T21:16:58Z

Looks like the CI is sad 😞

tests/integration/test_writes/test_rewrite_manifests.py:154: error: "rewrite_manifests" of "Table" does not return a value (it only ever returns None)  [func-returns-value]
tests/integration/test_writes/test_rewrite_manifests.py:227: error: "rewrite_manifests" of "Table" does not return a value (it only ever returns None)  [func-returns-value]

Fokko · 2025-05-16T21:19:26Z

pyiceberg/table/update/snapshot.py

+        ssc = SnapshotSummaryCollector()
+        partition_summary_limit = int(
+            self._transaction.table_metadata.properties.get(
+                TableProperties.WRITE_PARTITION_SUMMARY_LIMIT, TableProperties.WRITE_PARTITION_SUMMARY_LIMIT_DEFAULT
+            )
+        )
+        ssc.set_partition_summary_limit(partition_summary_limit)


Suggested change

ssc = SnapshotSummaryCollector()

partition_summary_limit = int(

self._transaction.table_metadata.properties.get(

TableProperties.WRITE_PARTITION_SUMMARY_LIMIT, TableProperties.WRITE_PARTITION_SUMMARY_LIMIT_DEFAULT

)

)

ssc.set_partition_summary_limit(partition_summary_limit)

ssc = SnapshotSummaryCollector(int(

self._transaction.table_metadata.properties.get(

TableProperties.WRITE_PARTITION_SUMMARY_LIMIT, TableProperties.WRITE_PARTITION_SUMMARY_LIMIT_DEFAULT

)

))

amitgilad3 · 2025-05-17T14:41:29Z

Hey @Fokko - just wanted to say thanks for reviewing (really appreciate it) .
I fixed all your comments and added a test to test v1 -> v2 , hope all pass now.
let me know if i have more work todo :)

amitgilad3 · 2025-06-03T12:13:40Z

@Fokko gentle ping, was wondering if we still have any blockers ?

…ntation `LegacyOAuth2AuthManager` (apache#1981)    # Rationale for this change Replace existing Auth handling with `LegacyOAuth2AuthManager`. Tracking issue: apache#1909 There will be follow up PRs to this PR that will address the following: - introduce a mechanism for using a custom `AuthManager` implementation, along with the ability to use a set of config parameters - introduce a `OAuth2AuthManager` that more closely follows the OAuth2 protocol, and also uses a separate thread to proactively refreshes the token, rather than reactively refreshing the token on `UnAuthorizedError` or the deprecated `AuthorizationExpiredError`. # Are these changes tested? Yes, both through unit and integration tests # Are there any user-facing changes? Yes - previously, if `TOKEN` and `CREDENTIAL` are both defined, `oauth/tokens` endpoint wouldn't be used to refresh the token with client credentials when the `RestCatalog` was initialized. However, `oauth/tokens` endpoint would be used on retries that handled 401 or 419 error. This erratic behavior will now be updated as follows: - if `CREDENTIAL` is defined, `oauth/tokens` endpoint will be used to fetch the access token using the client credentials both when the RestCatalog is initialized, and when the refresh_tokens call is made as a reaction to 401 or 419 error. - if both `CREDENTIAL` and `TOKEN` are defined, we will follow the above behavior. - if only `TOKEN` is defined, the initial token will be used instead

# Rationale for this change Add support for the Hugging Face filesystem in `fsspec`, which uses `hf://` paths. This allows to import [HF datasets](https://huggingface.co/datasets). Authentication is done using the `"hf.token"` property. # Are these changes tested? I tried locally but haven't added tests in test_fsspec.py (lmk if it's a requirement) # Are there any user-facing changes? No changes, it simply adds support for `hf://` URLs

Closes apache#2015 # Rationale for this change Enable the input of the ADLFS option `account_host` via the properties option `adls.account-host` # Are these changes tested? Yes # Are there any user-facing changes? The option is now available :)

# Rationale for this change `make test` fails locally since `Config().get_known_catalogs()` also reads my local `~/.pyiceberg.yaml` Follow up to apache#2088 # Are these changes tested? # Are there any user-facing changes?

Closes apache#1929 # Rationale for this change - Since we want to support snapshot write compatibility (apache#1772) and is part of the following parent issue apache#819 # Are these changes tested? Yes # Are there any user-facing changes? No  --------- Co-authored-by: Jayce Slesar <47452474+jayceslesar@users.noreply.github.com> Co-authored-by: Fokko Driesprong <fokko@apache.org>

…pache#2141)    # Rationale for this change Added new configuration parameter hive.kerberos-service-name (apache#2032) hive.kerberos-service-name Defaults to "hive" # Are these changes tested? added unit test. # Are there any user-facing changes? this change adds an optional configuration parameter for the hive catalog (hive.kerberos-service-name) which defaults to "hive". the change includes doc updates.  Co-authored-by: Colm Dougan <colm.dougan@cirata.com>

fixing apache#2122   # Rationale for this change # Are these changes tested? Yes tested locally # Are there any user-facing changes? Nope, just use them --------- Co-authored-by: Kevin Liu <kevinjqliu@users.noreply.github.com> Co-authored-by: Fokko Driesprong <fokko@apache.org>

Bumps [mypy-boto3-dynamodb](https://github.com/youtype/mypy_boto3_builder) from 1.38.4 to 1.39.0. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/youtype/mypy_boto3_builder/releases">mypy-boto3-dynamodb's releases</a>.</em></p> <blockquote> <h2>8.8.0 - Python 3.8 runtime is back</h2> <h3>Changed</h3> <ul> <li><code>[services]</code> <code>install_requires</code> section is calculated based on dependencies in use, so <code>typing-extensions</code> version is set properly</li> <li><code>[all]</code> Replaced <code>typing</code> imports with <code>collections.abc</code> with a fallback to <code>typing</code> for Python <3.9</li> <li><code>[all]</code> Added aliases for <code>builtins.list</code>, <code>builtins.set</code>, <code>builtins.dict</code>, and <code>builtins.type</code>, so Python 3.8 runtime should work as expected again (reported by <a href="https://github.com/YHallouard"><code>@YHallouard</code></a> in <a href="https://redirect.github.com/youtype/mypy_boto3_builder/issues/340">#340</a> and <a href="https://github.com/Omri-Ben-Yair"><code>@Omri-Ben-Yair</code></a> in <a href="https://redirect.github.com/youtype/mypy_boto3_builder/issues/336">#336</a>)</li> <li><code>[all]</code> Unions use the same type annotations as the rest of the structures due to proper fallbacks</li> </ul> <h3>Fixed</h3> <ul> <li><code>[services]</code> Universal input/output shapes were not replaced properly in service subresources</li> <li><code>[docs]</code> Simplified doc links rendering for services</li> <li><code>[services]</code> Cleaned up unnecessary imports in <code>client.pyi</code></li> <li><code>[builder]</code> Import records with fallback are always rendered</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li>See full diff in <a href="https://github.com/youtype/mypy_boto3_builder/commits">compare view</a></li> </ul> </details> <br /> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=mypy-boto3-dynamodb&package-manager=pip&previous-version=1.38.4&new-version=1.39.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [mypy-boto3-glue](https://github.com/youtype/mypy_boto3_builder) from 1.38.42 to 1.39.0. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/youtype/mypy_boto3_builder/releases">mypy-boto3-glue's releases</a>.</em></p> <blockquote> <h2>8.8.0 - Python 3.8 runtime is back</h2> <h3>Changed</h3> <ul> <li><code>[services]</code> <code>install_requires</code> section is calculated based on dependencies in use, so <code>typing-extensions</code> version is set properly</li> <li><code>[all]</code> Replaced <code>typing</code> imports with <code>collections.abc</code> with a fallback to <code>typing</code> for Python <3.9</li> <li><code>[all]</code> Added aliases for <code>builtins.list</code>, <code>builtins.set</code>, <code>builtins.dict</code>, and <code>builtins.type</code>, so Python 3.8 runtime should work as expected again (reported by <a href="https://github.com/YHallouard"><code>@YHallouard</code></a> in <a href="https://redirect.github.com/youtype/mypy_boto3_builder/issues/340">#340</a> and <a href="https://github.com/Omri-Ben-Yair"><code>@Omri-Ben-Yair</code></a> in <a href="https://redirect.github.com/youtype/mypy_boto3_builder/issues/336">#336</a>)</li> <li><code>[all]</code> Unions use the same type annotations as the rest of the structures due to proper fallbacks</li> </ul> <h3>Fixed</h3> <ul> <li><code>[services]</code> Universal input/output shapes were not replaced properly in service subresources</li> <li><code>[docs]</code> Simplified doc links rendering for services</li> <li><code>[services]</code> Cleaned up unnecessary imports in <code>client.pyi</code></li> <li><code>[builder]</code> Import records with fallback are always rendered</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li>See full diff in <a href="https://github.com/youtype/mypy_boto3_builder/commits">compare view</a></li> </ul> </details> <br /> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=mypy-boto3-glue&package-manager=pip&previous-version=1.38.42&new-version=1.39.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [huggingface-hub](https://github.com/huggingface/huggingface_hub) from 0.33.0 to 0.33.1. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/huggingface/huggingface_hub/releases">huggingface-hub's releases</a>.</em></p> <blockquote> <h2>[v0.33.1]: Inference Providers Bug Fixes, Tiny-Agents Message handling Improvement, and Inference Endpoints Health Check Update</h2> <p>Full Changelog: <a href="https://github.com/huggingface/huggingface_hub/compare/v0.33.0...v0.33.1">https://github.com/huggingface/huggingface_hub/compare/v0.33.0...v0.33.1</a></p> <p>This release introduces bug fixes for chat completion type compatibility and feature extraction parameters, enhanced message handling in tiny-agents, and updated inference endpoint health check:</p> <ul> <li>[Tiny agents] Add tool call to messages <a href="https://redirect.github.com/huggingface/huggingface_hub/issues/3159">#3159</a> by <a href="https://github.com/NielsRogge"><code>@NielsRogge</code></a></li> <li>fix: update payload preparation to merge parameters into the output dictionary <a href="https://redirect.github.com/huggingface/huggingface_hub/issues/3160">#3160</a> by <a href="https://github.com/mishig25"><code>@mishig25</code></a></li> <li>fix(inference_endpoints): use GET healthRoute instead of GET / to check status <a href="https://redirect.github.com/huggingface/huggingface_hub/issues/3165">#3165</a> by <a href="https://github.com/mfuntowicz"><code>@mfuntowicz</code></a></li> <li>Recursive filter_none in Inference Providers <a href="https://redirect.github.com/huggingface/huggingface_hub/issues/3178">#3178</a> by <a href="https://github.com/Wauplin"><code>@Wauplin</code></a></li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/huggingface/huggingface_hub/commit/c8ffea9ca815b10edf07085443d524bac479153d"><code>c8ffea9</code></a> Release: v0.33.1</li> <li><a href="https://github.com/huggingface/huggingface_hub/commit/e76fc3acfa3864803dc78b7f76b473fb81173bc8"><code>e76fc3a</code></a> [Tiny agents] Add tool call to messages (<a href="https://redirect.github.com/huggingface/huggingface_hub/issues/3159">#3159</a>)</li> <li><a href="https://github.com/huggingface/huggingface_hub/commit/57eacd2a246e476db9b445ced4ad6796b7c82fa3"><code>57eacd2</code></a> fix(inference_endpoints): use GET <code>healthRoute</code> instead of GET / to check sta...</li> <li><a href="https://github.com/huggingface/huggingface_hub/commit/24a0c75e186cc34c4208a84e5faad3063320b090"><code>24a0c75</code></a> Recursive filter_none in Inference Providers (<a href="https://redirect.github.com/huggingface/huggingface_hub/issues/3178">#3178</a>)</li> <li><a href="https://github.com/huggingface/huggingface_hub/commit/4ada292c260e8d2e234cb2743e0a35706f29f4ae"><code>4ada292</code></a> fix: update payload preparation to merge parameters into the output dictionar...</li> <li>See full diff in <a href="https://github.com/huggingface/huggingface_hub/compare/v0.33.0...v0.33.1">compare view</a></li> </ul> </details> <br /> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=huggingface-hub&package-manager=pip&previous-version=0.33.0&new-version=0.33.1)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [pyroaring](https://github.com/Ezibenroc/PyRoaringBitMap) from 1.0.1 to 1.0.2. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/Ezibenroc/PyRoaringBitMap/releases">pyroaring's releases</a>.</em></p> <blockquote> <h2>1.0.2</h2> <h2>What's Changed</h2> <ul> <li>Restrict Cython version to <code><3.1.0</code> by <a href="https://github.com/Ezibenroc"><code>@Ezibenroc</code></a> in <a href="https://redirect.github.com/Ezibenroc/PyRoaringBitMap/pull/131">Ezibenroc/PyRoaringBitMap#131</a></li> <li>Fix a bug in the hash computation of <code>FrozenBitMap</code> and <code>FrozenBitMap64</code> by <a href="https://github.com/chdemko"><code>@chdemko</code></a> in <a href="https://redirect.github.com/Ezibenroc/PyRoaringBitMap/pull/130">Ezibenroc/PyRoaringBitMap#130</a></li> <li>Upgrade CRoaring from v4.1.1 to v4.2.3 by <a href="https://github.com/Ezibenroc"><code>@Ezibenroc</code></a> in <a href="https://redirect.github.com/Ezibenroc/PyRoaringBitMap/pull/133">Ezibenroc/PyRoaringBitMap#133</a></li> </ul> <h2>New Contributors</h2> <ul> <li><a href="https://github.com/chdemko"><code>@chdemko</code></a> made their first contribution in <a href="https://redirect.github.com/Ezibenroc/PyRoaringBitMap/pull/130">Ezibenroc/PyRoaringBitMap#130</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/Ezibenroc/PyRoaringBitMap/compare/1.0.1...1.0.2">https://github.com/Ezibenroc/PyRoaringBitMap/compare/1.0.1...1.0.2</a></p> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/Ezibenroc/PyRoaringBitMap/commit/369daeb212ad69b4cf68e05b7c632fd2b9473e49"><code>369daeb</code></a> Version 1.0.2</li> <li><a href="https://github.com/Ezibenroc/PyRoaringBitMap/commit/2dce0191124464ccff30dbab8ce72c4c841a7d3c"><code>2dce019</code></a> Upgrade CRoaring to v4.2.3 (<a href="https://redirect.github.com/Ezibenroc/PyRoaringBitMap/issues/133">#133</a>)</li> <li><a href="https://github.com/Ezibenroc/PyRoaringBitMap/commit/52dfaef065ef9946240b001eb409acce7e8c4b71"><code>52dfaef</code></a> Add test_hash_eq_after_operations (<a href="https://redirect.github.com/Ezibenroc/PyRoaringBitMap/issues/132">#132</a>)</li> <li><a href="https://github.com/Ezibenroc/PyRoaringBitMap/commit/c12f08d71b603aa95d259e194bffb8e323d851dd"><code>c12f08d</code></a> Merge branch 'master' into contributor-feature</li> <li><a href="https://github.com/Ezibenroc/PyRoaringBitMap/commit/f4e43c12c4ba25868a0ea34838d96d137ff26327"><code>f4e43c1</code></a> Restrict Cython version (<a href="https://redirect.github.com/Ezibenroc/PyRoaringBitMap/issues/131">#131</a>)</li> <li><a href="https://github.com/Ezibenroc/PyRoaringBitMap/commit/a1bb2ff6b4f281c6b19dd4c223b1e0966872e287"><code>a1bb2ff</code></a> Fix class AbstractBitMap64</li> <li><a href="https://github.com/Ezibenroc/PyRoaringBitMap/commit/eafa2dac12738c009b60f4db01f3c935a35d2d16"><code>eafa2da</code></a> Fix <a href="https://redirect.github.com/Ezibenroc/PyRoaringBitMap/issues/129">#129</a></li> <li>See full diff in <a href="https://github.com/Ezibenroc/PyRoaringBitMap/compare/1.0.1...1.0.2">compare view</a></li> </ul> </details> <br /> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=pyroaring&package-manager=pip&previous-version=1.0.1&new-version=1.0.2)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Fixes: apache#306 --------- Co-authored-by: Kevin Liu <kevin.jq.liu@gmail.com>

…he#2036)    # Rationale for this change Like iceberg jar, we should also update hive storage descriptor after commit metadata see: https://github.com/apache/iceberg/blob/b504f9c51c6c0e0a5c0c5ff53f295e69b67d8e59/hive-metastore/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java#L170 # Are these changes tested? new UTs # Are there any user-facing changes? No

# Rationale for this change See apache#2013 Closes apache#2064 Continuing the trend, but with glue. # Are these changes tested? See test below # Are there any user-facing changes? When a user specifies property update on commit table, those parameters will be passed to the glue client.

# Rationale for this change scan's `row_filter` param is not super intuitive. I got tired of reading over the expression and parser code as I'm trying to build out statements, so I had some docs made up. # Are these changes tested? They are docs only, so not really? # Are there any user-facing changes? Yes there are docs for the expression and string syntaxes of `row_filter` now.  --------- Co-authored-by: Fokko Driesprong <fokko@apache.org>

# Rationale for this change - Added pyiceberg table integration so that pyiceberg `Table` can be pass in directly to datafusion's `register_table_provider` - Added `datafusion` as a optional dependency - Added docs for the integration: <img width="1279" alt="Screenshot 2025-07-06 at 10 59 44 AM" src="https://github.com/user-attachments/assets/f41f08e6-dd41-4012-ad96-2eaae805d28e" /> # Are these changes tested? Yes # Are there any user-facing changes?

# Rationale for this change # Are these changes tested? # Are there any user-facing changes?

I noticed that the docs needed some TLC. - Collapsed some lines to make it more compact. - Avoid imports where possible (eg transforms). - Update docs. - Add an example of the `to_arrow_batch_reader` earlier in the docs.    # Rationale for this change # Are these changes tested? # Are there any user-facing changes?  --------- Co-authored-by: Jayce Slesar <47452474+jayceslesar@users.noreply.github.com>

apache#2153)    # Rationale for this change Hello! Setting or removing table properties on console currently raise a `Writing is WIP` error. This PR adds the code to set and remove table properties. # Are these changes tested? Yes # Are there any user-facing changes? Yes, setting and removing table properties on console now works.

2. write tests for rewrite manifests

sungwy · 2025-07-25T13:15:32Z

Hi @amitgilad3 - sorry for the delayed turn around. There's a question in slack regarding this PR, and I wanted to take a stab at helping merging this in, in Fokko's absence.

Are you still interested in working on this PR? Would you be available to resolve the conflicts?

Anton-Tarazi · 2025-09-24T03:48:16Z

Wondering what the status of this PR is, happy to re-create if you're no longer able to work on it @amitgilad3

amitgilad3 · 2025-09-24T20:46:59Z

Hey @sungwy , @Anton-Tarazi i missed the messages , will fix this week all issues

aammar5 · 2025-10-15T18:45:04Z

I have a local rebased version of this PR, if it's needed ...

ForeverAngry · 2025-11-10T20:44:52Z

@amitgilad3 any updates on this? Im happy to help if need :)

amitgilad3 marked this pull request as draft February 13, 2025 20:21

Fokko self-requested a review February 18, 2025 15:10

Fokko approved these changes Mar 10, 2025

View reviewed changes

kevinjqliu mentioned this pull request Mar 11, 2025

Support metadata compaction #270

Open

amitgilad3 and others added 8 commits May 15, 2025 22:34

first attempt at init_manifests

14c27cf

initial implementation of rewrite_manifests

ee0392d

fix lint

7b82638

remove unused code

15a8f6a

commit

443ffda

1. support returning rewrite results

10affa8

2. write tests for rewrite manifests

add test

82d0d3a

fixes according to pr

6913f09

amitgilad3 force-pushed the rewrite_manifests branch from abbb01c to 6913f09 Compare May 15, 2025 19:35

more fixes

74e42e0

Fokko marked this pull request as ready for review May 16, 2025 21:11

Fokko reviewed May 16, 2025

View reviewed changes

amitgilad3 added 3 commits May 17, 2025 17:23

fixed comments to pr

ba5ebc4

validate seq number

d7dd666

remove redundant test

98db350

sungwy and others added 6 commits July 7, 2025 08:39

Build: Bump flask-cors from 5.0.1 to 6.0.0 (apache#2012)

e10e81d

Build: Bump setuptools from 78.1.0 to 78.1.1 (apache#2027)

b3f7d18

Build: Bump fastavro from 1.10.0 to 1.11.1 (apache#2026)

62cf1a7

kevinjqliu and others added 24 commits July 7, 2025 08:40

dep: update multidict, previous version yanked (apache#2157)

aa9db67

Feature: Write to branches (apache#941)

3a37b52

Fixes: apache#306 --------- Co-authored-by: Kevin Liu <kevin.jq.liu@gmail.com>

Fix the CI (apache#2171)

daa887d

first attempt at init_manifests

9583732

initial implementation of rewrite_manifests

06211b8

fix lint

31efcbc

1. support returning rewrite results

b3487ac

2. write tests for rewrite manifests

lint and rebase

0ad1eeb

linting errors

9bdbc86

ForeverAngry mentioned this pull request Nov 10, 2025

Tracking issues of PyIceberg 0.11 release #2574

Open

Rewrite manifests #1661

Are you sure you want to change the base?

Rewrite manifests #1661

Uh oh!

Conversation

amitgilad3 commented Feb 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Fokko left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Fokko commented Apr 17, 2025

Uh oh!

amitgilad3 commented Apr 17, 2025

Uh oh!

Fokko commented May 16, 2025

Uh oh!

Fokko May 16, 2025

Choose a reason for hiding this comment

Uh oh!

amitgilad3 commented May 17, 2025

Uh oh!

amitgilad3 commented Jun 3, 2025

Uh oh!

sungwy commented Jul 25, 2025

Uh oh!

Anton-Tarazi commented Sep 24, 2025

Uh oh!

amitgilad3 commented Sep 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aammar5 commented Oct 15, 2025

Uh oh!

ForeverAngry commented Nov 10, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

amitgilad3 commented Feb 13, 2025 •

edited

Loading

amitgilad3 commented Sep 24, 2025 •

edited

Loading