Rewrite manifests #1661

Open

amitgilad3 wants to merge 112 commits intoapache:mainfrom

amitgilad3:rewrite_manifests

Commits on May 17, 2025

Commits on Jul 7, 2025

Replace Deprecated (Current) OAuth2 Handling with AuthManager Implementation `LegacyOAuth2AuthManager` (#1981)

sungwy
authored and
amitgilad3
committed
Add Hugging Face filesystem support to fsspec (#1997)

lhoestq
authored and
amitgilad3
committed
Build: Bump flask-cors from 5.0.1 to 6.0.0 (#2012)

dependabot[bot]
authored and
amitgilad3
committed
Add ADLFS property `adls.account-host` (#2016)

vsdsantos
authored and
amitgilad3
committed
Build: Bump setuptools from 78.1.0 to 78.1.1 (#2027)

dependabot[bot]
authored and
amitgilad3
committed
Build: Bump fastavro from 1.10.0 to 1.11.1 (#2026)

dependabot[bot]
authored and
amitgilad3
committed
Build: Bump sqlalchemy from 2.0.40 to 2.0.41 (#2025)

dependabot[bot]
authored and
amitgilad3
committed
Build: Bump getdaft from 0.4.14 to 0.4.15 (#2023)

dependabot[bot]
authored and
amitgilad3
committed
Build: Bump mypy-boto3-glue from 1.38.12 to 1.38.18 (#2021)

dependabot[bot]
authored and
amitgilad3
committed
Build: Bump huggingface-hub from 0.31.2 to 0.31.4 (#2020)

dependabot[bot]
authored and
amitgilad3
committed
feat: `validate_deleted_data_files` (#1938)

authored and
amitgilad3
committed
Build: Bump mkdocs-material from 9.6.13 to 9.6.14 (#2024)

dependabot[bot]
authored and
amitgilad3
committed
Build: Bump cython from 3.0.12 to 3.1.1 (#2022)

dependabot[bot]
authored and
amitgilad3
committed
Fix the CI (#2049)

Fokko
authored and
amitgilad3
committed
Build: Bump mypy-boto3-glue from 1.38.18 to 1.38.22 (#2038)

dependabot[bot]
authored and
amitgilad3
committed
Build: Bump moto from 5.1.4 to 5.1.5 (#2039)

dependabot[bot]
authored and
amitgilad3
committed
Build: Bump pytest-mock from 3.14.0 to 3.14.1 (#2040)

dependabot[bot]
authored and
amitgilad3
committed
Build: Bump getdaft from 0.4.15 to 0.4.16 (#2042)

dependabot[bot]
authored and
amitgilad3
committed
Build: Bump pydantic from 2.11.4 to 2.11.5 (#2045)

dependabot[bot]
authored and
amitgilad3
committed
Build: Bump coverage from 7.8.0 to 7.8.2 (#2047)

dependabot[bot]
authored and
amitgilad3
committed
Build: Bump huggingface-hub from 0.31.4 to 0.32.1 (#2046)

dependabot[bot]
authored and
amitgilad3
committed
Build: Bump thrift from 0.21.0 to 0.22.0 (#2043)

dependabot[bot]
authored and
amitgilad3
committed
Build: Bump mkdocstrings-python from 1.16.10 to 1.16.11 (#2044)

dependabot[bot]
authored and
amitgilad3
committed
Build: Bump mkdocs-autorefs from 1.4.1 to 1.4.2 (#2048)

dependabot[bot]
authored and
amitgilad3
committed
Build: Bump cachetools from 5.5.2 to 6.0.0 (#2041)

dependabot[bot]
authored and
amitgilad3
committed
Add `.vscode` folder to `.gitignore` (#2053)

rambleraptor
authored and
amitgilad3
committed
Make `validation_history` private before release (#2054)

sungwy
authored and
amitgilad3
committed
Build: Bump datafusion from 46.0.0 to 47.0.0 (#2063)

dependabot[bot]
authored and
amitgilad3
committed
Build: Bump huggingface-hub from 0.32.2 to 0.32.3 (#2061)

dependabot[bot]
authored and
amitgilad3
committed
Build: Bump getdaft from 0.4.16 to 0.4.18 (#2060)

dependabot[bot]
authored and
amitgilad3
committed
Build: Bump pyspark from 3.5.5 to 3.5.6 (#2062)

dependabot[bot]
authored and
amitgilad3
committed
Leverage Iceberg-Rust for all the transforms (#1833)

authored and
amitgilad3
committed
Build: Bump cython from 3.1.1 to 3.1.2 (#2081)

dependabot[bot]
authored and
amitgilad3
committed
Build: Bump huggingface-hub from 0.32.3 to 0.32.4 (#2077)

dependabot[bot]
authored and
amitgilad3
committed
Build: Bump requests from 2.32.3 to 2.32.4 (#2078)

dependabot[bot]
authored and
amitgilad3
committed
Build: Bump duckdb from 1.2.2 to 1.3.0 (#2080)

dependabot[bot]
authored and
amitgilad3
committed
Build: Bump pandas from 2.2.3 to 2.3.0 (#2079)

dependabot[bot]
authored and
amitgilad3
committed
Build: Bump mkdocstrings-python from 1.16.11 to 1.16.12 (#2076)

dependabot[bot]
authored and
amitgilad3
committed
Robustify boto3 session handling (DynamoDB, RestCatalog) (#2071)

authored and
amitgilad3
committed
chore: remove non-test asserts (#2082)

jayceslesar
authored and
amitgilad3
committed
fix: glue drop_namespace to check non-iceberg tables (#2083)

geruh
authored and
amitgilad3
committed
Add v3 structs (#2065)

b-phi
authored and
amitgilad3
committed
Add Avro compression (#1976)

Fokko
authored and
amitgilad3
committed
REST: Pass `purgeRequest` to catalog (#2086)

jayceslesar
authored and
amitgilad3
committed
Bump Ruff to the latest version (#2073)

Fokko
authored and
amitgilad3
committed
Bump mypy in pre-commit (#2074)

Fokko
authored and
amitgilad3
committed
feat: support listing known catalogs (#2088)

Anton-Tarazi
authored and
amitgilad3
committed
fix: add metadata_properties to _construct_parameters when update hive table (#2013)

kadai0308
authored and
amitgilad3
committed
change catalog default warehouse location to not use hive-style warehouse location (#2059)

kevinjqliu
authored and
amitgilad3
committed
Bump PyIceberg to 0.9.1 (#2092)

Fokko
authored and
amitgilad3
committed
Bump tests to Spark 3.5.6 (#2091)

Fokko
authored and
amitgilad3
committed
Bump Iceberg-Java to 1.9.1 (#2093)

authored and
amitgilad3
committed
maint: use `URI` constant instead of 'uri' strings (#2094)

jayceslesar
authored and
amitgilad3
committed
Build: Bump pypa/cibuildwheel from 2.23.3 to 3.0.0 (#2099)

dependabot[bot]
authored and
amitgilad3
committed
Build: Bump moto from 5.1.5 to 5.1.6 (#2100)

dependabot[bot]
authored and
amitgilad3
committed
Build: Bump cachetools from 5.5.2 to 6.1.0 (#2101)

dependabot[bot]
authored and
amitgilad3
committed
Build: Bump huggingface-hub from 0.32.5 to 0.33.0 (#2103)

dependabot[bot]
authored and
amitgilad3
committed
Build: Bump typing-extensions from 4.13.2 to 4.14.0 (#2104)

dependabot[bot]
authored and
amitgilad3
committed
Build: Bump duckdb from 1.3.0 to 1.3.1 (#2105)

dependabot[bot]
authored and
amitgilad3
committed
Build: Bump boto3 from 1.37.3 to 1.38.27 (#2108)

dependabot[bot]
authored and
amitgilad3
committed
Build: Bump pydantic from 2.11.5 to 2.11.7 (#2107)

dependabot[bot]
authored and
amitgilad3
committed
Build: Bump coverage from 7.8.2 to 7.9.1 (#2106)

dependabot[bot]
authored and
amitgilad3
committed
Docs: fix manifest min count to merge description (#2116)

ThomasDsantos
authored and
amitgilad3
committed
Prefer `FileIO` over the PyArrow `FileSystem` (#2115)

Fokko
authored and
amitgilad3
committed
Add `ExpireSnapshots` (#1880)

authored and
amitgilad3
committed
Build: Bump third party deps (#2127)

Fokko
authored and
amitgilad3
committed
Support ADLS with Pyarrow file IO (#2111)

authored and
amitgilad3
committed
Improve upsert memory pressure (#1995)

authored and
amitgilad3
committed
Type mismatch Avro zstd decompression (#2128)

zhongyujiang
authored and
amitgilad3
committed
Expose S3 retry strategy (#2110)

Fokko
authored and
amitgilad3
committed
Refactor `Makefile` and run s3/adls/gcs tests (#2125)

kevinjqliu
authored and
amitgilad3
committed
perf: optimize `table.add_files` and `inspect.files` (#2133)

authored and
amitgilad3
committed
Run `test_create_table_transaction` for Rest V1 tables (#2137)

Fokko
authored and
amitgilad3
committed
Add warehouse parameter to the REST Catalog doc (#2066)

authored and
amitgilad3
committed
Build: Bump mypy-boto3-glue from 1.38.22 to 1.38.42 (#2144)

dependabot[bot]
authored and
amitgilad3
committed
test: use `create or replace` (#2140)

kevinjqliu
authored and
amitgilad3
committed
Update assertion for testing locally (#2113)

kevinjqliu
authored and
amitgilad3
committed
Validate added data files for snapshot compatibility (#2050)

authored and
amitgilad3
committed
Add `hive.kerberos-service-name` configuration option (#2032) (#2141)

authored and
amitgilad3
committed
Relax REST client `/v1/config` response constraints (#2148)

authored and
amitgilad3
committed
dep: update `multidict`, previous version yanked (#2157)

kevinjqliu
authored and
amitgilad3
committed
Build: Bump mypy-boto3-dynamodb from 1.38.4 to 1.39.0 (#2163)

dependabot[bot]
authored and
amitgilad3
committed
Build: Bump mypy-boto3-glue from 1.38.42 to 1.39.0 (#2164)

dependabot[bot]
authored and
amitgilad3
committed
Build: Bump huggingface-hub from 0.33.0 to 0.33.1 (#2165)

dependabot[bot]
authored and
amitgilad3
committed
Build: Bump pyroaring from 1.0.1 to 1.0.2 (#2162)

dependabot[bot]
authored and
amitgilad3
committed
Feature: Write to branches (#941)

authored and
amitgilad3
committed
Hive: update hive storage descriptor after commit schema change (#2036)

frankliee
authored and
amitgilad3
committed
fix: Propagate Table properties to Glue (#2097)

summermousa-vendia
authored and
amitgilad3
committed
Documented `row_filter` expressions (#1862)

authored and
amitgilad3
committed
Fix the CI (#2171)

Fokko
authored and
amitgilad3
committed
add iceberg datafusion integration (#2075)

kevinjqliu
authored and
amitgilad3
committed
Cleanup in `expression-dsl.md` (#2168)

Fokko
authored and
amitgilad3
committed
Docs: Condens Python API docs (#2139)

authored and
amitgilad3
committed
feat: add support for setting and removing table properties on console (#2153)

blinkseb
authored and
amitgilad3
committed
first attempt at init_manifests
amitgilad3
committed
initial implementation of rewrite_manifests
amitgilad3
committed
fix lint
amitgilad3
committed
1. support returning rewrite results
amitgilad3
committed
lint and rebase
amitgilad3
committed
linting errors
amitgilad3
committed

Rewrite manifests #1661amitgilad3 wants to merge 112 commits intoapache:mainapache/iceberg-python:mainfrom amitgilad3:rewrite_manifestsamitgilad3/iceberg-python:rewrite_manifestsCopy head branch name to clipboard

Commits

Commits on May 15, 2025

Commits on May 17, 2025

Commits on Jul 7, 2025

Rewrite manifests #1661
amitgilad3 wants to merge 112 commits intoapache:mainfrom
amitgilad3:rewrite_manifests