Skip to content

Conversation

@kevinjqliu
Copy link
Contributor

Rationale for this change

Older versions of google libraries throw

E   UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.

We found a temporary workaround by manually upgrading the libraries in #2127
However, other libraries might depend on these packages and will revert them back to the <2.0 versions. For example, #2145

This PR pins the top-level google-cloud-storage to >=2.0.0 to mitigate this issue.
The side effect is #2145 wont be necessary anymore since cachetools@6.1.0 wont satisfy the dependency constraints

Are these changes tested?

Are there any user-facing changes?

@Fokko
Copy link
Contributor

Fokko commented Jun 30, 2025

I'm confused by both the #2145 dependabot PR, but also this one 🤣

PyIceberg directly depends on cachetools:

cachetools = ">=5.5,<7.0"

But cachetools does not have any dependencies, so also not on google-cloud-storage. Why does it lower the GCP dependencies: https://github.com/apache/iceberg-python/pull/2145/files#diff-f53a023eedfa3fbf2925ec7dc76eecdc954ea94b7e47065393dbad519613dc89L1706-L1717 ?

Adding this constraint would put an unnecessary constraint on the users. We fail on errors, to make sure that we fix deprecations in time, but I don't think this is the right fix.

@kevinjqliu
Copy link
Contributor Author

But cachetools does not have any dependencies, so also not on google-cloud-storage. Why does it lower the GCP dependencies

I found the culprit! poetry show --tree:

├── google-cloud-storage *
│   ├── google-api-core >=2.15.0,<3.0.0 
│   │   ├── google-auth >=2.14.1,<3.0.0 
│   │   │   ├── cachetools >=2.0.0,<6.0 

google-auth sets an upper limit for cachetools. Confirmed here

So the resolver finds the version of google-auth without this constraint

├── google-cloud-storage *
│   ├── google-auth >=1.2.0 
│   │   ├── cachetools >=2.0.0 

Adding this constraint would put an unnecessary constraint on the users. We fail on errors, to make sure that we fix deprecations in time, but I don't think this is the right fix.

Agreed!

@kevinjqliu kevinjqliu closed this Jul 6, 2025
@kevinjqliu kevinjqliu deleted the kevinjqliu/pin-minimum-gcs-package branch July 6, 2025 19:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants