-
Notifications
You must be signed in to change notification settings - Fork 413
add support to configure anonymous in s3 clients #2323
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @gmweaver for adding this 🙌
Probably we want to add this to the docs as well: https://github.com/apache/iceberg-python/blob/main/mkdocs/docs/configuration.md#s3
Do you know if there is a PyArrow equialent? If so, it would be nice to add that option as well.
pyarrow has a similar config updated the PR to include. |
|
@gmweaver Thanks for being on top of this, could you run |
|
@Fokko sorry missed the |
kevinjqliu
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM! Thanks for adding this. Heres the docs for both fsspec and pyarrow
https://s3fs.readthedocs.io/en/latest/api.html
https://arrow.apache.org/docs/python/generated/pyarrow.fs.S3FileSystem.html
|
Thanks for the PR @gmweaver and thank you @Fokko for the review! |
Rationale for this change
Currently, it is not possible to configure
anonymousin the underlyings3fsclient (Falseby default).Resolves #2126
Are these changes tested?
Yes, unit tests added to confirm the setting is used when the property is set.
Are there any user-facing changes?
Yes, users can now set the property
s3.anonymousto True to enable. If not set, the behavior will be the same as it currently is.