Adding support for Google AuthManager #2072

rambleraptor · 2025-06-07T18:18:59Z

Rationale for this change

Matches this Java PR

Google's authentication for BigQuery involves using different authentication methods than the current AuthManagers support. This allows users to authenticate against the BigLake REST Catalog (formerly known as the BigQuery REST Catalog)

The GoogleAuthManager introduces authentication methods that are more convenient for GCP users, differing from the generic OAuth2 flow:

Application Default Credentials (ADC) Support: Provides out-of-the-box, zero-configuration authentication when the Iceberg client runs within GCP environments (e.g., Compute Engine, GKE, Cloud Functions). This is a significant usability improvement over manually setting up OAuth2 parameters.
Service Account Key File Support: Allows users to authenticate by directly providing a path to a GCP service account JSON key file. This is a common and secure way to authenticate applications with GCP services, and is more specific than the generic token or client secret mechanisms.
Configurable OAuth Scopes: While the generic OAuth2Manager also supports scopes, GoogleAuthManager defaults to scopes commonly used for GCP services and allows easy customization via the gcp.auth.scopes property.

Are these changes tested?

Unit tests included.

Are there any user-facing changes?

Adds support for Google authentication

Fokko

Thanks for adding this @rambleraptor 🙌

I'm not too familiar with the Google Auth system. WDYT @talatuyarer?

pyiceberg/catalog/rest/auth.py

talatuyarer · 2025-06-12T00:52:12Z

I'm not too familiar with the Google Auth system. WDYT @talatuyarer?

Let me try the PR on my local machine @Fokko

Fokko · 2025-07-03T09:10:19Z

@talatuyarer Any luck? :)

pyiceberg/catalog/rest/auth.py

talatuyarer · 2025-07-25T19:40:05Z

@rambleraptor Could you update your PR with latest master changes

rambleraptor · 2025-07-25T20:22:05Z

@talatuyarer changes updated to main branch!

Copilot

Pull Request Overview

This PR adds Google authentication support to the Iceberg Python client by implementing a GoogleAuthManager class, matching functionality from a corresponding Java PR. The implementation provides convenient authentication methods for GCP users including Application Default Credentials (ADC) support and service account key file authentication.

Implements GoogleAuthManager with support for default credentials and service account key files
Adds comprehensive unit tests covering different authentication scenarios
Configures optional google-auth dependency and mypy settings

Reviewed Changes

Copilot reviewed 3 out of 4 changed files in this pull request and generated 3 comments.

File	Description
pyiceberg/catalog/rest/auth.py	Implements GoogleAuthManager class with Google authentication support
tests/catalog/test_rest_auth.py	Adds comprehensive unit tests for GoogleAuthManager functionality
pyproject.toml	Adds optional google-auth dependency and mypy configuration

pyproject.toml

pyiceberg/catalog/rest/auth.py

kevinjqliu

LGTM! (and to copilot)

kevinjqliu · 2025-07-25T23:23:14Z

@talatuyarer could you take another look before I merge?

kevinjqliu · 2025-07-30T17:39:03Z

@rambleraptor looks like there a merge conflict with main

Fokko

This looks great @rambleraptor

talatuyarer

@rambleraptor I tested this PR. It is working as expected. Please update documentation. Beside of that LGTM!

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

rambleraptor · 2025-08-04T20:09:09Z

@Fokko @kevinjqliu @talatuyarer docs updated + merge conflict removed!

sungwy

awesome work @rambleraptor ! I'm running the CI, I will merge it once the CI passes

kevinjqliu

LGTM!

kevinjqliu · 2025-08-04T20:30:45Z

pyiceberg/catalog/rest/auth.py

+        self._auth_request = google.auth.transport.requests.Request()
+
+    def auth_header(self) -> str:
+        self.credentials.refresh(self._auth_request)


nit: is this .refresh called on every request?

Fokko reviewed Jun 10, 2025

View reviewed changes

pyiceberg/catalog/rest/auth.py Outdated Show resolved Hide resolved

Fokko requested a review from sungwy June 10, 2025 09:37

sungwy mentioned this pull request Jun 16, 2025

GCP: Add Google Authentication Support apache/iceberg#13212

Merged

talatuyarer reviewed Jul 22, 2025

View reviewed changes

pyiceberg/catalog/rest/auth.py Show resolved Hide resolved

kevinjqliu mentioned this pull request Jul 25, 2025

Add Support for Custom AuthManager implementation #2055

Merged

2 tasks

rambleraptor force-pushed the auth_manager branch from 39f5f22 to 1567dca Compare July 25, 2025 20:21

kevinjqliu requested a review from Copilot July 25, 2025 22:51

Copilot AI reviewed Jul 25, 2025

View reviewed changes

pyproject.toml Outdated Show resolved Hide resolved

pyiceberg/catalog/rest/auth.py Outdated Show resolved Hide resolved

pyiceberg/catalog/rest/auth.py Outdated Show resolved Hide resolved

kevinjqliu approved these changes Jul 25, 2025

View reviewed changes

kevinjqliu requested a review from talatuyarer July 25, 2025 23:23

kevinjqliu added this to the PyIceberg 0.10.0 milestone Jul 25, 2025

rambleraptor force-pushed the auth_manager branch from b7222b9 to c915752 Compare July 30, 2025 16:16

Fokko approved these changes Aug 4, 2025

View reviewed changes

talatuyarer approved these changes Aug 4, 2025

View reviewed changes

rambleraptor and others added 7 commits August 4, 2025 13:05

add google auth and test

ced235c

add unit tests

7e29ae1

lint changes

f366ff6

Update pyproject.toml

a51e55a

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update pyiceberg/catalog/rest/auth.py

d1aac67

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

use logger

5ca9dcb

add extra test

c958c5f

rambleraptor force-pushed the auth_manager branch from c915752 to c958c5f Compare August 4, 2025 20:06

docs

3f0c0c0

sungwy approved these changes Aug 4, 2025

View reviewed changes

kevinjqliu approved these changes Aug 4, 2025

View reviewed changes

sungwy merged commit 632d783 into apache:main Aug 4, 2025
11 checks passed

rambleraptor mentioned this pull request Aug 7, 2025

Expose a way for users to use a custom AuthManager when they define their REST Catalog #1960

Closed

gabeiglio pushed a commit to Netflix/iceberg-python that referenced this pull request Aug 13, 2025

Adding support for Google AuthManager (apache#2072)

01f53b0

BearDimonR mentioned this pull request Sep 11, 2025

Add GCP Service Account auth for Iceberg on GCS dlt-hub/dlt#2227

Closed

Adding support for Google AuthManager #2072

Adding support for Google AuthManager #2072

Uh oh!

Conversation

rambleraptor commented Jun 7, 2025

Rationale for this change

Are these changes tested?

Are there any user-facing changes?

Uh oh!

Fokko left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

talatuyarer commented Jun 12, 2025

Uh oh!

Fokko commented Jul 3, 2025

Uh oh!

Uh oh!

talatuyarer commented Jul 25, 2025

Uh oh!

rambleraptor commented Jul 25, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kevinjqliu left a comment

Choose a reason for hiding this comment

Uh oh!

kevinjqliu commented Jul 25, 2025

Uh oh!

kevinjqliu commented Jul 30, 2025

Uh oh!

Fokko left a comment

Choose a reason for hiding this comment

Uh oh!

talatuyarer left a comment

Choose a reason for hiding this comment

Uh oh!

rambleraptor commented Aug 4, 2025

Uh oh!

sungwy left a comment

Choose a reason for hiding this comment

Uh oh!

kevinjqliu left a comment

Choose a reason for hiding this comment

Uh oh!

kevinjqliu Aug 4, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants