Handle query returned with empty rows gracefully on bigquery enrichmement #36791

AMOOOMA · 2025-11-11T18:48:24Z

Handle query returned with empty rows gracefully on bigquery enrichmement handler
Returns the original unenriched data if no rows are returned from the enrichment table query

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

…nt handler

gemini-code-assist · 2025-11-11T18:48:41Z

Summary of Changes

Hello @AMOOOMA, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request enhances the BigQuery enrichment handler by providing a configurable mechanism to manage situations where an enrichment query returns no data. Previously, such scenarios might lead to errors, but now users can choose to either raise an exception or gracefully return the original, unenriched data, improving the robustness and flexibility of data pipelines.

Highlights

Configurable Empty Result Handling: Introduced a new throw_execption_on_empty_results parameter in BigQueryEnrichmentHandler to control behavior when queries yield no results.
Graceful Fallback: Implemented graceful handling for BigQuery enrichment queries that return empty rows, allowing the original unenriched data to be passed through instead of raising an error when throw_execption_on_empty_results is set to False.
Improved Logging: Added logging for cases where no matching rows are found during enrichment when graceful handling is enabled, providing better visibility into data flow.
Comprehensive Testing: Expanded integration tests to cover various scenarios of empty BigQuery results, including both exception-throwing and graceful fallback mechanisms for single and batched requests, and with and without custom query functions.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

github-actions · 2025-11-11T18:59:44Z

The Workflow run is cancelling this PR. It is an earlier duplicate of 1729654 run.

github-actions · 2025-11-11T18:59:49Z

The Workflow run is cancelling this PR. It is an earlier duplicate of 2083803 run.

github-actions · 2025-11-11T21:35:41Z

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

AMOOOMA · 2025-11-11T21:56:17Z

assign set of reviewers

test failure seems to be unrelated to the changes

github-actions · 2025-11-11T21:57:23Z

Assigning reviewers:

R: @claudevdm for label python.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

claudevdm · 2025-11-12T18:49:42Z

/gemini review

gemini-code-assist

Code Review

This pull request adds a useful feature to the BigQueryEnrichmentHandler to gracefully handle cases where a query returns no results, by returning the original unenriched data. The implementation is controlled by a new throw_exception_on_empty_results parameter and is well-covered by new integration tests. My review includes a critical correction for a typo in this new parameter's name, which affects the public API, along with suggestions to improve performance and code readability.

sdks/python/apache_beam/transforms/enrichment_handlers/bigquery.py

sdks/python/apache_beam/transforms/enrichment_handlers/bigquery_it_test.py

sdks/python/apache_beam/transforms/enrichment_handlers/bigquery.py

sdks/python/apache_beam/transforms/enrichment_handlers/bigquery_it_test.py

…y_it_test.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

…y.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

…y_it_test.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

…y.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

AMOOOMA · 2025-11-12T22:10:02Z

/gemini review

gemini-code-assist

Code Review

This pull request introduces a graceful handling mechanism for empty query results in the BigQuery enrichment handler, controlled by a new throw_exception_on_empty_results parameter. The changes are well-implemented and include a comprehensive set of integration tests. My review includes a few suggestions to improve code consistency and test coverage: using f-strings for exceptions and deferred formatting for logging, fixing a typo in a test name, and adding a test case for partial batch results with exceptions enabled.

sdks/python/apache_beam/transforms/enrichment_handlers/bigquery.py

sdks/python/apache_beam/transforms/enrichment_handlers/bigquery_it_test.py

AMOOOMA · 2025-11-12T22:20:31Z

/gemini review

gemini-code-assist

Code Review

This pull request introduces a graceful handling mechanism for BigQuery enrichment queries that return no results. By adding the throw_exception_on_empty_results parameter, users can now choose between raising an exception (the default, preserving old behavior for non-batched queries) or returning the original data unenriched, which is a great improvement for flexibility. The implementation is solid, covering both batched and non-batched requests, and is well-supported by a comprehensive set of new integration tests. I've added one suggestion to refactor the logic for handling missing responses in batched mode to improve clarity and efficiency. Overall, this is a valuable enhancement to the BigQuery enrichment handler.

sdks/python/apache_beam/transforms/enrichment_handlers/bigquery.py

AMOOOMA · 2025-11-12T22:28:23Z

/gemini review

gemini-code-assist

Code Review

This pull request adds a throw_exception_on_empty_results flag to the BigQueryEnrichmentHandler to allow for graceful handling of empty query results. The changes are well-implemented and include comprehensive integration tests. My review includes suggestions to improve the efficiency of query execution for non-batched requests and to enhance logging by making it more secure and readable, avoiding potentially large and sensitive query strings in logs.

sdks/python/apache_beam/transforms/enrichment_handlers/bigquery.py

AMOOOMA · 2025-11-12T22:41:43Z

Acked some of gemini's review in the end because it seems like it's not really needed.

claudevdm

Did you to run the bigquery_it_test to validate that that they pass? They arent triggered by the precommit actions.

AMOOOMA · 2025-11-13T04:35:09Z

Yep! I ran with the latest code and they are all green

claudevdm

Thanks for the fix!

github-actions · 2025-11-13T13:16:42Z

R: @tvalentyn for final approval

Handle query returned with empty rows gracefully on bigquery enrichme…

14262a3

…nt handler

github-actions bot added the python label Nov 11, 2025

AMOOOMA added 2 commits November 11, 2025 11:27

Ran isort linter

e3aeda3

Manually update import order

9b60412

github-actions bot added the Next Action: Reviewers label Nov 11, 2025

Merge branch 'apache:master' into master

16f9bc0

gemini-code-assist bot reviewed Nov 12, 2025

View reviewed changes

AMOOOMA and others added 6 commits November 12, 2025 14:03

Update sdks/python/apache_beam/transforms/enrichment_handlers/bigquer…

8d8f156

…y_it_test.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

Update sdks/python/apache_beam/transforms/enrichment_handlers/bigquer…

50ff67b

…y.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

Update sdks/python/apache_beam/transforms/enrichment_handlers/bigquer…

be3e26b

…y_it_test.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

Update sdks/python/apache_beam/transforms/enrichment_handlers/bigquer…

ce0e533

…y.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

Update sdks/python/apache_beam/transforms/enrichment_handlers/bigquer…

25cbd6c

…y.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

fix typo

eef48ee

gemini-code-assist bot reviewed Nov 12, 2025

View reviewed changes

Updated string formatting and added a test case

a75a273

gemini-code-assist bot reviewed Nov 12, 2025

View reviewed changes

sdks/python/apache_beam/transforms/enrichment_handlers/bigquery.py Outdated Show resolved Hide resolved

Clean up logic in checking unmatched requests

1d5b2d9

gemini-code-assist bot reviewed Nov 12, 2025

View reviewed changes

sdks/python/apache_beam/transforms/enrichment_handlers/bigquery.py Show resolved Hide resolved

sdks/python/apache_beam/transforms/enrichment_handlers/bigquery.py Show resolved Hide resolved

sdks/python/apache_beam/transforms/enrichment_handlers/bigquery.py Show resolved Hide resolved

claudevdm requested changes Nov 12, 2025

View reviewed changes

claudevdm approved these changes Nov 13, 2025

View reviewed changes

tvalentyn merged commit 63a8be9 into apache:master Nov 13, 2025
99 of 101 checks passed

Handle query returned with empty rows gracefully on bigquery enrichmement #36791

Handle query returned with empty rows gracefully on bigquery enrichmement #36791

Uh oh!

Conversation

AMOOOMA commented Nov 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

GitHub Actions Tests Status (on master branch)

Uh oh!

gemini-code-assist bot commented Nov 11, 2025

Summary of Changes

Highlights

Footnotes

Uh oh!

github-actions bot commented Nov 11, 2025

Uh oh!

github-actions bot commented Nov 11, 2025

Uh oh!

github-actions bot commented Nov 11, 2025

Uh oh!

AMOOOMA commented Nov 11, 2025

Uh oh!

github-actions bot commented Nov 11, 2025

Uh oh!

claudevdm commented Nov 12, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AMOOOMA commented Nov 12, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AMOOOMA commented Nov 12, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

AMOOOMA commented Nov 12, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AMOOOMA commented Nov 12, 2025

Uh oh!

claudevdm left a comment

Choose a reason for hiding this comment

Uh oh!

AMOOOMA commented Nov 13, 2025

Uh oh!

claudevdm left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Nov 13, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

AMOOOMA commented Nov 11, 2025 •

edited

Loading