feat: add FilterService

[ChenZiHong-Gavin]

This pull request introduces a new filtering capability to the graphgen framework, allowing users to define and apply custom filters within their data processing pipelines.

[gemini-code-assist]

Summary of Changes

Hello @ChenZiHong-Gavin, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a new filtering capability to the graphgen framework, allowing users to define and apply custom filters within their data processing pipelines. It provides foundational abstract classes for filters, a concrete RangeFilter implementation, and integrates this new service into the existing Engine. The changes also include minor refactoring in the engine's execution logic and improved handling of potentially empty output datasets, enhancing the overall robustness and flexibility of the system.

Highlights

• New FilterService: Introduced a new FilterService to enable filtering data within the graphgen pipeline, allowing for custom filtering logic based on specified criteria.
• Base Filter Classes: Added abstract base classes (BaseFilter and BaseValueFilter) to define the interface for different types of filters, promoting extensibility.
• RangeFilter Implementation: Provided a concrete implementation, RangeFilter, which allows filtering numeric values to keep them within a specified minimum and maximum range.
• Engine Refactoring and Robustness: The core Engine has been refactored to simplify the handling of different node types (aggregate, map_batch) and improved to gracefully handle cases where output paths for datasets might be empty.
• Example Usage: Included an example script and configuration file (filter.sh and filter_config.yaml) demonstrating how to integrate and use the new FilterService with a range filter on evaluation metrics.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a new FilterService to the data processing pipeline, allowing for filtering of data based on specified metrics. The implementation includes a RangeFilter model, a FilterService operator, and base classes for filters. The execution engine has been refactored to standardize on class-based operators, simplifying the logic for node execution. Additionally, the engine is now more robust, handling cases where a processing step results in an empty dataset.

My review focuses on improving robustness and code style. I've suggested a change to prevent potential KeyErrors in the FilterService and a minor style improvement to move an import to the top level. I also pointed out a missing newline in a shell script, which is a common best practice.

[gemini-code-assist]

Directly accessing item['metrics'] can lead to a KeyError if an item in the batch is missing the 'metrics' key. To make the code more robust, it's better to use item.get('metrics', {}) to safely handle cases where the key might not be present.

Suggested change

	value = item["metrics"].get(self.metric)
	value = item.get("metrics", {}).get(self.metric)

[gemini-code-assist]

For better code readability and to follow common Python conventions, imports should be placed at the top of the file. Please move from graphgen.models import RangeFilter to the top-level of the module.