Merge branch 'main' into tswast-flaky-ai-generate

Author: tswast

CHANGELOG.md

@@ -4,6 +4,34 @@
 
 [1]: https://pypi.org/project/bigframes/#history
 
+## [2.25.0](https://github.com/googleapis/python-bigquery-dataframes/compare/v2.24.0...v2.25.0) (2025-10-13)
+
+
+### Features
+
+* Add barh, pie plot types ([#2146](https://github.com/googleapis/python-bigquery-dataframes/issues/2146)) ([5cc3c5b](https://github.com/googleapis/python-bigquery-dataframes/commit/5cc3c5b1391a7dfa062b1d77f001726b013f6337))
+* Add Index.__eq__ for consts, aligned objects ([#2141](https://github.com/googleapis/python-bigquery-dataframes/issues/2141)) ([8514200](https://github.com/googleapis/python-bigquery-dataframes/commit/85142008ec895fa078d192bbab942d0257f70df3))
+* Add output_schema parameter to ai.generate() ([#2139](https://github.com/googleapis/python-bigquery-dataframes/issues/2139)) ([ef0b0b7](https://github.com/googleapis/python-bigquery-dataframes/commit/ef0b0b73843da2a93baf08e4cd5457fbb590b89c))
+* Create session-scoped `cut`, `DataFrame`, `MultiIndex`, `Index`, `Series`, `to_datetime`, and `to_timedelta` methods ([#2157](https://github.com/googleapis/python-bigquery-dataframes/issues/2157)) ([5e1e809](https://github.com/googleapis/python-bigquery-dataframes/commit/5e1e8098ecf212c91d73fa80d722d1cb3e46668b))
+* Replace ML.GENERATE_TEXT with AI.GENERATE for audio transcription ([#2151](https://github.com/googleapis/python-bigquery-dataframes/issues/2151)) ([a410d0a](https://github.com/googleapis/python-bigquery-dataframes/commit/a410d0ae43ef3b053b650804156eda0b1f569da9))
+* Support string literal inputs for AI functions ([#2152](https://github.com/googleapis/python-bigquery-dataframes/issues/2152)) ([7600001](https://github.com/googleapis/python-bigquery-dataframes/commit/760000122dc190ac8a3303234cf4cbee1bbb9493))
+
+
+### Bug Fixes
+
+* Address typo in error message ([#2142](https://github.com/googleapis/python-bigquery-dataframes/issues/2142)) ([cdf2dd5](https://github.com/googleapis/python-bigquery-dataframes/commit/cdf2dd55a0c03da50ab92de09788cafac0abf6f6))
+* Avoid possible circular imports in global session ([#2115](https://github.com/googleapis/python-bigquery-dataframes/issues/2115)) ([095c0b8](https://github.com/googleapis/python-bigquery-dataframes/commit/095c0b85a25a2e51087880909597cc62a0341c93))
+* Fix too many cluster columns requested by caching ([#2155](https://github.com/googleapis/python-bigquery-dataframes/issues/2155)) ([35c1c33](https://github.com/googleapis/python-bigquery-dataframes/commit/35c1c33b85d1b92e402aab73677df3ffe43a51b4))
+* Show progress even in job optional queries ([#2119](https://github.com/googleapis/python-bigquery-dataframes/issues/2119)) ([1f48d3a](https://github.com/googleapis/python-bigquery-dataframes/commit/1f48d3a62e7e6dac4acb39e911daf766b8e2fe62))
+* Yield row count from read session if otherwise unknown ([#2148](https://github.com/googleapis/python-bigquery-dataframes/issues/2148)) ([8997d4d](https://github.com/googleapis/python-bigquery-dataframes/commit/8997d4d7d9965e473195f98c550c80657035b7e1))
+
+
+### Documentation
+
+* Add a brief intro notebook for bbq AI functions ([#2150](https://github.com/googleapis/python-bigquery-dataframes/issues/2150)) ([1f434fb](https://github.com/googleapis/python-bigquery-dataframes/commit/1f434fb5c7c00601654b3ab19c6ad7fceb258bd6))
+* Fix ai function related docs ([#2149](https://github.com/googleapis/python-bigquery-dataframes/issues/2149)) ([93a0749](https://github.com/googleapis/python-bigquery-dataframes/commit/93a0749392b84f27162654fe5ea5baa329a23f99))
+* Remove progress bar from getting started template ([#2143](https://github.com/googleapis/python-bigquery-dataframes/issues/2143)) ([d13abad](https://github.com/googleapis/python-bigquery-dataframes/commit/d13abadbcd68d03997e8dc11bb7a2b14bbd57fcc))
+
 ## [2.24.0](https://github.com/googleapis/python-bigquery-dataframes/compare/v2.23.0...v2.24.0) (2025-10-07)
 
 

bigframes/bigquery/_operations/ai.py

@@ -65,14 +65,21 @@ def generate(
         1    Ottawa\\n
         Name: result, dtype: string
 
-        You get structured output when the `output_schema` parameter is set:
+    You get structured output when the `output_schema` parameter is set:
 
         >>> animals = bpd.Series(["Rabbit", "Spider"])
         >>> bbq.ai.generate(animals, output_schema={"number_of_legs": "INT64", "is_herbivore": "BOOL"})
         0    {'is_herbivore': True, 'number_of_legs': 4, 'f...
         1    {'is_herbivore': False, 'number_of_legs': 8, '...
         dtype: struct<is_herbivore: bool, number_of_legs: int64, full_response: extension<dbjson<JSONArrowType>>, status: string>[pyarrow]
 
+    .. note::
+
+        This product or feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the
+        Service Specific Terms(https://cloud.google.com/terms/service-terms#1). Pre-GA products and features are available "as is"
+        and might have limited support. For more information, see the launch stage descriptions
+        (https://cloud.google.com/products#product-launch-stages).
+
     Args:
         prompt (str | Series | List[str|Series] | Tuple[str|Series, ...]):
             A mixture of Series and string literals that specifies the prompt to send to the model. The Series can be BigFrames Series
@@ -165,6 +172,13 @@ def generate_bool(
         2    False
         Name: result, dtype: boolean
 
+    .. note::
+
+        This product or feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the
+        Service Specific Terms(https://cloud.google.com/terms/service-terms#1). Pre-GA products and features are available "as is"
+        and might have limited support. For more information, see the launch stage descriptions
+        (https://cloud.google.com/products#product-launch-stages).
+
     Args:
         prompt (str | Series | List[str|Series] | Tuple[str|Series, ...]):
             A mixture of Series and string literals that specifies the prompt to send to the model. The Series can be BigFrames Series
@@ -240,6 +254,13 @@ def generate_int(
         2    8
         Name: result, dtype: Int64
 
+    .. note::
+
+        This product or feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the
+        Service Specific Terms(https://cloud.google.com/terms/service-terms#1). Pre-GA products and features are available "as is"
+        and might have limited support. For more information, see the launch stage descriptions
+        (https://cloud.google.com/products#product-launch-stages).
+
     Args:
         prompt (str | Series | List[str|Series] | Tuple[str|Series, ...]):
             A mixture of Series and string literals that specifies the prompt to send to the model. The Series can be BigFrames Series
@@ -315,6 +336,13 @@ def generate_double(
         2    8.0
         Name: result, dtype: Float64
 
+    .. note::
+
+        This product or feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the
+        Service Specific Terms(https://cloud.google.com/terms/service-terms#1). Pre-GA products and features are available "as is"
+        and might have limited support. For more information, see the launch stage descriptions
+        (https://cloud.google.com/products#product-launch-stages).
+
     Args:
         prompt (str | Series | List[str|Series] | Tuple[str|Series, ...]):
             A mixture of Series and string literals that specifies the prompt to send to the model. The Series can be BigFrames Series
@@ -371,6 +399,7 @@ def if_(
     provides optimization such that not all rows are evaluated with the LLM.
 
     **Examples:**
+
         >>> import bigframes.pandas as bpd
         >>> import bigframes.bigquery as bbq
         >>> bpd.options.display.progress_bar = None
@@ -386,6 +415,13 @@ def if_(
         1         Illinois
         dtype: string
 
+    .. note::
+
+        This product or feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the
+        Service Specific Terms(https://cloud.google.com/terms/service-terms#1). Pre-GA products and features are available "as is"
+        and might have limited support. For more information, see the launch stage descriptions
+        (https://cloud.google.com/products#product-launch-stages).
+
     Args:
         prompt (str | Series | List[str|Series] | Tuple[str|Series, ...]):
             A mixture of Series and string literals that specifies the prompt to send to the model. The Series can be BigFrames Series
@@ -433,6 +469,13 @@ def classify(
         <BLANKLINE>
         [2 rows x 2 columns]
 
+    .. note::
+
+        This product or feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the
+        Service Specific Terms(https://cloud.google.com/terms/service-terms#1). Pre-GA products and features are available "as is"
+        and might have limited support. For more information, see the launch stage descriptions
+        (https://cloud.google.com/products#product-launch-stages).
+
     Args:
         input (str | Series | List[str|Series] | Tuple[str|Series, ...]):
             A mixture of Series and string literals that specifies the input to send to the model. The Series can be BigFrames Series
@@ -482,6 +525,13 @@ def score(
         2    3.0
         dtype: Float64
 
+    .. note::
+
+        This product or feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the
+        Service Specific Terms(https://cloud.google.com/terms/service-terms#1). Pre-GA products and features are available "as is"
+        and might have limited support. For more information, see the launch stage descriptions
+        (https://cloud.google.com/products#product-launch-stages).
+
     Args:
         prompt (str | Series | List[str|Series] | Tuple[str|Series, ...]):
             A mixture of Series and string literals that specifies the prompt to send to the model. The Series can be BigFrames Series

bigframes/core/compile/sqlglot/aggregations/ordered_unary_compiler.py

@@ -14,11 +14,8 @@
 
 from __future__ import annotations
 
-import typing
-
 import sqlglot.expressions as sge
 
-from bigframes.core import window_spec
 import bigframes.core.compile.sqlglot.aggregations.op_registration as reg
 import bigframes.core.compile.sqlglot.expressions.typed_expr as typed_expr
 from bigframes.operations import aggregations as agg_ops
@@ -29,9 +26,35 @@
 def compile(
     op: agg_ops.WindowOp,
     column: typed_expr.TypedExpr,
-    window: typing.Optional[window_spec.WindowSpec] = None,
-    order_by: typing.Sequence[sge.Expression] = [],
+    *,
+    order_by: tuple[sge.Expression, ...],
+) -> sge.Expression:
+    return ORDERED_UNARY_OP_REGISTRATION[op](op, column, order_by=order_by)
+
+
+@ORDERED_UNARY_OP_REGISTRATION.register(agg_ops.ArrayAggOp)
+def _(
+    op: agg_ops.ArrayAggOp,
+    column: typed_expr.TypedExpr,
+    *,
+    order_by: tuple[sge.Expression, ...],
 ) -> sge.Expression:
-    return ORDERED_UNARY_OP_REGISTRATION[op](
-        op, column, window=window, order_by=order_by
-    )
+    expr = column.expr
+    if len(order_by) > 0:
+        expr = sge.Order(this=column.expr, expressions=list(order_by))
+    return sge.IgnoreNulls(this=sge.ArrayAgg(this=expr))
+
+
+@ORDERED_UNARY_OP_REGISTRATION.register(agg_ops.StringAggOp)
+def _(
+    op: agg_ops.StringAggOp,
+    column: typed_expr.TypedExpr,
+    *,
+    order_by: tuple[sge.Expression, ...],
+) -> sge.Expression:
+    expr = column.expr
+    if len(order_by) > 0:
+        expr = sge.Order(this=expr, expressions=list(order_by))
+
+    expr = sge.GroupConcat(this=expr, separator=sge.convert(op.sep))
+    return sge.func("COALESCE", expr, sge.convert(""))

bigframes/core/compile/sqlglot/aggregations/unary_compiler.py

@@ -98,6 +98,27 @@ def _(
     return apply_window_if_present(sge.func("COUNT", column.expr), window)
 
 
+@UNARY_OP_REGISTRATION.register(agg_ops.DateSeriesDiffOp)
+def _(
+    op: agg_ops.DateSeriesDiffOp,
+    column: typed_expr.TypedExpr,
+    window: typing.Optional[window_spec.WindowSpec] = None,
+) -> sge.Expression:
+    if column.dtype != dtypes.DATE_DTYPE:
+        raise TypeError(f"Cannot perform date series diff on type {column.dtype}")
+    shift_op_impl = UNARY_OP_REGISTRATION[agg_ops.ShiftOp(0)]
+    shifted = shift_op_impl(agg_ops.ShiftOp(op.periods), column, window)
+    # Conversion factor from days to microseconds
+    conversion_factor = 24 * 60 * 60 * 1_000_000
+    return sge.Cast(
+        this=sge.DateDiff(
+            this=column.expr, expression=shifted, unit=sge.Identifier(this="DAY")
+        )
+        * sge.convert(conversion_factor),
+        to="INT64",
+    )
+
+
 @UNARY_OP_REGISTRATION.register(agg_ops.DenseRankOp)
 def _(
     op: agg_ops.DenseRankOp,
@@ -151,6 +172,23 @@ def _(
     )
 
 
+@UNARY_OP_REGISTRATION.register(agg_ops.DiffOp)
+def _(
+    op: agg_ops.DiffOp,
+    column: typed_expr.TypedExpr,
+    window: typing.Optional[window_spec.WindowSpec] = None,
+) -> sge.Expression:
+    shift_op_impl = UNARY_OP_REGISTRATION[agg_ops.ShiftOp(0)]
+    shifted = shift_op_impl(agg_ops.ShiftOp(op.periods), column, window)
+    if column.dtype in (dtypes.BOOL_DTYPE, dtypes.INT_DTYPE, dtypes.FLOAT_DTYPE):
+        if column.dtype == dtypes.BOOL_DTYPE:
+            return sge.NEQ(this=column.expr, expression=shifted)
+        else:
+            return sge.Sub(this=column.expr, expression=shifted)
+    else:
+        raise TypeError(f"Cannot perform diff on type {column.dtype}")
+
+
 @UNARY_OP_REGISTRATION.register(agg_ops.MaxOp)
 def _(
     op: agg_ops.MaxOp,
@@ -240,6 +278,27 @@ def _(
     return apply_window_if_present(sge.func("COUNT", sge.convert(1)), window)
 
 
+@UNARY_OP_REGISTRATION.register(agg_ops.ShiftOp)
+def _(
+    op: agg_ops.ShiftOp,
+    column: typed_expr.TypedExpr,
+    window: typing.Optional[window_spec.WindowSpec] = None,
+) -> sge.Expression:
+    if op.periods == 0:  # No-op
+        return column.expr
+    if op.periods > 0:
+        return apply_window_if_present(
+            sge.func("LAG", column.expr, sge.convert(op.periods)),
+            window,
+            include_framing_clauses=False,
+        )
+    return apply_window_if_present(
+        sge.func("LEAD", column.expr, sge.convert(-op.periods)),
+        window,
+        include_framing_clauses=False,
+    )
+
+
 @UNARY_OP_REGISTRATION.register(agg_ops.SumOp)
 def _(
     op: agg_ops.SumOp,
@@ -255,3 +314,20 @@ def _(
     # Will be null if all inputs are null. Pandas defaults to zero sum though.
     zero = pd.to_timedelta(0) if column.dtype == dtypes.TIMEDELTA_DTYPE else 0
     return sge.func("IFNULL", expr, ir._literal(zero, column.dtype))
+
+
+@UNARY_OP_REGISTRATION.register(agg_ops.TimeSeriesDiffOp)
+def _(
+    op: agg_ops.TimeSeriesDiffOp,
+    column: typed_expr.TypedExpr,
+    window: typing.Optional[window_spec.WindowSpec] = None,
+) -> sge.Expression:
+    if column.dtype != dtypes.TIMESTAMP_DTYPE:
+        raise TypeError(f"Cannot perform time series diff on type {column.dtype}")
+    shift_op_impl = UNARY_OP_REGISTRATION[agg_ops.ShiftOp(0)]
+    shifted = shift_op_impl(agg_ops.ShiftOp(op.periods), column, window)
+    return sge.TimestampDiff(
+        this=column.expr,
+        expression=shifted,
+        unit=sge.Identifier(this="MICROSECOND"),
+    )

bigframes/version.py

@@ -12,8 +12,8 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-__version__ = "2.24.0"
+__version__ = "2.25.0"
 
 # {x-release-please-start-date}
-__release_date__ = "2025-10-07"
+__release_date__ = "2025-10-13"
 # {x-release-please-end}

docs/templates/toc.yml

@@ -219,7 +219,8 @@
     - name: BigQuery built-in functions
       uid: bigframes.bigquery
     - name: BigQuery AI Functions
-      uid: bigframes.bigquery.ai
+      uid: bigframes.bigquery._operations.ai
+      status: beta
     name: bigframes.bigquery
   - items:
     - name: GeoSeries

notebooks/generative_ai/ai_functions.ipynb

@@ -56,7 +56,7 @@
    "id": "aee05821",
    "metadata": {},
    "source": [
-    "This notebook provides a brief introduction to how to use BigFrames AI functions"
+    "This notebook provides a brief introduction to AI functions in BigQuery Dataframes."
    ]
   },
   {
@@ -145,7 +145,7 @@
    "id": "b606c51f",
    "metadata": {},
    "source": [
-    "You can also include additional model parameters into your function call, as long as they satisfy the structure of `generateContent` [request body format](https://cloud.google.com/vertex-ai/docs/reference/rest/v1/projects.locations.endpoints/generateContent#request-body). In the next example, you use `maxOutputTokens` to limite the length of the generated content."
+    "You can also include additional model parameters into your function call, as long as they conform to the structure of `generateContent` [request body format](https://cloud.google.com/vertex-ai/docs/reference/rest/v1/projects.locations.endpoints/generateContent#request-body). In the next example, you use `maxOutputTokens` to limit the length of the generated content."
    ]
   },
   {
@@ -186,7 +186,7 @@
    "source": [
     "The answers are cut short as expected.\n",
     "\n",
-    "In addition to `ai.generate`, you can use `ai.generate_bool`, `ai.generate_int`, and `ai.generate_double` for other type of outputs."
+    "In addition to `ai.generate`, you can use `ai.generate_bool`, `ai.generate_int`, and `ai.generate_double` for other output types."
    ]
   },
   {
@@ -196,7 +196,7 @@
    "source": [
     "## ai.if_\n",
     "\n",
-    "`ai.if_` generates a series of booleans, unlike `ai.generate_bool` where you get a series of structs. It's a handy tool for filtering your data. not only because it directly returns a boolean, but also because it provides more optimization during data processing. Here is an example of using `ai.if_`:"
+    "`ai.if_` generates a series of booleans. It's a handy tool for joining and filtering your data, not only because it directly returns boolean values, but also because it provides more optimization during data processing. Here is an example of using `ai.if_`:"
    ]
   },
   {
@@ -284,7 +284,7 @@
    "id": "63b5a59f",
    "metadata": {},
    "source": [
-    "`ai.score` ranks your input based on the prompt. You can then sort your data based on their ranks. For example:"
+    "`ai.score` ranks your input based on the prompt and assigns a double value (i.e. a score) to each item. You can then sort your data based on their scores. For example:"
    ]
   },
   {
@@ -460,7 +460,7 @@
    "id": "9e4037bc",
    "metadata": {},
    "source": [
-    "Note that this function can only return the values that are present in your provided categories. If your categories do not cover all cases, your will get wrong answers:"
+    "Note that this function can only return the values that are provided in the `categories` argument. If your categories do not cover all cases, your may get wrong answers:"
    ]
   },
   {

tests/unit/core/compile/sqlglot/aggregations/snapshots/test_ordered_unary_compiler/test_array_agg/out.sql

@@ -0,0 +1,12 @@
+WITH `bfcte_0` AS (
+  SELECT
+    `int64_col` AS `bfcol_0`
+  FROM `bigframes-dev`.`sqlglot_test`.`scalar_types`
+), `bfcte_1` AS (
+  SELECT
+    ARRAY_AGG(`bfcol_0` IGNORE NULLS ORDER BY `bfcol_0` IS NULL ASC, `bfcol_0` ASC) AS `bfcol_1`
+  FROM `bfcte_0`
+)
+SELECT
+  `bfcol_1` AS `int64_col`
+FROM `bfcte_1`