From 59cc515bf2ed9562ab4d9b98c6f4700a2d986a81 Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Thu, 28 Aug 2025 23:49:35 +0000 Subject: [PATCH 01/10] A103: xDS Composite Filter --- A103-xds-composite-filter.md | 127 +++++++++++++++++++++++++++++++++++ 1 file changed, 127 insertions(+) create mode 100644 A103-xds-composite-filter.md diff --git a/A103-xds-composite-filter.md b/A103-xds-composite-filter.md new file mode 100644 index 000000000..a5456bb1d --- /dev/null +++ b/A103-xds-composite-filter.md @@ -0,0 +1,127 @@ +A103: xDS Composite Filter +---- +* Author(s): markdroth +* Approver: ejona86, dfawley +* Status: {Draft, In Review, Ready for Implementation, Implemented} +* Implemented in: +* Last updated: 2025-08-28 +* Discussion at: (filled after thread exists) + +## Abstract + +gRPC will support the [xDS Composite filter][composite], which is a +"wrapper" filter that dynamically determines which filter to use based +on request attributes. + +[composite]: https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/composite_filter + +## Background + +xDS support in the gRPC client and server are described in [A27] and +[A36], respectively. xDS HTTP filter support is described in [A39]. + +The composite filter will make use of the Unified Matching API and CEL +support described in [A77]. + +### Related Proposals: +* [A27: xDS-Based Global Load Balancing][A27] +* [A39: xDS HTTP Filter Support][A39] +* [A36: xDS-Enabled Servers][A36] +* [A77: xDS Server-Side Rate Limiting][A77] (pending) + +[A27]: A27-xds-global-load-balancing.md +[A36]: A36-xds-for-servers.md +[A39]: A39-xds-http-filters.md +[A77]: https://github.com/grpc/proposal/pull/414 + +## Proposal + +We will support the composite filter in both the gRPC client and gRPC +server. + +### xDS Resource Validation + +Today, the composite filter supports configuring only one filter as a +result of the matching tree. However, we have use-cases where we need +to select a chain of more than one filter based on the matching tree. +As a result, we are proposing a change to the composite filter's config +to allow selecting a chain of filters +(https://github.com/envoyproxy/envoy/pull/40885). + +The composite filter is configured via the +[`envoy.extensions.common.matching.v3.ExtensionWithMatcher` +proto](https://github.com/envoyproxy/envoy/blob/0685d7bf568485eb112df2a9c73248cb8bfc1c37/api/envoy/extensions/common/matching/v3/extension_matcher.proto#L25). +Within it, gRPC will look at the following fields: +- [extension_config](https://github.com/envoyproxy/envoy/blob/0685d7bf568485eb112df2a9c73248cb8bfc1c37/api/envoy/extensions/common/matching/v3/extension_matcher.proto#L34C39-L34C55): + This must contain a + [`envoy.extensions.filters.http.composite.v3.Composite` + proto](https://github.com/envoyproxy/envoy/blob/0685d7bf568485eb112df2a9c73248cb8bfc1c37/api/envoy/extensions/filters/http/composite/v3/composite.proto#L33), + which has no fields. +- [xds_matcher](https://github.com/envoyproxy/envoy/blob/0685d7bf568485eb112df2a9c73248cb8bfc1c37/api/envoy/extensions/common/matching/v3/extension_matcher.proto#L31): + Specifies a unified matcher tree indicating the config of the filter + to use. The actions in this tree must be one of two types: + - [`envoy.extensions.filters.common.matcher.action.v3.SkipFilter`](https://github.com/envoyproxy/envoy/blob/0685d7bf568485eb112df2a9c73248cb8bfc1c37/api/envoy/extensions/filters/common/matcher/action/v3/skip_action.proto#L24): + This indicates that the no filter will be executed. + - [`envoy.extensions.filters.http.composite.v3.ExecuteFilterAction`](https://github.com/envoyproxy/envoy/blob/0685d7bf568485eb112df2a9c73248cb8bfc1c37/api/envoy/extensions/filters/http/composite/v3/composite.proto#L49): + This indicates which filter should be executed. Within it: + - [typed_config](https://github.com/envoyproxy/envoy/blob/0685d7bf568485eb112df2a9c73248cb8bfc1c37/api/envoy/extensions/filters/http/composite/v3/composite.proto#L54C39-L54C51): + The filter to configure. It will be validated using the xDS HTTP filter + registry, just as the top-level filters in the HTTP connection manager + config are. This field is ignored if `filter_chain` is set. + - [dynamic_config](https://github.com/envoyproxy/envoy/blob/0685d7bf568485eb112df2a9c73248cb8bfc1c37/api/envoy/extensions/filters/http/composite/v3/composite.proto#L59): + TODO: Do we need to support ECDS here? Also need to define + precedence rules. + - filter_chain (new field added in + https://github.com/envoyproxy/envoy/pull/40885): If set, the + `typed_config` field is ignored. This specifies a chain of + filters to call, in order. Each filter in the chain will be + validated using the xDS HTTP filterregistry, just as the top-level + filters in the HTTP connection manager config are. + - [sample_percent](https://github.com/envoyproxy/envoy/blob/0685d7bf568485eb112df2a9c73248cb8bfc1c37/api/envoy/extensions/filters/http/composite/v3/composite.proto#L69C43-L69C57): + - Optional; if unset, the specified filter(s) are always executed. + If set, for each RPC, a random number will be generated between + 0 and 100, and if that number is less than the specified + threshold, the specified filter(s) will be executed. + Within this field: + - [default_value](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/config/core/v3/base.proto#L648): + This field must be present. The configured value will be capped at + 100%. + - runtime_key: This field will be ignored, since gRPC does not + have a runtime system. +- matcher: gRPC will not support this deprecated field. + +We will also support per-route overrides via the +[`envoy.extensions.common.matching.v3.ExtensionWithMatcherPerRoute` +proto](https://github.com/envoyproxy/envoy/blob/0685d7bf568485eb112df2a9c73248cb8bfc1c37/api/envoy/extensions/common/matching/v3/extension_matcher.proto#L39C9-L39C37). +In this proto, the +[xds_matcher](https://github.com/envoyproxy/envoy/blob/0685d7bf568485eb112df2a9c73248cb8bfc1c37/api/envoy/extensions/common/matching/v3/extension_matcher.proto#L41) +must be validated the same way as the corresponding field in the +top-level config. The value of this field will replace the value of the +field in the top-level config. + +### CEL Attributes + +In addition to the CEL request attributes described in [A77], we will +also add support for the following [configuration +attributes](https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/advanced/attributes.html#configuration-attributes): +- `xds.route_metadata` +- TODO: others? + +### Temporary environment variable protection + +Support for the composite filter will be guarded by the +`GRPC_EXPERIMENTAL_XDS_COMPOSITE_FILTER` environment variable. This +guard will be removed once the feature passes interop tests. + +## Rationale + +The composite filter API seems a little unusual. One would naively +have expected it to be structured by having the matcher tree live +in the config for the composite filter directly, rather than using +`ExtensionWithMatcher`. Unfortunately, that's not the way the API evolved +in Envoy, so we'll stick with what already exists for compatibility +reasons. + +## Implementation + +Will be implemented in C-core, Java, Go, and Node. From 4412a75621c5448e56802c46292118a3690526cc Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Fri, 29 Aug 2025 00:20:38 +0000 Subject: [PATCH 02/10] more CEL fields --- A103-xds-composite-filter.md | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/A103-xds-composite-filter.md b/A103-xds-composite-filter.md index a5456bb1d..f7e937230 100644 --- a/A103-xds-composite-filter.md +++ b/A103-xds-composite-filter.md @@ -104,8 +104,16 @@ field in the top-level config. In addition to the CEL request attributes described in [A77], we will also add support for the following [configuration attributes](https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/advanced/attributes.html#configuration-attributes): -- `xds.route_metadata` -- TODO: others? +(TODO: document these fields!) +- `xds.route_metadata`: TODO: tie to metadata registry from A83 +- `source.ip`: TODO: behavior on client side? +- `source.port`: TODO: behavior on client side? +- `connection.requested_server_name`: TODO: behavior on client side? Do + we have access to this data on server side? +- `connection.tls_version`: TODO: behavior on client side? Do we have + access to this data on server side? +- `connection.sha256_peer_certificate_digest`: TODO: behavior on client + side? Do we have access to this data on server side? ### Temporary environment variable protection From 32c8b04b4eb8dc9173ff81833ac32c64504e8ad1 Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Sat, 30 Aug 2025 00:04:57 +0000 Subject: [PATCH 03/10] more details --- A103-xds-composite-filter.md | 62 ++++++++++++++++++++++++------------ 1 file changed, 41 insertions(+), 21 deletions(-) diff --git a/A103-xds-composite-filter.md b/A103-xds-composite-filter.md index f7e937230..3dfbc26e0 100644 --- a/A103-xds-composite-filter.md +++ b/A103-xds-composite-filter.md @@ -4,7 +4,7 @@ A103: xDS Composite Filter * Approver: ejona86, dfawley * Status: {Draft, In Review, Ready for Implementation, Implemented} * Implemented in: -* Last updated: 2025-08-28 +* Last updated: 2025-08-29 * Discussion at: (filled after thread exists) ## Abstract @@ -28,11 +28,13 @@ support described in [A77]. * [A39: xDS HTTP Filter Support][A39] * [A36: xDS-Enabled Servers][A36] * [A77: xDS Server-Side Rate Limiting][A77] (pending) +* [A83: xDS GCP Authentication Filter][A83] [A27]: A27-xds-global-load-balancing.md [A36]: A36-xds-for-servers.md [A39]: A39-xds-http-filters.md [A77]: https://github.com/grpc/proposal/pull/414 +[A83]: A83-xds-gcp-authn-filter.md ## Proposal @@ -67,16 +69,18 @@ Within it, gRPC will look at the following fields: - [typed_config](https://github.com/envoyproxy/envoy/blob/0685d7bf568485eb112df2a9c73248cb8bfc1c37/api/envoy/extensions/filters/http/composite/v3/composite.proto#L54C39-L54C51): The filter to configure. It will be validated using the xDS HTTP filter registry, just as the top-level filters in the HTTP connection manager - config are. This field is ignored if `filter_chain` is set. + config are. This field is ignored if `filter_chain` is set. It + is an error if neither `typed_config` nor `filter_chain` are set. - [dynamic_config](https://github.com/envoyproxy/envoy/blob/0685d7bf568485eb112df2a9c73248cb8bfc1c37/api/envoy/extensions/filters/http/composite/v3/composite.proto#L59): - TODO: Do we need to support ECDS here? Also need to define - precedence rules. + This field will be ignored for now, since gRPC does not currently + support ECDS. Support for ECDS will be added in a subsequent gRFC. - filter_chain (new field added in - https://github.com/envoyproxy/envoy/pull/40885): If set, the - `typed_config` field is ignored. This specifies a chain of - filters to call, in order. Each filter in the chain will be - validated using the xDS HTTP filterregistry, just as the top-level - filters in the HTTP connection manager config are. + https://github.com/envoyproxy/envoy/pull/40885): This specifies a + chain of filters to call, in order. Each filter in the chain will + be validated using the xDS HTTP filter registry, just as the + top-level filters in the HTTP connection manager config are. + If set, the `typed_config` field is ignored. It is an error if + neither `typed_config` nor `filter_chain` are set. - [sample_percent](https://github.com/envoyproxy/envoy/blob/0685d7bf568485eb112df2a9c73248cb8bfc1c37/api/envoy/extensions/filters/http/composite/v3/composite.proto#L69C43-L69C57): - Optional; if unset, the specified filter(s) are always executed. If set, for each RPC, a random number will be generated between @@ -99,21 +103,37 @@ must be validated the same way as the corresponding field in the top-level config. The value of this field will replace the value of the field in the top-level config. +The parsed representation of the composite filter's config will be a +matcher tree, using the same unified matcher API implemented for [A77]. +The actions in the matcher tree will be parsed filter configs. + ### CEL Attributes In addition to the CEL request attributes described in [A77], we will -also add support for the following [configuration -attributes](https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/advanced/attributes.html#configuration-attributes): -(TODO: document these fields!) -- `xds.route_metadata`: TODO: tie to metadata registry from A83 -- `source.ip`: TODO: behavior on client side? -- `source.port`: TODO: behavior on client side? -- `connection.requested_server_name`: TODO: behavior on client side? Do - we have access to this data on server side? -- `connection.tls_version`: TODO: behavior on client side? Do we have - access to this data on server side? -- `connection.sha256_peer_certificate_digest`: TODO: behavior on client - side? Do we have access to this data on server side? +also add support for some additional [CEL +attributes](https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/advanced/attributes.html#configuration-attributes) +that we expect to be useful for the composite filter. + +On both the gRPC client and server sides, we will add support for the +`xds.route_metadata` attribute. To support this, we will add support +for parsing the [`Route.metadata` +field](https://github.com/envoyproxy/envoy/blob/f384ab2b3e3aa0564ef25f57dc2ed8ad61eaf0cb/api/envoy/config/route/v3/route_components.proto#L319) +in the xDS RouteConfiguration. This field will be validated the same +way as cluster metadata, as described in [A83]. The parsed metadata map +will be added to the route in the parsed RouteConfiguration resource, +and that map will be accessed by this CEL attribute. + +TODO: from CEL's perspective, the `xds.route_metadata` attribute is +supposed to be an xDS metadata proto. how do we make that work without +depending on protobuf? + +We will also add support for the following attributes on the gRPC server +side only (these attributes are not relevant on the client side): +- `source.ip` +- `source.port` +- `connection.requested_server_name` +- `connection.tls_version` +- `connection.sha256_peer_certificate_digest` ### Temporary environment variable protection From b8987fb29a895619b7a1b14d72105d9743299a06 Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Sat, 30 Aug 2025 00:20:15 +0000 Subject: [PATCH 04/10] matcher tree encodes skip actions --- A103-xds-composite-filter.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/A103-xds-composite-filter.md b/A103-xds-composite-filter.md index 3dfbc26e0..d0a868968 100644 --- a/A103-xds-composite-filter.md +++ b/A103-xds-composite-filter.md @@ -105,7 +105,9 @@ field in the top-level config. The parsed representation of the composite filter's config will be a matcher tree, using the same unified matcher API implemented for [A77]. -The actions in the matcher tree will be parsed filter configs. +The actions in the matcher tree will be one of two possible values: a +parsed filter config, or an indication that the filter should be skipped +(in the case of a `SkipFilter` proto). ### CEL Attributes From c93221303ae80afc1559e0403b1c4bc8c042810c Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Fri, 5 Sep 2025 00:57:54 +0000 Subject: [PATCH 05/10] recursion limit --- A103-xds-composite-filter.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/A103-xds-composite-filter.md b/A103-xds-composite-filter.md index d0a868968..33ab1466f 100644 --- a/A103-xds-composite-filter.md +++ b/A103-xds-composite-filter.md @@ -4,7 +4,7 @@ A103: xDS Composite Filter * Approver: ejona86, dfawley * Status: {Draft, In Review, Ready for Implementation, Implemented} * Implemented in: -* Last updated: 2025-08-29 +* Last updated: 2025-09-04 * Discussion at: (filled after thread exists) ## Abstract @@ -109,6 +109,9 @@ The actions in the matcher tree will be one of two possible values: a parsed filter config, or an indication that the filter should be skipped (in the case of a `SkipFilter` proto). +Note that in order to avoid potential stack overflows, we will impose +a maximum recursion depth of 8 when parsing HTTP filter configs. + ### CEL Attributes In addition to the CEL request attributes described in [A77], we will From 94c4bfbdd170535d2b441452b8e2d72b26213203 Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Fri, 5 Sep 2025 01:18:48 +0000 Subject: [PATCH 06/10] support only filter_metadata, not typed_filter_metadata --- A103-xds-composite-filter.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/A103-xds-composite-filter.md b/A103-xds-composite-filter.md index 33ab1466f..e941029e5 100644 --- a/A103-xds-composite-filter.md +++ b/A103-xds-composite-filter.md @@ -120,17 +120,17 @@ attributes](https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/adva that we expect to be useful for the composite filter. On both the gRPC client and server sides, we will add support for the -`xds.route_metadata` attribute. To support this, we will add support -for parsing the [`Route.metadata` +`xds.route_metadata.filter_metadata` attribute. To support this, we will +add support for parsing the [`Route.metadata` field](https://github.com/envoyproxy/envoy/blob/f384ab2b3e3aa0564ef25f57dc2ed8ad61eaf0cb/api/envoy/config/route/v3/route_components.proto#L319) in the xDS RouteConfiguration. This field will be validated the same way as cluster metadata, as described in [A83]. The parsed metadata map will be added to the route in the parsed RouteConfiguration resource, -and that map will be accessed by this CEL attribute. - -TODO: from CEL's perspective, the `xds.route_metadata` attribute is -supposed to be an xDS metadata proto. how do we make that work without -depending on protobuf? +and that map will be accessed by this CEL attribute. Note that we will +support only `filter_metadata`, not `typed_filter_metadata`, so that we +do not have to handle protobuf descriptor functionality; to that end, we +will use only those entries in the parsed metadata map that correspond +to `google.protobuf.Struct` type. We will also add support for the following attributes on the gRPC server side only (these attributes are not relevant on the client side): From 4d4f41a236fb8429b784cc63aa6390d963e50ba6 Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Fri, 5 Sep 2025 01:23:38 +0000 Subject: [PATCH 07/10] add filter behavior section --- A103-xds-composite-filter.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/A103-xds-composite-filter.md b/A103-xds-composite-filter.md index e941029e5..3afd296a9 100644 --- a/A103-xds-composite-filter.md +++ b/A103-xds-composite-filter.md @@ -140,6 +140,17 @@ side only (these attributes are not relevant on the client side): - `connection.tls_version` - `connection.sha256_peer_certificate_digest` +### Filter Behavior + +Because all of the CEL attributes that we are exposing are available +when we see the client's initial metadata, that is the point at which +the filter will evaluate the matcher tree and decide which filter chain +to use. After that point, all other filter hooks will be delegated to +the chosen filter chain. + +If we ever in the future need to support other attributes that are not +yet available at this point, we will need to make changes here. + ### Temporary environment variable protection Support for the composite filter will be guarded by the From 13868f62d3a2821b1a93f45c7a8fd02a10d7ad95 Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Tue, 16 Sep 2025 00:24:09 +0000 Subject: [PATCH 08/10] fix typo --- A103-xds-composite-filter.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/A103-xds-composite-filter.md b/A103-xds-composite-filter.md index 3afd296a9..6afe48666 100644 --- a/A103-xds-composite-filter.md +++ b/A103-xds-composite-filter.md @@ -63,7 +63,7 @@ Within it, gRPC will look at the following fields: Specifies a unified matcher tree indicating the config of the filter to use. The actions in this tree must be one of two types: - [`envoy.extensions.filters.common.matcher.action.v3.SkipFilter`](https://github.com/envoyproxy/envoy/blob/0685d7bf568485eb112df2a9c73248cb8bfc1c37/api/envoy/extensions/filters/common/matcher/action/v3/skip_action.proto#L24): - This indicates that the no filter will be executed. + This indicates that no filter will be executed. - [`envoy.extensions.filters.http.composite.v3.ExecuteFilterAction`](https://github.com/envoyproxy/envoy/blob/0685d7bf568485eb112df2a9c73248cb8bfc1c37/api/envoy/extensions/filters/http/composite/v3/composite.proto#L49): This indicates which filter should be executed. Within it: - [typed_config](https://github.com/envoyproxy/envoy/blob/0685d7bf568485eb112df2a9c73248cb8bfc1c37/api/envoy/extensions/filters/http/composite/v3/composite.proto#L54C39-L54C51): From c6bbc1e6c050936e6bcd0717bfae39d2ccbc1e28 Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Thu, 18 Sep 2025 22:58:43 +0000 Subject: [PATCH 09/10] add mailing list link --- A103-xds-composite-filter.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/A103-xds-composite-filter.md b/A103-xds-composite-filter.md index 6afe48666..9f1f39376 100644 --- a/A103-xds-composite-filter.md +++ b/A103-xds-composite-filter.md @@ -5,7 +5,7 @@ A103: xDS Composite Filter * Status: {Draft, In Review, Ready for Implementation, Implemented} * Implemented in: * Last updated: 2025-09-04 -* Discussion at: (filled after thread exists) +* Discussion at: https://groups.google.com/g/grpc-io/c/es5taH0OZS8 ## Abstract From 27fd0ad7ba714d343f3e03113649d815a81fb983 Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Mon, 1 Dec 2025 23:14:55 +0000 Subject: [PATCH 10/10] fix attribute name --- A103-xds-composite-filter.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/A103-xds-composite-filter.md b/A103-xds-composite-filter.md index 9f1f39376..0d694b8fc 100644 --- a/A103-xds-composite-filter.md +++ b/A103-xds-composite-filter.md @@ -4,7 +4,7 @@ A103: xDS Composite Filter * Approver: ejona86, dfawley * Status: {Draft, In Review, Ready for Implementation, Implemented} * Implemented in: -* Last updated: 2025-09-04 +* Last updated: 2025-12-01 * Discussion at: https://groups.google.com/g/grpc-io/c/es5taH0OZS8 ## Abstract @@ -116,7 +116,7 @@ a maximum recursion depth of 8 when parsing HTTP filter configs. In addition to the CEL request attributes described in [A77], we will also add support for some additional [CEL -attributes](https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/advanced/attributes.html#configuration-attributes) +attributes](https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/advanced/attributes.html) that we expect to be useful for the composite filter. On both the gRPC client and server sides, we will add support for the @@ -134,7 +134,7 @@ to `google.protobuf.Struct` type. We will also add support for the following attributes on the gRPC server side only (these attributes are not relevant on the client side): -- `source.ip` +- `source.address` - `source.port` - `connection.requested_server_name` - `connection.tls_version`