From 8028bf3905e8502154287f5571bff766cb38e90a Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Wed, 12 Mar 2025 23:42:55 +0000 Subject: [PATCH 01/27] WIP --- A93-xds-ext-proc.md | 181 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 181 insertions(+) create mode 100644 A93-xds-ext-proc.md diff --git a/A93-xds-ext-proc.md b/A93-xds-ext-proc.md new file mode 100644 index 000000000..389ad4a01 --- /dev/null +++ b/A93-xds-ext-proc.md @@ -0,0 +1,181 @@ +A93: xDS ExtProc Support +---- +* Author(s): @markdroth +* Approver: @ejona86, @dfawley +* Status: {Draft, In Review, Ready for Implementation, Implemented} +* Implemented in: +* Last updated: 2025-03-12 +* Discussion at: (filled after thread exists) + +## Abstract + +We will add support for the xDS ext_proc filter in gRPC, both on the +client and server sides. + +## Background + +The ext_proc filter provides support for making side-channel call-outs +to perform filtering. + +The ext_proc filter will use the existing infrastructure for xDS +HTTP filters, which was originally introduced in gRFCs [A39]. We will +support this filter on both the gRPC client and server side. + +Note that this filter will make use of the `allowed_grpc_services` map in +the bootstrap config, described in [A77]. It will also make use of the +`trusted_xds_server` server feature introduced in [A81]. + +### Related Proposals: +* [A39: xDS HTTP Filter Support][A39] +* [A77: xDS Server-Side Rate Limiting][A77] (pending) +* [A81: xDS Authority Rewriting][A81] + + +* [A36: xDS-Enabled Servers][A36] +* [A41: xDS RBAC Support][A41] + +[A39]: A39-xds-http-filters.md +[A77]: https://github.com/grpc/proposal/pull/414 +[A81]: A81-xds-authority-rewriting.md + + +[A36]: A36-xds-for-servers.md +[A41]: A41-xds-rbac.md + +## Proposal + + + +### Filter Configuration + +We will support the following fields in the [`ExternalProcessor` +proto](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L101C9-L101C26): +- [grpc_service](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L129): + This field must be present. Inside of it: + - [google_grpc](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/config/core/v3/grpc_service.proto#L303): + This field must be present. Inside of it: + - [target_uri](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/config/core/v3/grpc_service.proto#L254): + This field must be non-empty and must be a valid target URI. The + value specified here must be present in the `allowed_grpc_services` + map in the bootstrap config, which will also determine the credentials + to use, as described in [A77]. + - All other fields are ignored. + - All other fields are ignored. +- [failure_mode_allow](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L177) +- [processing_mode](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L181C18-L181C33): + Required. Inside of it: + - [request_header_mode](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/processing_mode.proto#L118C18-L118C37), + [response_header_mode](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/processing_mode.proto#L121C18-L121C38), + and + [response_trailer_mode](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/processing_mode.proto#L133) + will be supported. + - [request_body_mode](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/processing_mode.proto#L124) + and + [response_body_mode](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/processing_mode.proto#L127): + The only modes we support here are + [`NONE`](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/processing_mode.proto#L66) + and `GRPC` (to be added). + - We ignore the request_trailer_mode field, since gRPC never sends + request trailers. +- [request_attributes](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L188) + and + [response_attributes](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L195C19-L195C38) +- [message_timeout](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L205): + If present, the value must obey the restrictions specified in the + [`google.protobuf.Duration` + documentation](https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#google.protobuf.Duration), + and it must have a positive value. +- [mutation_rules](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L225C55-L225C69): + TODO: Figure out how these rules apply to gRPC. +- [max_message_timeout](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L230C28-L230C47): + If present, the value must obey the restrictions specified in the + [`google.protobuf.Duration` + documentation](https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#google.protobuf.Duration), + and it must have a positive value. +- [forward_rules](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L237C25-L237C38): + TODO: Which headers do we allow forwarding with or without + `trusted_xds_server`? +- [allow_mode_override](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L249C8-L249C27) +- [disable_immediate_response](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L256C8-L256C34) +- [observability_mode](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L283C8-L283C26): + TODO: Figure out flow control story! +- [deferred_close_timeout](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L305C28-L305C50): + If present, the value must obey the restrictions specified in the + [`google.protobuf.Duration` + documentation](https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#google.protobuf.Duration), + and it must have a positive value. +- [send_body_without_waiting_for_header_response](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L321C8-L321C53) +- [allowed_override_modes](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L331C27-L331C49): + Same validation rules as processing_mode field above. + +The following fields will be ignored by gRPC: +- http_service: It doesn't make sense for gRPC to support non-gRPC + mechanisms for contacting the ext_authz server. +- stat_prefix: This does not apply to gRPC. +- filter_metadata, metadata_options, on_processing_response: gRPC does not + currently support dynamic metadata. +- disable_clear_route_cache, route_cache_action: We don't currently support + recomputing the route. We could consider adding this in the future if we + have a use-case for it. + +We will support the following fields in the per-route config: +- [overrides](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L406C22-L406C31): + Optional. Inside of it: + - [processing_mode](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L414): + Same as in top-level filter config. + - [grpc_service](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L431C30-L431C42): + Same as in top-level filter config. + - [grpc_initial_metadata](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L444C39-L444C60): + TODO: Why is this only in the per-route override config? + - We will ignore the metadata_options field. + +TODO: If we need the `disabled` flag, maybe just support that via the +`FilterConfig.disabled` field in `typed_per_filter_config` instead of +doing it in an ext_proc-specific way? Just make sure that the semantics +are the same -- ext_proc config says "A set of overrides in a more +specific configuration will override a "disabled" flag set in a +less-specific one." + +### Communication With the ext_proc Server + +TODO: Fill this in! + +The [`ProcessingRequest` +message](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L62) +sent to the server will be populated as follows: +- + +We will handle the [`ProcessingResponse`](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L132C9-L132C27) +as follows: +- + +### Header Rewriting + +TODO: Update! + +gRPC will support rewriting the `:authority` field only if the +`trusted_xds_server` server feature is present in the bootstrap config, +regardless of what settings are present in the ext_authz filter config. + +If the ext_authz server attempts to overwrite the `host` header, that +change will actually apply to the `:authority` header instead. + +Note that gRPC will not support rewriting the `:scheme` or `:method` +headers, regardless of the value of this field. + +### Temporary environment variable protection + +[Name the environment variable(s) used to enable/disable the feature(s) this proposal introduces and their default(s). Generally, features that are enabled by I/O should include this type of control until they have passed some testing criteria, which should also be detailed here. This section may be omitted if there are none.] + +## Rationale + +[A discussion of alternate approaches and the trade offs, advantages, and disadvantages of the specified approach.] + + +## Implementation + +[A description of the steps in the implementation, who will do them, and when. If a particular language is going to get the implementation first, this section should list the proposed order.] + +## Open issues (if applicable) + +[A discussion of issues relating to this proposal for which the author does not know the solution. This section may be omitted if there are none.] From 7a9afceac4a787bd469d2c1d68ac3e0ed8cd1628 Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Thu, 13 Mar 2025 20:41:17 +0000 Subject: [PATCH 02/27] more WIP --- A93-xds-ext-proc.md | 174 +++++++++++++++++++++++++++++++++----------- 1 file changed, 133 insertions(+), 41 deletions(-) diff --git a/A93-xds-ext-proc.md b/A93-xds-ext-proc.md index 389ad4a01..4bffbb228 100644 --- a/A93-xds-ext-proc.md +++ b/A93-xds-ext-proc.md @@ -4,7 +4,7 @@ A93: xDS ExtProc Support * Approver: @ejona86, @dfawley * Status: {Draft, In Review, Ready for Implementation, Implemented} * Implemented in: -* Last updated: 2025-03-12 +* Last updated: 2025-03-13 * Discussion at: (filled after thread exists) ## Abstract @@ -15,11 +15,11 @@ client and server sides. ## Background The ext_proc filter provides support for making side-channel call-outs -to perform filtering. +to perform filtering/interception. -The ext_proc filter will use the existing infrastructure for xDS -HTTP filters, which was originally introduced in gRFCs [A39]. We will -support this filter on both the gRPC client and server side. +The ext_proc filter will use the existing infrastructure for xDS HTTP +filters, described in gRFCs [A39]. We will support this filter on both +the gRPC client and server side. Note that this filter will make use of the `allowed_grpc_services` map in the bootstrap config, described in [A77]. It will also make use of the @@ -30,26 +30,57 @@ the bootstrap config, described in [A77]. It will also make use of the * [A77: xDS Server-Side Rate Limiting][A77] (pending) * [A81: xDS Authority Rewriting][A81] - -* [A36: xDS-Enabled Servers][A36] -* [A41: xDS RBAC Support][A41] - [A39]: A39-xds-http-filters.md [A77]: https://github.com/grpc/proposal/pull/414 [A81]: A81-xds-authority-rewriting.md - -[A36]: A36-xds-for-servers.md -[A41]: A41-xds-rbac.md - ## Proposal - +We will support the ext_proc filter in gRPC on both the client and +server side. + +### Payload Handling + +The existing ext_proc protocol's handling of request payloads has a +significant impedence mismatch with gRPC. + +The ext_proc protocol is designed for HTTP payloads, meaning that the +ext_proc client sends the raw contents of the HTTP/2 DATA frames to +the ext_proc server. However, gRPC has its own framing of individual +messages inside of the HTTP/2 DATA frames. An ext_proc server accessing +the payload for a gRPC stream is really going to be interested only in +the deframed gRPC messages, one at a time, not the raw DATA frames. + +This means that any existing ext_proc server that was designed to handle +gRPC traffic is going to have to handle deframing the gRPC messages from +the HTTP/2 DATA frames. It will also need to handle buffering while +the payload is sent to it in chunks, because a single gRPC message could +be spread across multiple HTTP/2 DATA frames. This is a fair amount of +work for the ext_proc server to do, so it seems better for the ext_proc +client to do the work of deframing the gRPC messages, and sending only +complete deframed messages to the ext_proc server, one at a time. + +Furthermore, the ext_proc filter in gRPC will not actually have access +to the raw HTTP/2 DATA frames in the first place. In our architecture, +filters see individual gRPC messages, and the framing/deframing is handled +in the transport layer. This means that an ext_proc filter running on the +gRPC client side will see the messages before they have been framed to be +sent by the transport, and an ext_proc filter running on the gRPC server +side will see the messages after they have been received and deframed by +the transport. + +Therefore, we propose adding a new ext_proc `BodySendMode` called +`GRPC`. In this mode, the ext_proc client would handle deframing the +gRPC messages, and it would send each gRPC message to the ext_proc server +as a separate `request_body`. This would be the only processing mode +supported in gRPC. We would request that Envoy implement the same mode, +so that users can switch back and forth between proxy and proxyless data +planes without breaking their ext_proc servers. ### Filter Configuration We will support the following fields in the [`ExternalProcessor` -proto](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L101C9-L101C26): +proto](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L101): - [grpc_service](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L129): This field must be present. Inside of it: - [google_grpc](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/config/core/v3/grpc_service.proto#L303): @@ -62,10 +93,10 @@ proto](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4f - All other fields are ignored. - All other fields are ignored. - [failure_mode_allow](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L177) -- [processing_mode](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L181C18-L181C33): +- [processing_mode](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L181): Required. Inside of it: - - [request_header_mode](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/processing_mode.proto#L118C18-L118C37), - [response_header_mode](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/processing_mode.proto#L121C18-L121C38), + - [request_header_mode](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/processing_mode.proto#L118), + [response_header_mode](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/processing_mode.proto#L121), and [response_trailer_mode](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/processing_mode.proto#L133) will be supported. @@ -79,33 +110,33 @@ proto](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4f request trailers. - [request_attributes](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L188) and - [response_attributes](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L195C19-L195C38) + [response_attributes](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L195) - [message_timeout](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L205): If present, the value must obey the restrictions specified in the [`google.protobuf.Duration` documentation](https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#google.protobuf.Duration), and it must have a positive value. -- [mutation_rules](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L225C55-L225C69): +- [mutation_rules](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L225): TODO: Figure out how these rules apply to gRPC. -- [max_message_timeout](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L230C28-L230C47): +- [max_message_timeout](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L230): If present, the value must obey the restrictions specified in the [`google.protobuf.Duration` documentation](https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#google.protobuf.Duration), and it must have a positive value. -- [forward_rules](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L237C25-L237C38): +- [forward_rules](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L237): TODO: Which headers do we allow forwarding with or without `trusted_xds_server`? -- [allow_mode_override](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L249C8-L249C27) -- [disable_immediate_response](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L256C8-L256C34) -- [observability_mode](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L283C8-L283C26): +- [allow_mode_override](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L249) +- [disable_immediate_response](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L256) +- [observability_mode](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L283): TODO: Figure out flow control story! -- [deferred_close_timeout](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L305C28-L305C50): +- [deferred_close_timeout](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L305): If present, the value must obey the restrictions specified in the [`google.protobuf.Duration` documentation](https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#google.protobuf.Duration), and it must have a positive value. -- [send_body_without_waiting_for_header_response](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L321C8-L321C53) -- [allowed_override_modes](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L331C27-L331C49): +- [send_body_without_waiting_for_header_response](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L321) +- [allowed_override_modes](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L331): Same validation rules as processing_mode field above. The following fields will be ignored by gRPC: @@ -119,13 +150,13 @@ The following fields will be ignored by gRPC: have a use-case for it. We will support the following fields in the per-route config: -- [overrides](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L406C22-L406C31): +- [overrides](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L406): Optional. Inside of it: - [processing_mode](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L414): Same as in top-level filter config. - - [grpc_service](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L431C30-L431C42): + - [grpc_service](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L431): Same as in top-level filter config. - - [grpc_initial_metadata](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L444C39-L444C60): + - [grpc_initial_metadata](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L444): TODO: Why is this only in the per-route override config? - We will ignore the metadata_options field. @@ -138,24 +169,78 @@ less-specific one." ### Communication With the ext_proc Server -TODO: Fill this in! - The [`ProcessingRequest` message](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L62) sent to the server will be populated as follows: -- - -We will handle the [`ProcessingResponse`](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L132C9-L132C27) +- [request_headers](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L76) + and + [response_headers](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L81). + Inside of them: + - [headers](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L216) + - [end_of_stream](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L227) +- [request_body](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L85) + and + [response_body](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L89). + Inside of them: + - [body](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L235) + - [end_of_stream](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L239) +- [response_trailers](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L103). + Inside of it: + - [trailers](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L247) +- [attributes](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L113) +- [observability_mode](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L126) +- Note: We will not populate request_trailers, because gRPC never sends + request trailers. +- Note: We will not populate metadata_context, because gRPC does not + support dynamic metadata. + +We will handle the [`ProcessingResponse`](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L132) as follows: -- +- [request_headers](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L139) + and + [response_headers](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L143). + Inside of them: + - [response](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L257). + Inside of it: + - [status](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L303): + This must be `CONTINUE`. gRPC will not support + `CONTINUE_AND_REPLACE`, because those semantics don't work with + gRPC. + - [header_mutation](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L308): + Inside of it: + - [set_headers](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L375) + - [remove_headers](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L379) + - Note: We do not support body_mutation in response to headers. + - Note: We do not support trailers, since that works only with + `CONTINUE_AND_REPLACE`. + - Note: We do not support clear_route_cache. +- [request_body](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L147) + and + [response_body](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L151): + Inside of them: + - [response](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L257). + Inside of it: + - [body_mutation](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L317) + - [body](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L401) + - Note: We do not support clear_body or streamed_response. + - Note: We do not support header mutations in response to a body. + - Note: We do not support trailers, since that works only with + `CONTINUE_AND_REPLACE`. + - Note: We do not support clear_route_cache. +- [response_trailers](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L159): + - [header_mutation](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L273) + Inside of it: + - [set_headers](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L375) + - [remove_headers](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L379) ### Header Rewriting -TODO: Update! +TODO: What other restrictions are needed here? gRPC will support rewriting the `:authority` field only if the `trusted_xds_server` server feature is present in the bootstrap config, -regardless of what settings are present in the ext_authz filter config. +regardless of what the ext_proc server specifies. TODO: Ignore or fail +if the server says to rewrite but the server feature is not present? If the ext_authz server attempts to overwrite the `host` header, that change will actually apply to the `:authority` header instead. @@ -169,8 +254,15 @@ headers, regardless of the value of this field. ## Rationale -[A discussion of alternate approaches and the trade offs, advantages, and disadvantages of the specified approach.] - +For payload handling, we could have considered having the gRPC ext_proc +filter artificially add the 5-byte gRPC frame header to each message +that we send to the ext_proc server. However, this seems like a +sub-optimal approach, because (a) it would waste bandwidth sending data +that really isn't useful, (b) it would not avoid the need for the +ext_proc server to handle the gRPC framing, and (c) it would further +spread use of the gRPC framing to ext_proc servers, which will make it +awkward for gRPC to support other transports with different framing in +the future. ## Implementation From bb9125a607364d4e6266fdbf26127d7192cc9349 Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Thu, 10 Jul 2025 22:08:51 +0000 Subject: [PATCH 03/27] nail down some details --- A93-xds-ext-proc.md | 75 ++++++++++++++++++++++++++------------------- 1 file changed, 43 insertions(+), 32 deletions(-) diff --git a/A93-xds-ext-proc.md b/A93-xds-ext-proc.md index 4bffbb228..42588d36c 100644 --- a/A93-xds-ext-proc.md +++ b/A93-xds-ext-proc.md @@ -4,7 +4,7 @@ A93: xDS ExtProc Support * Approver: @ejona86, @dfawley * Status: {Draft, In Review, Ready for Implementation, Implemented} * Implemented in: -* Last updated: 2025-03-13 +* Last updated: 2025-07-10 * Discussion at: (filled after thread exists) ## Abstract @@ -117,15 +117,18 @@ proto](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4f documentation](https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#google.protobuf.Duration), and it must have a positive value. - [mutation_rules](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L225): - TODO: Figure out how these rules apply to gRPC. + Optional. Inside of it: + - [disallow_all](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/config/common/mutation_rules/v3/mutation_rules.proto#L70) + - [allow_expression](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/config/common/mutation_rules/v3/mutation_rules.proto#L75) + - [disallow_expression](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/config/common/mutation_rules/v3/mutation_rules.proto#L79) + - [disallow_is_error](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/config/common/mutation_rules/v3/mutation_rules.proto#L87) + - allow_all_routing, disallow_system, allow_envoy: These fields will be ignored. - [max_message_timeout](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L230): If present, the value must obey the restrictions specified in the [`google.protobuf.Duration` documentation](https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#google.protobuf.Duration), and it must have a positive value. -- [forward_rules](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L237): - TODO: Which headers do we allow forwarding with or without - `trusted_xds_server`? +- [forward_rules](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L237) - [allow_mode_override](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L249) - [disable_immediate_response](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L256) - [observability_mode](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L283): @@ -149,23 +152,41 @@ The following fields will be ignored by gRPC: recomputing the route. We could consider adding this in the future if we have a use-case for it. -We will support the following fields in the per-route config: +### Filter Configuration Overrides + +We will support `typed_per_filter_config` config overrides for this +filter, as described in [A39]. + +We will support the following fields in the +[`ExtProcPerRoute`](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L395) +proto: - [overrides](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L406): Optional. Inside of it: - [processing_mode](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L414): Same as in top-level filter config. - [grpc_service](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L431): Same as in top-level filter config. - - [grpc_initial_metadata](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L444): - TODO: Why is this only in the per-route override config? + - [grpc_initial_metadata](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L444) + - [request_attributes](https://github.com/envoyproxy/envoy/blob/72833beab4fdc87f7fc53ec31ab70fd734581720/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L442) + and + [response_attributes](https://github.com/envoyproxy/envoy/blob/72833beab4fdc87f7fc53ec31ab70fd734581720/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L447) + - [failure_mode_allow](https://github.com/envoyproxy/envoy/blob/72833beab4fdc87f7fc53ec31ab70fd734581720/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L468C29-L468C47) - We will ignore the metadata_options field. -TODO: If we need the `disabled` flag, maybe just support that via the -`FilterConfig.disabled` field in `typed_per_filter_config` instead of -doing it in an ext_proc-specific way? Just make sure that the semantics -are the same -- ext_proc config says "A set of overrides in a more -specific configuration will override a "disabled" flag set in a -less-specific one." +Note that we will not use the [`disabled` +field](https://github.com/envoyproxy/envoy/blob/72833beab4fdc87f7fc53ec31ab70fd734581720/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L420) +in the `ExtProcPerRoute` proto itself; instead, we will support disabling +via a more generic mechanism that can apply to any filter. Specifically, +we will honor the `disabled` field in both the [`HttpFilter` +message](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/network/http_connection_manager/v3/http_connection_manager.proto#L1229) +in the HCM config and in the [`FilterConfig` wrapper +message](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/config/route/v3/route_components.proto#L2562) +that can be used in `typed_per_filter_config` fields. + +Note that the most specific `typed_per_filter_config` will be used. For +example, if there is an override config at the virtual host level that +disables the filter but then another config at the route level that does +not disable the filter, the filter will be enabled. ### Communication With the ext_proc Server @@ -235,22 +256,16 @@ as follows: ### Header Rewriting -TODO: What other restrictions are needed here? - -gRPC will support rewriting the `:authority` field only if the -`trusted_xds_server` server feature is present in the bootstrap config, -regardless of what the ext_proc server specifies. TODO: Ignore or fail -if the server says to rewrite but the server feature is not present? - -If the ext_authz server attempts to overwrite the `host` header, that -change will actually apply to the `:authority` header instead. - -Note that gRPC will not support rewriting the `:scheme` or `:method` -headers, regardless of the value of this field. +gRPC will not support rewriting the `:scheme`, `:method`, `:path`, +`:authority`, or `host` headers, regardless of what settings are present +in the ext_proc filter config. If the server specifies a rewrite for +one of these headers, that rewrite will be ignored. ### Temporary environment variable protection -[Name the environment variable(s) used to enable/disable the feature(s) this proposal introduces and their default(s). Generally, features that are enabled by I/O should include this type of control until they have passed some testing criteria, which should also be detailed here. This section may be omitted if there are none.] +Support for the ext_authz filter will be guarded by the +`GRPC_EXPERIMENTAL_XDS_EXT_PROC` environment variable. This guard will +be removed once the feature passes interop tests. ## Rationale @@ -266,8 +281,4 @@ the future. ## Implementation -[A description of the steps in the implementation, who will do them, and when. If a particular language is going to get the implementation first, this section should list the proposed order.] - -## Open issues (if applicable) - -[A discussion of issues relating to this proposal for which the author does not know the solution. This section may be omitted if there are none.] +Will be implemented in C-core, Java, Go, and Node. From c90bc7db3ee50604e4b5373565ecb9dca22e48d8 Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Fri, 22 Aug 2025 00:22:33 +0000 Subject: [PATCH 04/27] reference A102 --- A93-xds-ext-proc.md | 21 ++++++--------------- 1 file changed, 6 insertions(+), 15 deletions(-) diff --git a/A93-xds-ext-proc.md b/A93-xds-ext-proc.md index 42588d36c..f782d49f8 100644 --- a/A93-xds-ext-proc.md +++ b/A93-xds-ext-proc.md @@ -4,7 +4,7 @@ A93: xDS ExtProc Support * Approver: @ejona86, @dfawley * Status: {Draft, In Review, Ready for Implementation, Implemented} * Implemented in: -* Last updated: 2025-07-10 +* Last updated: 2025-08-21 * Discussion at: (filled after thread exists) ## Abstract @@ -18,21 +18,21 @@ The ext_proc filter provides support for making side-channel call-outs to perform filtering/interception. The ext_proc filter will use the existing infrastructure for xDS HTTP -filters, described in gRFCs [A39]. We will support this filter on both +filters, described in gRFC [A39]. We will support this filter on both the gRPC client and server side. Note that this filter will make use of the `allowed_grpc_services` map in -the bootstrap config, described in [A77]. It will also make use of the +the bootstrap config, described in [A102]. It will also make use of the `trusted_xds_server` server feature introduced in [A81]. ### Related Proposals: * [A39: xDS HTTP Filter Support][A39] -* [A77: xDS Server-Side Rate Limiting][A77] (pending) * [A81: xDS Authority Rewriting][A81] +* [A102: xDS GrpcService Support][A102] (pending) [A39]: A39-xds-http-filters.md -[A77]: https://github.com/grpc/proposal/pull/414 [A81]: A81-xds-authority-rewriting.md +[A102]: https://github.com/grpc/proposal/pull/510 ## Proposal @@ -82,16 +82,7 @@ planes without breaking their ext_proc servers. We will support the following fields in the [`ExternalProcessor` proto](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L101): - [grpc_service](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L129): - This field must be present. Inside of it: - - [google_grpc](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/config/core/v3/grpc_service.proto#L303): - This field must be present. Inside of it: - - [target_uri](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/config/core/v3/grpc_service.proto#L254): - This field must be non-empty and must be a valid target URI. The - value specified here must be present in the `allowed_grpc_services` - map in the bootstrap config, which will also determine the credentials - to use, as described in [A77]. - - All other fields are ignored. - - All other fields are ignored. + This field must be present. It will be validated as described in [A102]. - [failure_mode_allow](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L177) - [processing_mode](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L181): Required. Inside of it: From 634cfec6f18bf12d09948faa94b69b2b21d9c259 Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Mon, 25 Aug 2025 23:21:12 +0000 Subject: [PATCH 05/27] flow control for o11y mode --- A93-xds-ext-proc.md | 37 ++++++++++++++++++++++++++++++++++--- 1 file changed, 34 insertions(+), 3 deletions(-) diff --git a/A93-xds-ext-proc.md b/A93-xds-ext-proc.md index f782d49f8..35d6f10a4 100644 --- a/A93-xds-ext-proc.md +++ b/A93-xds-ext-proc.md @@ -4,7 +4,7 @@ A93: xDS ExtProc Support * Approver: @ejona86, @dfawley * Status: {Draft, In Review, Ready for Implementation, Implemented} * Implemented in: -* Last updated: 2025-08-21 +* Last updated: 2025-08-25 * Discussion at: (filled after thread exists) ## Abstract @@ -123,7 +123,8 @@ proto](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4f - [allow_mode_override](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L249) - [disable_immediate_response](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L256) - [observability_mode](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L283): - TODO: Figure out flow control story! + See [Observability Mode and Flow + Control](#observability-mode-and-flow-control) below. - [deferred_close_timeout](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L305): If present, the value must obey the restrictions specified in the [`google.protobuf.Duration` @@ -252,9 +253,39 @@ gRPC will not support rewriting the `:scheme`, `:method`, `:path`, in the ext_proc filter config. If the server specifies a rewrite for one of these headers, that rewrite will be ignored. +### Observability Mode and Flow Control + +In observability mode, the contents of the data plane RPC are sent to +the ext_proc server, but the ext_proc server is not expected to make any +modifications to the data plane RPC, so the data plane RPC proceeds +without waiting for a response from the ext_proc server. One challenge +in this mode is flow control: if we are blocked sending a message to the +ext_proc server by flow control, then we need some push-back on the data +plane RPC, or else we would have to buffer messages to be sent to the +ext_proc server, which can cause OOMs. (Envoy has noted this problem in +https://github.com/envoyproxy/envoy/issues/33319 but has not proposed a +solution yet.) + +For gRPC, we will address this problem by requiring the message to the +ext_proc server to pass flow control before we allow the message to +proceed on the data plane RPC. Due to differences in flow control +semantics across languages, this will look a little different in each +one: + +- In C-core, we will wait for the write to complete on the ext_proc + stream before we allow the message to continue on the data plane RPC. +- In Java, the application has to explicitly check whether there is flow + control push-back before doing a write. For the purposes of that API, + Java will not consider the flow control available until it passes flow + control on both the ext_proc stream and on the data plane stream. +- In Go, a write is blocking and doesn't return until the write passed + flow control. So observability mode doesn't need to do anything + special to handle flow control; it will simply not wait for the + ext_proc response before sending the message on the data plane stream. + ### Temporary environment variable protection -Support for the ext_authz filter will be guarded by the +Support for the ext_proc filter will be guarded by the `GRPC_EXPERIMENTAL_XDS_EXT_PROC` environment variable. This guard will be removed once the feature passes interop tests. From 85de0ec72a0d6b9cd784fcd1bb2d55edbf1badcf Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Fri, 12 Sep 2025 00:42:48 +0000 Subject: [PATCH 06/27] added a lot more details, and a lot of TODOs --- A93-xds-ext-proc.md | 355 ++++++++++++++++++++++++++++++++++++-------- 1 file changed, 290 insertions(+), 65 deletions(-) diff --git a/A93-xds-ext-proc.md b/A93-xds-ext-proc.md index 35d6f10a4..1b2a3da44 100644 --- a/A93-xds-ext-proc.md +++ b/A93-xds-ext-proc.md @@ -4,7 +4,7 @@ A93: xDS ExtProc Support * Approver: @ejona86, @dfawley * Status: {Draft, In Review, Ready for Implementation, Implemented} * Implemented in: -* Last updated: 2025-08-25 +* Last updated: 2025-09-11 * Discussion at: (filled after thread exists) ## Abstract @@ -25,12 +25,12 @@ Note that this filter will make use of the `allowed_grpc_services` map in the bootstrap config, described in [A102]. It will also make use of the `trusted_xds_server` server feature introduced in [A81]. -### Related Proposals: +### Related Proposals: * [A39: xDS HTTP Filter Support][A39] * [A81: xDS Authority Rewriting][A81] * [A102: xDS GrpcService Support][A102] (pending) -[A39]: A39-xds-http-filters.md +[A39]: A39-xds-http-filters.md [A81]: A81-xds-authority-rewriting.md [A102]: https://github.com/grpc/proposal/pull/510 @@ -72,10 +72,14 @@ the transport. Therefore, we propose adding a new ext_proc `BodySendMode` called `GRPC`. In this mode, the ext_proc client would handle deframing the gRPC messages, and it would send each gRPC message to the ext_proc server -as a separate `request_body`. This would be the only processing mode -supported in gRPC. We would request that Envoy implement the same mode, -so that users can switch back and forth between proxy and proxyless data -planes without breaking their ext_proc servers. +as a separate `request_body`. This is a streaming mode, meaning that the +ext_proc client may send a subsequent message to the ext_proc server while +still waiting for a reply from the ext_proc server for a previous message. + +The new `GRPC` mode will be the only processing mode supported in gRPC. +It would be desirable for Envoy to implement the same mode, so that +users can switch back and forth between proxy and proxyless data planes +without breaking their ext_proc servers. ### Filter Configuration @@ -83,14 +87,18 @@ We will support the following fields in the [`ExternalProcessor` proto](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L101): - [grpc_service](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L129): This field must be present. It will be validated as described in [A102]. -- [failure_mode_allow](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L177) +- [failure_mode_allow](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L177): + By default, if the RPC to the ext_proc server fails with a non-OK + status, the data plane RPC will be failed with status UNAVAILABLE. If + this field is set to true, then the data plane RPC will instead be + allowed to continue, with no further action taken by the ext_proc filter. - [processing_mode](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L181): Required. Inside of it: - [request_header_mode](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/processing_mode.proto#L118), [response_header_mode](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/processing_mode.proto#L121), and [response_trailer_mode](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/processing_mode.proto#L133) - will be supported. + will be supported as described in the proto file. - [request_body_mode](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/processing_mode.proto#L124) and [response_body_mode](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/processing_mode.proto#L127): @@ -99,10 +107,25 @@ proto](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4f and `GRPC` (to be added). - We ignore the request_trailer_mode field, since gRPC never sends request trailers. +- [allow_mode_override](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L249): + If true, allows the ext_proc server to dynamically override the + processing mode for an individual data plane RPC. TODO: should we + support this for GRPC mode? - [request_attributes](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L188) and - [response_attributes](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L195) + [response_attributes](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L195): + Attributes to be sent to ext_proc server along with client-to-server + and server-to-client events, respectively. The set of supported + attributes is the same as what we support for any CEL expression in xDS. + any unsupported attribute name will be ignored. See [Attributes Sent to + ext_proc Server](#attributes-sent-to-the-ext_proc-server) below for details. - [message_timeout](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L205): + TODO: Describe how this works! + If present, the value must obey the restrictions specified in the + [`google.protobuf.Duration` + documentation](https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#google.protobuf.Duration), + and it must have a positive value. +- [max_message_timeout](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L230): If present, the value must obey the restrictions specified in the [`google.protobuf.Duration` documentation](https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#google.protobuf.Duration), @@ -114,23 +137,21 @@ proto](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4f - [disallow_expression](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/config/common/mutation_rules/v3/mutation_rules.proto#L79) - [disallow_is_error](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/config/common/mutation_rules/v3/mutation_rules.proto#L87) - allow_all_routing, disallow_system, allow_envoy: These fields will be ignored. -- [max_message_timeout](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L230): - If present, the value must obey the restrictions specified in the - [`google.protobuf.Duration` - documentation](https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#google.protobuf.Duration), - and it must have a positive value. -- [forward_rules](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L237) -- [allow_mode_override](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L249) -- [disable_immediate_response](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L256) +- [forward_rules](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L237): + TODO: document +- [disable_immediate_response](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L256): + TODO: document - [observability_mode](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L283): - See [Observability Mode and Flow - Control](#observability-mode-and-flow-control) below. + See [Observability Mode](#observability-mode) below. - [deferred_close_timeout](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L305): + TODO: document purpose. If present, the value must obey the restrictions specified in the [`google.protobuf.Duration` documentation](https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#google.protobuf.Duration), and it must have a positive value. -- [send_body_without_waiting_for_header_response](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L321) +- [send_body_without_waiting_for_header_response](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L321): + TODO: do we need this with GRPC body send mode? Seems to apply only + to STREAMING mode. - [allowed_override_modes](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L331): Same validation rules as processing_mode field above. @@ -180,7 +201,45 @@ example, if there is an override config at the virtual host level that disables the filter but then another config at the route level that does not disable the filter, the filter will be enabled. -### Communication With the ext_proc Server +### Filter Behavior + +TODO: ExtProc channel retention (simple approach for now, probably +globally shared channel is fine?) + +For every event in the data plane RPC (client headers, client message, +client half-close, server headers, server message, and server trailers), +the filter can be configured to send the contents of that event to the +ext_proc server. The specific behavior for each event is covered below. + +For each data plane RPC, the first time the filter needs to send an +event to the ext_proc server, the filter will create a stream to the +ext_proc server. That ext_proc stream will be associated with that +data plane RPC, and all communication with the ext_proc server for that +specific data plane RPC will be done on that ext_proc stream. + +The filter can be configured to wait for a response from the ext_proc +server for a given event before allowing that event to continue on the +data plane RPC. Note that even if the filter is waiting for a response +for a given event, it may be configured to send the next event before it +has gotten the response to the previous event, in which case there will +be multiple events in flight on the ext_proc stream at the same time. +For example, if the filter previously sent a message about client +headers and is waiting for a response for that event and then sees a +client message, it may be configured not to wait for the response to +the client headers event before sending a message on the ext_proc stream +about the client message. + +Note that the stream to the ext_proc server may be terminated at any time. +If the stream terminates with OK status, that indicates to the filter that +it no longer needs to send any more events to the ext_proc server for that +data plane RPC; all remaining events may proceed on the data plane RPC +without any further action taken by the ext_proc filter. If the stream +terminates with a non-OK status, then by default the data plane RPC will +be failed with UNAVAILABLE status. However, if the `failure_mode_allow` +config field is set to true, then the data plane RPC will instead be +allowed to continue, with no further action taken by the ext_proc filter. + +### Messages Sent To the ext_proc Server The [`ProcessingRequest` message](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L62) @@ -188,83 +247,190 @@ sent to the server will be populated as follows: - [request_headers](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L76) and [response_headers](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L81). + Populated when sending client headers or server headers, respectively. Inside of them: - - [headers](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L216) - - [end_of_stream](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L227) + - [headers](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L216): + Contains the headers. + - [end_of_stream](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L227): + This will always be false for client headers. For server headers, + it will be true when the server sends a Trailers-Only response. - [request_body](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L85) and [response_body](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L89). + Populated when sending a client message or server message, respectively. Inside of them: - - [body](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L235) - - [end_of_stream](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L239) + - [body](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L235): + Contains the serialized message. Will be unset if the client sends + a half-close when there is no message to send (i.e., if the client + never sent any message on the stream, or if the half-close is sent + after the filter has already sent the last message to the ext_proc + server). + - [end_of_stream](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L239): + For client messages, may be true if the client sent a half-close at + the same time as the last message. For server messages, will always + be false. - [response_trailers](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L103). - Inside of it: - - [trailers](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L247) -- [attributes](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L113) -- [observability_mode](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L126) + Populated when sending server trailers. Inside of it: + - [trailers](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L247): + Contains the trailers. +- [attributes](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L113): + See [Attributes Sent to ext_proc + Server](#attributes-sent-to-the-ext_proc-server) below. +- [observability_mode](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L126): + Will be set to the value of the `observability_mode` config field. - Note: We will not populate request_trailers, because gRPC never sends request trailers. - Note: We will not populate metadata_context, because gRPC does not support dynamic metadata. +#### Attributes Sent to ext_proc Server + +The +[`attributes`](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L113) +field in the ext_proc request will be populated based on the filter's +configuration. The field will have only one entry, whose key will be +the fixed string `envoy.filters.http.ext_proc`, and the value will be a +`google.protobuf.Struct` message that contains a map from attribute name +to attribute value. + +The list of attributes to include is specified by the +[`request_attributes`](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L188) +config field, for client-to-server events, or the +[`response_attributes`](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L195) +config field, for server-to-client events. The set of supported attribute +names is the same as what we support for any CEL expression in xDS. + +For example, consider the case where the `request_attributes` config +field contains the attribute `request.path` and the filter is processing +an RPC to the method `Service.Method`. When sending client headers, +client messages, or client half-close to the ext_proc server, the +`google.protobuf.Struct` message will contain an entry with key +`request.path` and value `/Service/Method`. + +### Messages Received From the ext_proc Server + We will handle the [`ProcessingResponse`](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L132) as follows: - [request_headers](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L139) and [response_headers](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L143). + Sent in response to client headers and server headers, respectively. Inside of them: - [response](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L257). Inside of it: - [status](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L303): - This must be `CONTINUE`. gRPC will not support - `CONTINUE_AND_REPLACE`, because those semantics don't work with - gRPC. + This must be `CONTINUE`. gRPC will not support `CONTINUE_AND_REPLACE`, + because those semantics don't work with gRPC; if that value is seen, + the filter will cancel the ext_proc stream and treat it as if it + failed with a non-OK status (see above for details). - [header_mutation](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L308): - Inside of it: - - [set_headers](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L375) - - [remove_headers](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L379) + Header mutations. See [Header rewriting](#header-rewriting) below. - Note: We do not support body_mutation in response to headers. + This field will be ignored in this context. - Note: We do not support trailers, since that works only with - `CONTINUE_AND_REPLACE`. + `CONTINUE_AND_REPLACE`. This field will be ignored. - Note: We do not support clear_route_cache. - [request_body](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L147) and [response_body](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L151): + Sent in response to client messages and server messages, respectively. Inside of them: - [response](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L257). Inside of it: - - [body_mutation](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L317) - - [body](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L401) + - [status](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L303): + This must be `CONTINUE`. gRPC will not support `CONTINUE_AND_REPLACE`, + because those semantics don't work with gRPC; if that value is seen, + the filter will cancel the ext_proc stream and treat it as if it + failed with a non-OK status (see above for details). + - [body_mutation](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L317): + - [body](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L401): + Replaces the original serialized message. - Note: We do not support clear_body or streamed_response. - - Note: We do not support header mutations in response to a body. + - Note: We do not support header_mutation in response to a body. + This field will be ignored in this context. - Note: We do not support trailers, since that works only with - `CONTINUE_AND_REPLACE`. + `CONTINUE_AND_REPLACE`. This field will be ignored. - Note: We do not support clear_route_cache. - [response_trailers](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L159): - - [header_mutation](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L273) - Inside of it: - - [set_headers](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L375) - - [remove_headers](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L379) - -### Header Rewriting - -gRPC will not support rewriting the `:scheme`, `:method`, `:path`, -`:authority`, or `host` headers, regardless of what settings are present -in the ext_proc filter config. If the server specifies a rewrite for -one of these headers, that rewrite will be ignored. - -### Observability Mode and Flow Control - -In observability mode, the contents of the data plane RPC are sent to -the ext_proc server, but the ext_proc server is not expected to make any -modifications to the data plane RPC, so the data plane RPC proceeds -without waiting for a response from the ext_proc server. One challenge -in this mode is flow control: if we are blocked sending a message to the -ext_proc server by flow control, then we need some push-back on the data -plane RPC, or else we would have to buffer messages to be sent to the -ext_proc server, which can cause OOMs. (Envoy has noted this problem in -https://github.com/envoyproxy/envoy/issues/33319 but has not proposed a -solution yet.) + Sent in response to server trailers. Inside of it: + - [header_mutation](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L273): + Header mutations. See [Header rewriting](#header-rewriting) below. +- [immediate_response](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L168): + May be sent in response to any message from the data plane. If sent + in response to a server trailers event, sets the status and optionally + headers to be included in the trailers. If sent in response to any + other event, then the behavior differs depending on whether the + ext_proc filter is running on the gRPC client or the gRPC server. On + the gRPC client side, it will cause the data plane RPC to immediately + fail with the specified status as if it were an out-of-band + cancellation. On the gRPC server side, it will cause the server to + immediately send trailers with the specified status. The filter may + cancel the ext_proc stream after seeing this. Inside this message: + - [grpc_status](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L353): + The status code to send on the data plane RPC. + - [details](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L358): + The status message to send on the data plane RPC. + - [headers](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L346): + Headers to modify. Use of these header modifications is best effort, + depending on which event it was sent in response to and whether the + filter is running in the gRPC client or server. + - We will ignore the status and body fields, since these don't apply + to gRPC. +- [mode_override](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L189C60-L189C73): + If the `allow_mode_override` config field is set to false, this field + will be ignored. + TODO: should we support this in GRPC mode? +- [override_message_timeout](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L204C28-L204C52): + TODO: document behavior +- We ignore the dynamic_metadata field, since it is not relevant to gRPC. + +#### Header Rewriting + +When responding to a client headers, server headers, or server trailers +event, the ext_proc server can return a +[`HeaderMutation`](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L369) +message that says how to mutate the headers. That message will be +handled as follows: +- [set_headers](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L375): + Headers to set or mutate. Inside this message: + - [header](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/config/core/v3/base.proto#L458): + Required. Within it: + - [key](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/config/core/v3/base.proto#L404): + The header name. Must be non-empty and all lower-case. Length + must not exceed 16384. The entry will be ignored if the key is + `host` or starts with a `:`. + - [raw_value](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/config/core/v3/base.proto#L422): + The header value. Length must not exceed 16384. + - The value field will be ignored. + - [append_action](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/config/core/v3/base.proto#L476): + We honor the 4 enum values as described in the proto file. + - [keep_empty_value](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/config/core/v3/base.proto#L480): + By default, any header mutation that results in a header with an + empty value will cause the header key to be removed. If this field + is set to true, then such empty headers will be kept. + - We do not support the deprecated append field. +- [remove_headers](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L379): + Header names to remove. The filter will ignore `host` and any entry + that starts with `:`. + +### Observability Mode + +If the +[`observability_mode`](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L283) +config field is set to true, the contents of the data plane RPC events +are sent to the ext_proc server, but the ext_proc server is not expected +to make any modifications to the data plane RPC, so the data plane +RPC proceeds without waiting for a response from the ext_proc server. +Note that in this mode, all messages on the ext_proc stream will have the +[`observability_mode`](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L126) +field set. + +One challenge in this mode is flow control: if we are blocked sending +a message to the ext_proc server by flow control, then we need some +push-back on the data plane RPC, or else we would have to buffer messages +to be sent to the ext_proc server, which can cause OOMs. (Envoy has +noted this problem in https://github.com/envoyproxy/envoy/issues/33319 +but has not proposed a solution yet.) For gRPC, we will address this problem by requiring the message to the ext_proc server to pass flow control before we allow the message to @@ -283,6 +449,65 @@ one: special to handle flow control; it will simply not wait for the ext_proc response before sending the message on the data plane stream. +### Sending Client Headers to the ext_proc Server + +If the `request_header_mode` config field is set to `SKIP`, then nothing +will be done for this event. The headers will be allowed to proceed on +the data plane RPC immediately. + +Otherwise, the filter will create a stream to the ext_proc server for this +data plane RPC and send a message on that stream, which will populate the +[`request_headers`](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L76) +field with the client headers. + +In observability mode, the client headers will be allowed to continue on +the data plane RPC. Otherwise, the client headers will be delayed until +receiving a response from the ext_proc server. + +### ext_proc Response for Client Headers + +TODO: fill this in + +### Sending Client Message to the ext_proc Server + +TODO: fill this in + +### ext_proc Response for Client Message + +TODO: fill this in + +### Sending Client Half-Close to the ext_proc Server + +TODO: fill this in + +### ext_proc Response for Client Half-Close + +TODO: fill this in + +### Sending Server Headers to the ext_proc Server + +TODO: fill this in + +### ext_proc Response for Server Headers + +TODO: fill this in + +### Sending Server Message to the ext_proc Server + +TODO: fill this in + +### ext_proc Response for Server Message + +TODO: fill this in + +### Sending Server Trailers to the ext_proc Server + +TODO: fill this in + +### ext_proc Response for Server Trailers + +TODO: fill this in + ### Temporary environment variable protection Support for the ext_proc filter will be guarded by the From 8a9fefc00a57d54a7b98b723dc712616a45b7863 Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Fri, 12 Sep 2025 19:42:51 +0000 Subject: [PATCH 07/27] lots more details --- A93-xds-ext-proc.md | 384 ++++++++++++++++++++++++++++---------------- 1 file changed, 248 insertions(+), 136 deletions(-) diff --git a/A93-xds-ext-proc.md b/A93-xds-ext-proc.md index 1b2a3da44..22263f1c6 100644 --- a/A93-xds-ext-proc.md +++ b/A93-xds-ext-proc.md @@ -4,7 +4,7 @@ A93: xDS ExtProc Support * Approver: @ejona86, @dfawley * Status: {Draft, In Review, Ready for Implementation, Implemented} * Implemented in: -* Last updated: 2025-09-11 +* Last updated: 2025-09-12 * Discussion at: (filled after thread exists) ## Abstract @@ -39,6 +39,85 @@ the bootstrap config, described in [A102]. It will also make use of the We will support the ext_proc filter in gRPC on both the client and server side. +### Filter Behavior + +TODO: ExtProc channel retention (simple approach for now, probably +globally shared channel is fine?) + +For every event in the data plane RPC (client headers, client message, +client half-close, server headers, server message, and server trailers), +the filter can be configured to send the contents of that event to the +ext_proc server. The specific behavior for each event is covered below. + +For each data plane RPC, the first time the filter needs to send an +event to the ext_proc server, the filter will create a stream to the +ext_proc server. That ext_proc stream will be associated with that +data plane RPC, and all communication with the ext_proc server for that +specific data plane RPC will be done on that ext_proc stream. + +If not in [observability mode](#observability-mode), then for each event +that the filter sends to the ext_proc server, it will wait for a response +from the ext_proc server before allowing that event to proceed on the data +plane RPC. The response from the ext_proc server will tell the filter +what modifications to make to the event before allowing it to proceed +on the data plane RPC (e.g., it may modify headers or message bodies). + +Each event should be sent on the ext_proc stream as it occurs, even if +the filter has not yet received a response from the ext_proc server +for a previous event, which means that there may be multiple events +in flight on the ext_proc stream at the same time. For example, +if the filter previously sent a message about client headers and is +waiting for a response for that event and then sees a client message, +it should immediately send the message on the ext_proc stream about the +client message event (assuming that it is configured to send the client +message event). + +Note that the stream to the ext_proc server may be terminated at any time. +If the stream terminates with OK status, that indicates to the filter that +it no longer needs to send any more events to the ext_proc server for that +data plane RPC; all remaining events may proceed on the data plane RPC +without any further action taken by the ext_proc filter. If the stream +terminates with a non-OK status, then by default the data plane RPC will +be failed with UNAVAILABLE status. However, if the `failure_mode_allow` +config field is set to true, then the data plane RPC will instead be +allowed to continue, with no further action taken by the ext_proc filter. + +### Observability Mode + +If the +[`observability_mode`](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L283) +config field is set to true, the contents of the data plane RPC events +are sent to the ext_proc server, but the ext_proc server is not expected +to make any modifications to the data plane RPC, so the data plane +RPC proceeds without waiting for a response from the ext_proc server. +Note that in this mode, all messages on the ext_proc stream will have the +[`observability_mode`](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L126) +field set. + +One challenge in this mode is flow control: if we are blocked sending +a message to the ext_proc server by flow control, then we need some +push-back on the data plane RPC, or else we would have to buffer messages +to be sent to the ext_proc server, which can cause OOMs. (Envoy has +noted this problem in https://github.com/envoyproxy/envoy/issues/33319 +but has not proposed a solution yet.) + +For gRPC, we will address this problem by requiring the message to the +ext_proc server to pass flow control before we allow the message to +proceed on the data plane RPC. Due to differences in flow control +semantics across languages, this will look a little different in each +one: + +- In C-core, we will wait for the write to complete on the ext_proc + stream before we allow the message to continue on the data plane RPC. +- In Java, the application has to explicitly check whether there is flow + control push-back before doing a write. For the purposes of that API, + Java will not consider the flow control available until it passes flow + control on both the ext_proc stream and on the data plane stream. +- In Go, a write is blocking and doesn't return until the write passed + flow control. So observability mode doesn't need to do anything + special to handle flow control; it will simply not wait for the + ext_proc response before sending the message on the data plane stream. + ### Payload Handling The existing ext_proc protocol's handling of request payloads has a @@ -51,7 +130,7 @@ messages inside of the HTTP/2 DATA frames. An ext_proc server accessing the payload for a gRPC stream is really going to be interested only in the deframed gRPC messages, one at a time, not the raw DATA frames. -This means that any existing ext_proc server that was designed to handle +This means that any existing ext_proc server that wants to handle gRPC traffic is going to have to handle deframing the gRPC messages from the HTTP/2 DATA frames. It will also need to handle buffering while the payload is sent to it in chunks, because a single gRPC message could @@ -60,29 +139,43 @@ work for the ext_proc server to do, so it seems better for the ext_proc client to do the work of deframing the gRPC messages, and sending only complete deframed messages to the ext_proc server, one at a time. +More seriously, ext_proc currently assumes that all of the contents of +the DATA frames are a single payload, and the ext_proc server is only +allowed to send a single response to modify that payload. That model is +incompatible with gRPC streaming semantics, where the ext_proc server +may need to modify each message individually and cannot wait until the +end of the stream to do so. + Furthermore, the ext_proc filter in gRPC will not actually have access to the raw HTTP/2 DATA frames in the first place. In our architecture, -filters see individual gRPC messages, and the framing/deframing is handled -in the transport layer. This means that an ext_proc filter running on the -gRPC client side will see the messages before they have been framed to be -sent by the transport, and an ext_proc filter running on the gRPC server -side will see the messages after they have been received and deframed by -the transport. - -Therefore, we propose adding a new ext_proc `BodySendMode` called -`GRPC`. In this mode, the ext_proc client would handle deframing the -gRPC messages, and it would send each gRPC message to the ext_proc server -as a separate `request_body`. This is a streaming mode, meaning that the -ext_proc client may send a subsequent message to the ext_proc server while -still waiting for a reply from the ext_proc server for a previous message. +the framing/deframing is handled in the transport layer, and filters +see only individual gRPC messages. This means that an ext_proc filter +running on the gRPC client side will see the messages before they have +been framed to be sent by the transport, and an ext_proc filter running on +the gRPC server side will see the messages after they have been received +and deframed by the transport. + +Therefore, we propose adding a new ext_proc `BodySendMode` called `GRPC` +(see https://github.com/envoyproxy/envoy/pull/38753). In this mode, the +ext_proc client would handle deframing the gRPC messages. It will send +each gRPC message to the ext_proc server as a separate `request_body`, +and the ext_proc server will send a separate response for each one. +This is a streaming mode, meaning that the ext_proc client may send +a subsequent message to the ext_proc server while still waiting for a +reply from the ext_proc server for a previous message. The new `GRPC` mode will be the only processing mode supported in gRPC. -It would be desirable for Envoy to implement the same mode, so that -users can switch back and forth between proxy and proxyless data planes -without breaking their ext_proc servers. +It is be desirable for Envoy to implement the same mode, so that users +can switch back and forth between proxy and proxyless data planes without +breaking their ext_proc servers. ### Filter Configuration +The filter supports both a top-level configuration and an override +config. + +#### Top-Level Configuration + We will support the following fields in the [`ExternalProcessor` proto](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L101): - [grpc_service](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L129): @@ -109,8 +202,11 @@ proto](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4f request trailers. - [allow_mode_override](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L249): If true, allows the ext_proc server to dynamically override the - processing mode for an individual data plane RPC. TODO: should we - support this for GRPC mode? + processing mode for an individual data plane RPC. +- [allowed_override_modes](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L331): + Ignored unless `allow_mode_override` is true. This list indicates the + set of allowed override modes, ignoring the request header mode. If + this list is empty, then any override sent by the server is honored. - [request_attributes](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L188) and [response_attributes](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L195): @@ -120,40 +216,58 @@ proto](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4f any unsupported attribute name will be ignored. See [Attributes Sent to ext_proc Server](#attributes-sent-to-the-ext_proc-server) below for details. - [message_timeout](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L205): - TODO: Describe how this works! - If present, the value must obey the restrictions specified in the - [`google.protobuf.Duration` + A timeout to apply between sending an event to the ext_proc server + and receiving a response for that event. If the timeout elapses + before a response is received, the filter treats that as if the + ext_proc stream failed with a non-OK status. If unset, defaults to + 200ms. If present, the value must obey the restrictions specified in + the [`google.protobuf.Duration` documentation](https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#google.protobuf.Duration), and it must have a positive value. - [max_message_timeout](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L230): - If present, the value must obey the restrictions specified in the - [`google.protobuf.Duration` + The maximum timeout that the ext_proc server is allowed to request an + increase to. If unset, timeout overrides from the ext_proc server + will be ignored. If present, the value must obey the restrictions + specified in the [`google.protobuf.Duration` documentation](https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#google.protobuf.Duration), and it must have a positive value. - [mutation_rules](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L225): Optional. Inside of it: - - [disallow_all](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/config/common/mutation_rules/v3/mutation_rules.proto#L70) - - [allow_expression](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/config/common/mutation_rules/v3/mutation_rules.proto#L75) - - [disallow_expression](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/config/common/mutation_rules/v3/mutation_rules.proto#L79) - - [disallow_is_error](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/config/common/mutation_rules/v3/mutation_rules.proto#L87) + - [disallow_all](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/config/common/mutation_rules/v3/mutation_rules.proto#L70): + If true, disallows all header mutations from the ext_proc server. + - [allow_expression](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/config/common/mutation_rules/v3/mutation_rules.proto#L75): + If set, specifically allows any matching header that is not also + matched by `disallow_expression`. Note that regexes should be checked + for validity as part of resource validation. + - [disallow_expression](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/config/common/mutation_rules/v3/mutation_rules.proto#L79): + If set, specifically disallows any matching header. Note that regexes + should be checked for validity as part of resource validation. + - [disallow_is_error](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/config/common/mutation_rules/v3/mutation_rules.proto#L87): + If true, a disallowed header will cause the filter to fail the data + plane RPC with INTERNAL status. - allow_all_routing, disallow_system, allow_envoy: These fields will be ignored. - [forward_rules](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L237): - TODO: document + Configures which headers from the data plane RPC will be included in + the message to the ext_proc server. In this message: + - [allowed_headers](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L386) + and + [disallowed_headers](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L390): + These fields behave as documented in the proto file. - [disable_immediate_response](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L256): - TODO: document + If true, then if the ext_proc server sends a message with the + `immediate_response` field populated, the filter will act as if the + ext_proc stream terminated with a non-OK status. - [observability_mode](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L283): See [Observability Mode](#observability-mode) below. - [deferred_close_timeout](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L305): - TODO: document purpose. - If present, the value must obey the restrictions specified in the - [`google.protobuf.Duration` + In observability mode, the data plane stream may terminate before the + ext_proc server has finished reading all data off of the ext_proc + stream. To avoid that, we delay closing the ext_proc stream from the + client side for a short time after the data plane stream is destroyed. + If unset, the delay is 5 seconds. If present, the value must obey the + restrictions specified in the [`google.protobuf.Duration` documentation](https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#google.protobuf.Duration), and it must have a positive value. -- [send_body_without_waiting_for_header_response](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L321): - TODO: do we need this with GRPC body send mode? Seems to apply only - to STREAMING mode. -- [allowed_override_modes](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L331): - Same validation rules as processing_mode field above. The following fields will be ignored by gRPC: - http_service: It doesn't make sense for gRPC to support non-gRPC @@ -164,8 +278,10 @@ The following fields will be ignored by gRPC: - disable_clear_route_cache, route_cache_action: We don't currently support recomputing the route. We could consider adding this in the future if we have a use-case for it. +- send_body_without_waiting_for_header_response: This is relevant only + in `STREAMED` body send mode, which gRPC does not support. -### Filter Configuration Overrides +#### Override Configuration We will support `typed_per_filter_config` config overrides for this filter, as described in [A39]. @@ -201,45 +317,12 @@ example, if there is an override config at the virtual host level that disables the filter but then another config at the route level that does not disable the filter, the filter will be enabled. -### Filter Behavior +### Communication With the ext_proc Server -TODO: ExtProc channel retention (simple approach for now, probably -globally shared channel is fine?) - -For every event in the data plane RPC (client headers, client message, -client half-close, server headers, server message, and server trailers), -the filter can be configured to send the contents of that event to the -ext_proc server. The specific behavior for each event is covered below. - -For each data plane RPC, the first time the filter needs to send an -event to the ext_proc server, the filter will create a stream to the -ext_proc server. That ext_proc stream will be associated with that -data plane RPC, and all communication with the ext_proc server for that -specific data plane RPC will be done on that ext_proc stream. +This section describes the protocol for communication with the ext_proc +server. -The filter can be configured to wait for a response from the ext_proc -server for a given event before allowing that event to continue on the -data plane RPC. Note that even if the filter is waiting for a response -for a given event, it may be configured to send the next event before it -has gotten the response to the previous event, in which case there will -be multiple events in flight on the ext_proc stream at the same time. -For example, if the filter previously sent a message about client -headers and is waiting for a response for that event and then sees a -client message, it may be configured not to wait for the response to -the client headers event before sending a message on the ext_proc stream -about the client message. - -Note that the stream to the ext_proc server may be terminated at any time. -If the stream terminates with OK status, that indicates to the filter that -it no longer needs to send any more events to the ext_proc server for that -data plane RPC; all remaining events may proceed on the data plane RPC -without any further action taken by the ext_proc filter. If the stream -terminates with a non-OK status, then by default the data plane RPC will -be failed with UNAVAILABLE status. However, if the `failure_mode_allow` -config field is set to true, then the data plane RPC will instead be -allowed to continue, with no further action taken by the ext_proc filter. - -### Messages Sent To the ext_proc Server +#### Messages Sent To the ext_proc Server The [`ProcessingRequest` message](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L62) @@ -260,7 +343,7 @@ sent to the server will be populated as follows: Populated when sending a client message or server message, respectively. Inside of them: - [body](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L235): - Contains the serialized message. Will be unset if the client sends + Contains the serialized message. Will be empty if the client sends a half-close when there is no message to send (i.e., if the client never sent any message on the stream, or if the half-close is sent after the filter has already sent the last message to the ext_proc @@ -278,6 +361,15 @@ sent to the server will be populated as follows: Server](#attributes-sent-to-the-ext_proc-server) below. - [observability_mode](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L126): Will be set to the value of the `observability_mode` config field. +- [protocol_config](https://github.com/envoyproxy/envoy/blob/564612e32eafc10a7a7fd490cdb5cc7149e5802b/api/envoy/service/ext_proc/v3/external_processor.proto#L153): + Populated only on the first message that the filter sends on the + ext_proc stream. Inside of it: + - [request_body_mode](https://github.com/envoyproxy/envoy/blob/564612e32eafc10a7a7fd490cdb5cc7149e5802b/api/envoy/service/ext_proc/v3/external_processor.proto#L67) + and + [response_body_mode](https://github.com/envoyproxy/envoy/blob/564612e32eafc10a7a7fd490cdb5cc7149e5802b/api/envoy/service/ext_proc/v3/external_processor.proto#L72): + Populated from the filter config's processing mode. + - The send_body_without_waiting_for_header_response field will never + be set, since that applies only in STREAMED body send mode. - Note: We will not populate request_trailers, because gRPC never sends request trailers. - Note: We will not populate metadata_context, because gRPC does not @@ -307,7 +399,7 @@ client messages, or client half-close to the ext_proc server, the `google.protobuf.Struct` message will contain an entry with key `request.path` and value `/Service/Method`. -### Messages Received From the ext_proc Server +#### Messages Received From the ext_proc Server We will handle the [`ProcessingResponse`](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L132) as follows: @@ -377,9 +469,7 @@ as follows: - We will ignore the status and body fields, since these don't apply to gRPC. - [mode_override](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L189C60-L189C73): - If the `allow_mode_override` config field is set to false, this field - will be ignored. - TODO: should we support this in GRPC mode? + See [Processing Mode Override](#processing-mode-override) below. - [override_message_timeout](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L204C28-L204C52): TODO: document behavior - We ignore the dynamic_metadata field, since it is not relevant to gRPC. @@ -413,43 +503,59 @@ handled as follows: Header names to remove. The filter will ignore `host` and any entry that starts with `:`. -### Observability Mode - -If the -[`observability_mode`](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L283) -config field is set to true, the contents of the data plane RPC events -are sent to the ext_proc server, but the ext_proc server is not expected -to make any modifications to the data plane RPC, so the data plane -RPC proceeds without waiting for a response from the ext_proc server. -Note that in this mode, all messages on the ext_proc stream will have the -[`observability_mode`](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L126) -field set. - -One challenge in this mode is flow control: if we are blocked sending -a message to the ext_proc server by flow control, then we need some -push-back on the data plane RPC, or else we would have to buffer messages -to be sent to the ext_proc server, which can cause OOMs. (Envoy has -noted this problem in https://github.com/envoyproxy/envoy/issues/33319 -but has not proposed a solution yet.) - -For gRPC, we will address this problem by requiring the message to the -ext_proc server to pass flow control before we allow the message to -proceed on the data plane RPC. Due to differences in flow control -semantics across languages, this will look a little different in each -one: - -- In C-core, we will wait for the write to complete on the ext_proc - stream before we allow the message to continue on the data plane RPC. -- In Java, the application has to explicitly check whether there is flow - control push-back before doing a write. For the purposes of that API, - Java will not consider the flow control available until it passes flow - control on both the ext_proc stream and on the data plane stream. -- In Go, a write is blocking and doesn't return until the write passed - flow control. So observability mode doesn't need to do anything - special to handle flow control; it will simply not wait for the - ext_proc response before sending the message on the data plane stream. - -### Sending Client Headers to the ext_proc Server +#### Processing Mode Override + +The `mode_override` field in the ext_proc response allows the ext_proc +server to tell the client to change its behavior for the duration of +the RPC. For example, the ext_proc server may decide after seeing the +client headers event that it does not actually need to see the client +message events. + +If the `allow_mode_override` config field is set to false, then the +`mode_override` field will be ignored. + +If the `allowed_override_modes` config field is non-empty and +the value of the `mode_override` field does not match any entry in +`allowed_override_modes`, the mode override will be ignored. Note that +this matching will ignore the value of the `request_header_mode` field, +since that mode cannot be overridden. (The filter will already have +sent the request headers to the ext_proc server before the ext_proc +server can send back a response changing the processing mode.) + +Otherwise, when the filter receives a response from the ext_proc server +with `mode_override` set, it will comply with the requested change for +all subsequent events on the data plane RPC. However, note that due to +the streaming nature of the ext_proc communication, there is no +guarantee that the filter will see the message before it has already +processed events from the data plane RPC that would have been affected +by the change. + +A common use-case for this is when the filter is configured to send +client messages, and in the response to the client headers event, the +ext_proc server uses `mode_override` to set `request_body_mode` to +`NONE`. The filter may have already sent one or more client message +events by the time it sees this override. In this case, the ext_proc +server will ignore those client message events, and the filter should (a) +stop sending any subsequent client message events and (b) stop waiting +for responses to any client message events that it has already sent and +allow those client messages to proceed on the data plane RPC. + +Note that when using `mode_override` to *enable* sending an event that the +filter was configured to *disable*, the event may have already occurred +by the time the filter sees the `mode_override`. For example, if the +filter was originally configured to not send server headers and the +ext_proc server tries to enable that in a response to a client message +event, it's possible that the filter has already allowed server headers +to proceed on the data plane RPC before it saw that `mode_override`. +Therefore, ext_proc servers should be aware that this usage is +best-effort and not guaranteed. + +### Event Handling + +This section documents the handling of each event in the data plane RPC +and each response from the ext_proc server. + +#### Sending Client Headers to the ext_proc Server If the `request_header_mode` config field is set to `SKIP`, then nothing will be done for this event. The headers will be allowed to proceed on @@ -464,52 +570,54 @@ In observability mode, the client headers will be allowed to continue on the data plane RPC. Otherwise, the client headers will be delayed until receiving a response from the ext_proc server. -### ext_proc Response for Client Headers +#### ext_proc Response for Client Headers TODO: fill this in -### Sending Client Message to the ext_proc Server +#### Sending Client Message to the ext_proc Server TODO: fill this in -### ext_proc Response for Client Message +#### ext_proc Response for Client Message TODO: fill this in -### Sending Client Half-Close to the ext_proc Server +#### Sending Client Half-Close to the ext_proc Server TODO: fill this in -### ext_proc Response for Client Half-Close +#### ext_proc Response for Client Half-Close TODO: fill this in -### Sending Server Headers to the ext_proc Server +#### Sending Server Headers to the ext_proc Server TODO: fill this in -### ext_proc Response for Server Headers +#### ext_proc Response for Server Headers TODO: fill this in -### Sending Server Message to the ext_proc Server +#### Sending Server Message to the ext_proc Server TODO: fill this in -### ext_proc Response for Server Message +#### ext_proc Response for Server Message TODO: fill this in -### Sending Server Trailers to the ext_proc Server +#### Sending Server Trailers to the ext_proc Server TODO: fill this in -### ext_proc Response for Server Trailers +#### ext_proc Response for Server Trailers TODO: fill this in ### Temporary environment variable protection +TODO: do we need separate env vars for client and server sides? + Support for the ext_proc filter will be guarded by the `GRPC_EXPERIMENTAL_XDS_EXT_PROC` environment variable. This guard will be removed once the feature passes interop tests. @@ -519,12 +627,16 @@ be removed once the feature passes interop tests. For payload handling, we could have considered having the gRPC ext_proc filter artificially add the 5-byte gRPC frame header to each message that we send to the ext_proc server. However, this seems like a -sub-optimal approach, because (a) it would waste bandwidth sending data -that really isn't useful, (b) it would not avoid the need for the -ext_proc server to handle the gRPC framing, and (c) it would further -spread use of the gRPC framing to ext_proc servers, which will make it -awkward for gRPC to support other transports with different framing in -the future. +sub-optimal approach, for the following reasons: +- It would still not work for streaming cases, where the ext_proc server + needs to be able to modify individual messages as they are sent, + rather than only modifying the entire stream as a single payload. +- It would waste bandwidth sending data that really isn't useful. +- It would not avoid the need for the ext_proc server to handle the + gRPC framing. +- It would further spread use of the gRPC framing to ext_proc servers, + which will make it awkward for gRPC to support other transports with + different framing in the future. ## Implementation From be854e062d45dc2a4df36b577b2c05f243ea3ddc Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Fri, 12 Sep 2025 19:44:07 +0000 Subject: [PATCH 08/27] fix section header scope --- A93-xds-ext-proc.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/A93-xds-ext-proc.md b/A93-xds-ext-proc.md index 22263f1c6..98294a18e 100644 --- a/A93-xds-ext-proc.md +++ b/A93-xds-ext-proc.md @@ -82,7 +82,7 @@ be failed with UNAVAILABLE status. However, if the `failure_mode_allow` config field is set to true, then the data plane RPC will instead be allowed to continue, with no further action taken by the ext_proc filter. -### Observability Mode +#### Observability Mode If the [`observability_mode`](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L283) @@ -118,7 +118,7 @@ one: special to handle flow control; it will simply not wait for the ext_proc response before sending the message on the data plane stream. -### Payload Handling +#### Payload Handling The existing ext_proc protocol's handling of request payloads has a significant impedence mismatch with gRPC. From 1d9e41d94c39183c7628d7d5e7e6849280840eab Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Fri, 12 Sep 2025 20:12:09 +0000 Subject: [PATCH 09/27] trace context --- A93-xds-ext-proc.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/A93-xds-ext-proc.md b/A93-xds-ext-proc.md index 98294a18e..8c3841f98 100644 --- a/A93-xds-ext-proc.md +++ b/A93-xds-ext-proc.md @@ -49,11 +49,13 @@ client half-close, server headers, server message, and server trailers), the filter can be configured to send the contents of that event to the ext_proc server. The specific behavior for each event is covered below. -For each data plane RPC, the first time the filter needs to send an -event to the ext_proc server, the filter will create a stream to the -ext_proc server. That ext_proc stream will be associated with that -data plane RPC, and all communication with the ext_proc server for that -specific data plane RPC will be done on that ext_proc stream. +For each data plane RPC, the first time the filter needs to send an event +to the ext_proc server, the filter will create a stream to the ext_proc +server. That ext_proc stream will be associated with that data plane RPC, +and all communication with the ext_proc server for that specific data +plane RPC will be done on that ext_proc stream. The filter will pass +the trace context from the data plane RPC to the ext_proc RPC, so that +the ext_proc RPC appears as a child span on the data plane RPC's trace. If not in [observability mode](#observability-mode), then for each event that the filter sends to the ext_proc server, it will wait for a response From 15ad1796e3bf78940fc0fba1dbbff2af134bed29 Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Fri, 12 Sep 2025 20:41:21 +0000 Subject: [PATCH 10/27] flow control, and remove event list --- A93-xds-ext-proc.md | 88 +++++++++++++-------------------------------- 1 file changed, 25 insertions(+), 63 deletions(-) diff --git a/A93-xds-ext-proc.md b/A93-xds-ext-proc.md index 8c3841f98..959f19cac 100644 --- a/A93-xds-ext-proc.md +++ b/A93-xds-ext-proc.md @@ -120,6 +120,20 @@ one: special to handle flow control; it will simply not wait for the ext_proc response before sending the message on the data plane stream. +#### Flow Control and Streaming RPCs + +If not in observability mode, if the data plane RPC is a streaming RPC, +then there may be multiple client messages or server messages queued up +at the same time waiting for responses from the ext_proc server. Note +that we specifically do *not* want to wait until we see a response from +one message before sending the next message to the ext_proc server, +because that would add per-message latency for a full round-trip to the +ext_proc server, which would dramatically affect streaming performance. + +Implementations must ensure proper flow control for this situation. + +TODO: figure out details of how this will work + #### Payload Handling The existing ext_proc protocol's handling of request payloads has a @@ -476,6 +490,13 @@ as follows: TODO: document behavior - We ignore the dynamic_metadata field, since it is not relevant to gRPC. +Note that the responses from the ext_proc server must come back in the +same order that the events were sent by the filter. For example, if the +client sends a client headers event and a client message event and the +ext_proc server responds to the client message event first, that is +considered a protocol error. The filter will treat that as if the +ext_proc stream failed with a non-OK status. + #### Header Rewriting When responding to a client headers, server headers, or server trailers @@ -552,69 +573,10 @@ to proceed on the data plane RPC before it saw that `mode_override`. Therefore, ext_proc servers should be aware that this usage is best-effort and not guaranteed. -### Event Handling - -This section documents the handling of each event in the data plane RPC -and each response from the ext_proc server. - -#### Sending Client Headers to the ext_proc Server - -If the `request_header_mode` config field is set to `SKIP`, then nothing -will be done for this event. The headers will be allowed to proceed on -the data plane RPC immediately. - -Otherwise, the filter will create a stream to the ext_proc server for this -data plane RPC and send a message on that stream, which will populate the -[`request_headers`](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L76) -field with the client headers. - -In observability mode, the client headers will be allowed to continue on -the data plane RPC. Otherwise, the client headers will be delayed until -receiving a response from the ext_proc server. - -#### ext_proc Response for Client Headers - -TODO: fill this in - -#### Sending Client Message to the ext_proc Server - -TODO: fill this in - -#### ext_proc Response for Client Message - -TODO: fill this in - -#### Sending Client Half-Close to the ext_proc Server - -TODO: fill this in - -#### ext_proc Response for Client Half-Close - -TODO: fill this in - -#### Sending Server Headers to the ext_proc Server - -TODO: fill this in - -#### ext_proc Response for Server Headers - -TODO: fill this in - -#### Sending Server Message to the ext_proc Server - -TODO: fill this in - -#### ext_proc Response for Server Message - -TODO: fill this in - -#### Sending Server Trailers to the ext_proc Server - -TODO: fill this in - -#### ext_proc Response for Server Trailers - -TODO: fill this in +To implement this, the filter will store the processing mode separately +for each data plane RPC. The processing mode for an RPC will be +initialized based on the filter's config, but it may be modified later +by subsequent overrides. ### Temporary environment variable protection From f99a8684ec3f12c553ca8336fde7e370815e090a Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Fri, 12 Sep 2025 22:45:31 +0000 Subject: [PATCH 11/27] no message timeout --- A93-xds-ext-proc.md | 22 ++++------------------ 1 file changed, 4 insertions(+), 18 deletions(-) diff --git a/A93-xds-ext-proc.md b/A93-xds-ext-proc.md index 959f19cac..2ea2299b9 100644 --- a/A93-xds-ext-proc.md +++ b/A93-xds-ext-proc.md @@ -231,22 +231,6 @@ proto](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4f attributes is the same as what we support for any CEL expression in xDS. any unsupported attribute name will be ignored. See [Attributes Sent to ext_proc Server](#attributes-sent-to-the-ext_proc-server) below for details. -- [message_timeout](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L205): - A timeout to apply between sending an event to the ext_proc server - and receiving a response for that event. If the timeout elapses - before a response is received, the filter treats that as if the - ext_proc stream failed with a non-OK status. If unset, defaults to - 200ms. If present, the value must obey the restrictions specified in - the [`google.protobuf.Duration` - documentation](https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#google.protobuf.Duration), - and it must have a positive value. -- [max_message_timeout](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L230): - The maximum timeout that the ext_proc server is allowed to request an - increase to. If unset, timeout overrides from the ext_proc server - will be ignored. If present, the value must obey the restrictions - specified in the [`google.protobuf.Duration` - documentation](https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#google.protobuf.Duration), - and it must have a positive value. - [mutation_rules](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L225): Optional. Inside of it: - [disallow_all](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/config/common/mutation_rules/v3/mutation_rules.proto#L70): @@ -286,6 +270,8 @@ proto](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4f and it must have a positive value. The following fields will be ignored by gRPC: +- message_timeout and max_message_timeout: Message timeouts do not make + sense in GRPC body send mode. - http_service: It doesn't make sense for gRPC to support non-gRPC mechanisms for contacting the ext_authz server. - stat_prefix: This does not apply to gRPC. @@ -486,8 +472,8 @@ as follows: to gRPC. - [mode_override](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L189C60-L189C73): See [Processing Mode Override](#processing-mode-override) below. -- [override_message_timeout](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L204C28-L204C52): - TODO: document behavior +- We will ignore override_message_timeout, since GRPC body send mode + does not support timeouts. - We ignore the dynamic_metadata field, since it is not relevant to gRPC. Note that the responses from the ext_proc server must come back in the From 08230f0ec35823610ca2d372a60f454efa5fc441 Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Fri, 12 Sep 2025 23:03:59 +0000 Subject: [PATCH 12/27] add TODO about metrics --- A93-xds-ext-proc.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/A93-xds-ext-proc.md b/A93-xds-ext-proc.md index 2ea2299b9..a1a594499 100644 --- a/A93-xds-ext-proc.md +++ b/A93-xds-ext-proc.md @@ -185,6 +185,15 @@ It is be desirable for Envoy to implement the same mode, so that users can switch back and forth between proxy and proxyless data planes without breaking their ext_proc servers. +#### Metrics + +TODO: define metrics for tracking time between entering and exiting the +ext_proc filter for the following events: +- client headers +- client half-close +- server headers +- server trailers + ### Filter Configuration The filter supports both a top-level configuration and an override From ba1133a56ffb95513b47f484df7901600bad7719 Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Sat, 13 Sep 2025 00:02:48 +0000 Subject: [PATCH 13/27] avoid the need for flow control by using full duplex streaming --- A93-xds-ext-proc.md | 93 ++++++++++++++++++++++++++++----------------- 1 file changed, 58 insertions(+), 35 deletions(-) diff --git a/A93-xds-ext-proc.md b/A93-xds-ext-proc.md index a1a594499..2bcd0fdbe 100644 --- a/A93-xds-ext-proc.md +++ b/A93-xds-ext-proc.md @@ -47,7 +47,9 @@ globally shared channel is fine?) For every event in the data plane RPC (client headers, client message, client half-close, server headers, server message, and server trailers), the filter can be configured to send the contents of that event to the -ext_proc server. The specific behavior for each event is covered below. +ext_proc server. The responses from the ext_proc server will indicate +what modifications to make to the data plane RPC before allowing it to +proceed. The specific behavior for each event is covered below. For each data plane RPC, the first time the filter needs to send an event to the ext_proc server, the filter will create a stream to the ext_proc @@ -57,23 +59,6 @@ plane RPC will be done on that ext_proc stream. The filter will pass the trace context from the data plane RPC to the ext_proc RPC, so that the ext_proc RPC appears as a child span on the data plane RPC's trace. -If not in [observability mode](#observability-mode), then for each event -that the filter sends to the ext_proc server, it will wait for a response -from the ext_proc server before allowing that event to proceed on the data -plane RPC. The response from the ext_proc server will tell the filter -what modifications to make to the event before allowing it to proceed -on the data plane RPC (e.g., it may modify headers or message bodies). - -Each event should be sent on the ext_proc stream as it occurs, even if -the filter has not yet received a response from the ext_proc server -for a previous event, which means that there may be multiple events -in flight on the ext_proc stream at the same time. For example, -if the filter previously sent a message about client headers and is -waiting for a response for that event and then sees a client message, -it should immediately send the message on the ext_proc stream about the -client message event (assuming that it is configured to send the client -message event). - Note that the stream to the ext_proc server may be terminated at any time. If the stream terminates with OK status, that indicates to the filter that it no longer needs to send any more events to the ext_proc server for that @@ -84,6 +69,48 @@ be failed with UNAVAILABLE status. However, if the `failure_mode_allow` config field is set to true, then the data plane RPC will instead be allowed to continue, with no further action taken by the ext_proc filter. +#### Events on the ext_proc Stream + +On the ext_proc stream, the events sent and received must be in the same +order as on the data plane. For client-to-server events, the order must +be headers, followed by zero or more messages, followed by a half-close. +For server-to-client events, the order must be headers, followed by zero +or more messages, followed by trailers. It is fine to interleave +client-to-server events with server-to-client events, since the two +directions are independent of each other. It is also fine for some of +the events to be missing on the ext_proc stream; for example, if the +filter is not configured to send client headers but is configured to +send client messages, then it can start by sending a client message. +But if two different events are sent on the stream, they must be in the +correct order relative to each other, both to and from the ext_proc +server. + +When sending client headers, server headers, and server trailers events to +the ext_proc server, if not in [observability mode](#observability-mode), +the filter will hold on to the headers and wait for a response for that +event from the ext_proc server before allowing the event to proceed on +the data plane RPC. The response from the ext_proc server will tell +the filter what modifications to make to the headers before allowing +them to proceed on the data plane RPC. + +In contrast, when sending client or server messages to the ext_proc +server, if not in [observability mode](#observability-mode), the +filter will *not* hold on to the contents of those messages. Instead, +the ext_proc server will be responsible for sending back the full set of +messages to be used on the data plane RPC. That set of messages may have +absolutely no relationship to the messages originally sent to the ext_proc +server; the ext_proc server may decide to drop messages, modify messages, +pass messages back unmodified, or completely replace all of the messages +on the stream. Note that the number of messages produced by the ext_proc +server may not match the number of messages originally sent to it. + +Events on the data plane RPC should be sent on the ext_proc stream as +they occur, even if the filter has not yet received a response from the +ext_proc server for a previous event. For example, if the filter is +waiting for a response for client headers, if it sees a client message, +it can send the client message event on the ext_proc stream immediately +(assuming it is configured to send client messages). + #### Observability Mode If the @@ -120,20 +147,6 @@ one: special to handle flow control; it will simply not wait for the ext_proc response before sending the message on the data plane stream. -#### Flow Control and Streaming RPCs - -If not in observability mode, if the data plane RPC is a streaming RPC, -then there may be multiple client messages or server messages queued up -at the same time waiting for responses from the ext_proc server. Note -that we specifically do *not* want to wait until we see a response from -one message before sending the next message to the ext_proc server, -because that would add per-message latency for a full round-trip to the -ext_proc server, which would dramatically affect streaming performance. - -Implementations must ensure proper flow control for this situation. - -TODO: figure out details of how this will work - #### Payload Handling The existing ext_proc protocol's handling of request payloads has a @@ -176,9 +189,8 @@ Therefore, we propose adding a new ext_proc `BodySendMode` called `GRPC` ext_proc client would handle deframing the gRPC messages. It will send each gRPC message to the ext_proc server as a separate `request_body`, and the ext_proc server will send a separate response for each one. -This is a streaming mode, meaning that the ext_proc client may send -a subsequent message to the ext_proc server while still waiting for a -reply from the ext_proc server for a previous message. +This is a streaming mode similar to the existing `FULL_DUPLEX_STREAMED` +mode. The new `GRPC` mode will be the only processing mode supported in gRPC. It is be desirable for Envoy to implement the same mode, so that users @@ -597,6 +609,17 @@ sub-optimal approach, for the following reasons: which will make it awkward for gRPC to support other transports with different framing in the future. +We considered supporting non-streaming body send modes, but that would +have significantly increased latency for streaming RPCs, because we would +have needed to wait a full RTT with the ext_proc server between each +message on the stream. We also considered a more pipelined approach +where the data plane would have queued the messages and the ext_proc +server would have been required to send back a response for each message +indicating an optional replacement, but that approach would have (a) +not allowed the ext_proc server to modify the number of messages on +the stream and (b) would have required implementing some form of flow +control to impose push-back upon hitting some maximum buffer size. + ## Implementation Will be implemented in C-core, Java, Go, and Node. From 384985888768ce9ec1b0d6d38cf105f067ac429b Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Sat, 13 Sep 2025 00:13:55 +0000 Subject: [PATCH 14/27] use streamed body response, and add request_drain field --- A93-xds-ext-proc.md | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-) diff --git a/A93-xds-ext-proc.md b/A93-xds-ext-proc.md index 2bcd0fdbe..3a1a9863d 100644 --- a/A93-xds-ext-proc.md +++ b/A93-xds-ext-proc.md @@ -458,9 +458,14 @@ as follows: the filter will cancel the ext_proc stream and treat it as if it failed with a non-OK status (see above for details). - [body_mutation](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L317): - - [body](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L401): - Replaces the original serialized message. - - Note: We do not support clear_body or streamed_response. + - [streamed_response](https://github.com/envoyproxy/envoy/blob/564612e32eafc10a7a7fd490cdb5cc7149e5802b/api/envoy/service/ext_proc/v3/external_processor.proto#L447): + Replaces the original serialized message. Within it: + - [body](https://github.com/envoyproxy/envoy/blob/564612e32eafc10a7a7fd490cdb5cc7149e5802b/api/envoy/service/ext_proc/v3/external_processor.proto#L419): + The serialized message body. + - [end_of_stream](https://github.com/envoyproxy/envoy/blob/564612e32eafc10a7a7fd490cdb5cc7149e5802b/api/envoy/service/ext_proc/v3/external_processor.proto#L424C8-L424C21): + If true, indicates that a half-close should be sent after the + message. Honored only on client-to-server messages. + - Note: We do not support body or clear_body. - Note: We do not support header_mutation in response to a body. This field will be ignored in this context. - Note: We do not support trailers, since that works only with @@ -493,6 +498,12 @@ as follows: to gRPC. - [mode_override](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L189C60-L189C73): See [Processing Mode Override](#processing-mode-override) below. +- request_drain (new field being added in + https://github.com/envoyproxy/envoy/pull/38753): If true, the filter + will send a half-close on the ext_proc stream. It will then continue + sending message bodies received from the ext_proc server until the + ext_proc stream terminates with OK status. After that, any subsequent + message on the stream will be passed through as-is. - We will ignore override_message_timeout, since GRPC body send mode does not support timeouts. - We ignore the dynamic_metadata field, since it is not relevant to gRPC. From 1a896bb3d641fd1de555d2a5959dd5b9a515eb45 Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Tue, 16 Sep 2025 18:37:07 +0000 Subject: [PATCH 15/27] flesh out metrics --- A93-xds-ext-proc.md | 51 ++++++++++++++++++++++++++++++++++++++------- 1 file changed, 44 insertions(+), 7 deletions(-) diff --git a/A93-xds-ext-proc.md b/A93-xds-ext-proc.md index 3a1a9863d..5749c137b 100644 --- a/A93-xds-ext-proc.md +++ b/A93-xds-ext-proc.md @@ -4,7 +4,7 @@ A93: xDS ExtProc Support * Approver: @ejona86, @dfawley * Status: {Draft, In Review, Ready for Implementation, Implemented} * Implemented in: -* Last updated: 2025-09-12 +* Last updated: 2025-09-16 * Discussion at: (filled after thread exists) ## Abstract @@ -29,10 +29,18 @@ the bootstrap config, described in [A102]. It will also make use of the * [A39: xDS HTTP Filter Support][A39] * [A81: xDS Authority Rewriting][A81] * [A102: xDS GrpcService Support][A102] (pending) +* [A60: xDS-Based Stateful Session Affinity for Weighted Clusters][A60] +* [A79: Non-Per-Call Metrics Architecture][A79] +* [A66: OpenTelemetry Metrics][A66] +* [A89: Backend Service Metric Label][A89] [A39]: A39-xds-http-filters.md [A81]: A81-xds-authority-rewriting.md [A102]: https://github.com/grpc/proposal/pull/510 +[A60]: A60-xds-stateful-session-affinity-weighted-clusters.md +[A79]: A79-non-per-call-metrics-architecture.md +[A89]: A89-backend-service-metric-label.md +[A66]: A66-otel-stats.md ## Proposal @@ -199,12 +207,41 @@ breaking their ext_proc servers. #### Metrics -TODO: define metrics for tracking time between entering and exiting the -ext_proc filter for the following events: -- client headers -- client half-close -- server headers -- server trailers +The ext_authz filter will export metrics using the non-per-call metrics +architecture defined in [A79]. There will be a separate set of metrics +on client side and server side, because (a) there are additional labels +that are relevant on the client but not on the server, and (b) it may be +useful to differentiate between authorization behavior on the client vs. +the server. + +##### Client-Side Metrics + +The client-side metrics will have the following labels: + +| Name | Disposition | Description | +| ----------- | ----------- | ----------- | +| grpc.target | required | The target of the gRPC channel in which ext_authz is used, as the defined in [A66]. | +| grpc.lb.backend_service | optional | The backend service to which the traffic is being sent, as defined in [A89]. This will be populated from the xDS cluster name, which will be passed to the ext_authz filter as described in [A60]. | + +The following client-side metrics will be exported: + +| Name | Type | Unit | Labels | Description | +| ------------- | ----- | ----- | ------- | ----------- | +| grpc.client_ext_proc.client_headers_duration | Histogram | s | grpc.target, grpc.lb.backend_service | Time between when the ext_proc filter sees the client's headers and when it allows those headers to continue on to the next filter. | +| grpc.client_ext_proc.client_half_close_duration | Histogram | s | grpc.target, grpc.lb.backend_service | Time between when the ext_proc filter sees the client's half-close and when it allows that half-close to continue on to the next filter. | +| grpc.client_ext_proc.server_headers_duration | Histogram | s | grpc.target, grpc.lb.backend_service | Time between when the ext_proc filter sees the server's headers and when it allows those headers to continue on to the next filter. | +| grpc.client_ext_proc.server_trailers_duration | Histogram | s | grpc.target, grpc.lb.backend_service | Time between when the ext_proc filter sees the server's trailers and when it allows those trailers to continue on to the next filter. | + +##### Server-Side Metrics + +The following server-side metrics will be exported: + +| Name | Type | Unit | Labels | Description | +| ------------- | ----- | ----- | ------- | ----------- | +| grpc.server_ext_proc.client_headers_duration | Histogram | s | | Time between when the ext_proc filter sees the client's headers and when it allows those headers to continue on to the next filter. | +| grpc.server_ext_proc.client_half_close_duration | Histogram | s | | Time between when the ext_proc filter sees the client's half-close and when it allows that half-close to continue on to the next filter. | +| grpc.server_ext_proc.server_headers_duration | Histogram | s | | Time between when the ext_proc filter sees the server's headers and when it allows those headers to continue on to the next filter. | +| grpc.server_ext_proc.server_trailers_duration | Histogram | s | | Time between when the ext_proc filter sees the server's trailers and when it allows those trailers to continue on to the next filter. | ### Filter Configuration From e532a11f419190bd9226622685fbf53dc7fcf03f Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Tue, 16 Sep 2025 20:55:29 +0000 Subject: [PATCH 16/27] fix typo --- A93-xds-ext-proc.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/A93-xds-ext-proc.md b/A93-xds-ext-proc.md index 5749c137b..d661d0fdb 100644 --- a/A93-xds-ext-proc.md +++ b/A93-xds-ext-proc.md @@ -150,7 +150,7 @@ one: control push-back before doing a write. For the purposes of that API, Java will not consider the flow control available until it passes flow control on both the ext_proc stream and on the data plane stream. -- In Go, a write is blocking and doesn't return until the write passed +- In Go, a write is blocking and doesn't return until the write passes flow control. So observability mode doesn't need to do anything special to handle flow control; it will simply not wait for the ext_proc response before sending the message on the data plane stream. From 08079869f07721bb32a8c3bdc410aaa403d3b529 Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Tue, 16 Sep 2025 23:25:11 +0000 Subject: [PATCH 17/27] add end_of_stream_without_message fields --- A93-xds-ext-proc.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/A93-xds-ext-proc.md b/A93-xds-ext-proc.md index d661d0fdb..c91e4a75a 100644 --- a/A93-xds-ext-proc.md +++ b/A93-xds-ext-proc.md @@ -112,6 +112,10 @@ pass messages back unmodified, or completely replace all of the messages on the stream. Note that the number of messages produced by the ext_proc server may not match the number of messages originally sent to it. +A client half-close is typically represented to the ext_proc server as an +end-of-stream indicator on a header or message event, rather than being +a discrete event. + Events on the data plane RPC should be sent on the ext_proc stream as they occur, even if the filter has not yet received a response from the ext_proc server for a previous event. For example, if the filter is @@ -412,6 +416,12 @@ sent to the server will be populated as follows: For client messages, may be true if the client sent a half-close at the same time as the last message. For server messages, will always be false. + - end_of_stream_without_message (new field being added in + https://github.com/envoyproxy/envoy/pull/38753): Will be set to true + if the client sends a half-close when there is no message to send + (i.e., if the client never sent any message on the stream, or if + the half-close is sent after the filter has already sent the last + message to the ext_proc server). - [response_trailers](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L103). Populated when sending server trailers. Inside of it: - [trailers](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L247): @@ -502,6 +512,9 @@ as follows: - [end_of_stream](https://github.com/envoyproxy/envoy/blob/564612e32eafc10a7a7fd490cdb5cc7149e5802b/api/envoy/service/ext_proc/v3/external_processor.proto#L424C8-L424C21): If true, indicates that a half-close should be sent after the message. Honored only on client-to-server messages. + - end_of_stream_without_message (new field being added in + https://github.com/envoyproxy/envoy/pull/38753): Will be set to true + to indicate a half-close with no message to send. - Note: We do not support body or clear_body. - Note: We do not support header_mutation in response to a body. This field will be ignored in this context. From ccaee2117e7f97a9d52f16c35d1c6529b45c20d5 Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Tue, 16 Sep 2025 23:32:41 +0000 Subject: [PATCH 18/27] move metrics section --- A93-xds-ext-proc.md | 76 ++++++++++++++++++++++----------------------- 1 file changed, 38 insertions(+), 38 deletions(-) diff --git a/A93-xds-ext-proc.md b/A93-xds-ext-proc.md index c91e4a75a..b14b49098 100644 --- a/A93-xds-ext-proc.md +++ b/A93-xds-ext-proc.md @@ -209,44 +209,6 @@ It is be desirable for Envoy to implement the same mode, so that users can switch back and forth between proxy and proxyless data planes without breaking their ext_proc servers. -#### Metrics - -The ext_authz filter will export metrics using the non-per-call metrics -architecture defined in [A79]. There will be a separate set of metrics -on client side and server side, because (a) there are additional labels -that are relevant on the client but not on the server, and (b) it may be -useful to differentiate between authorization behavior on the client vs. -the server. - -##### Client-Side Metrics - -The client-side metrics will have the following labels: - -| Name | Disposition | Description | -| ----------- | ----------- | ----------- | -| grpc.target | required | The target of the gRPC channel in which ext_authz is used, as the defined in [A66]. | -| grpc.lb.backend_service | optional | The backend service to which the traffic is being sent, as defined in [A89]. This will be populated from the xDS cluster name, which will be passed to the ext_authz filter as described in [A60]. | - -The following client-side metrics will be exported: - -| Name | Type | Unit | Labels | Description | -| ------------- | ----- | ----- | ------- | ----------- | -| grpc.client_ext_proc.client_headers_duration | Histogram | s | grpc.target, grpc.lb.backend_service | Time between when the ext_proc filter sees the client's headers and when it allows those headers to continue on to the next filter. | -| grpc.client_ext_proc.client_half_close_duration | Histogram | s | grpc.target, grpc.lb.backend_service | Time between when the ext_proc filter sees the client's half-close and when it allows that half-close to continue on to the next filter. | -| grpc.client_ext_proc.server_headers_duration | Histogram | s | grpc.target, grpc.lb.backend_service | Time between when the ext_proc filter sees the server's headers and when it allows those headers to continue on to the next filter. | -| grpc.client_ext_proc.server_trailers_duration | Histogram | s | grpc.target, grpc.lb.backend_service | Time between when the ext_proc filter sees the server's trailers and when it allows those trailers to continue on to the next filter. | - -##### Server-Side Metrics - -The following server-side metrics will be exported: - -| Name | Type | Unit | Labels | Description | -| ------------- | ----- | ----- | ------- | ----------- | -| grpc.server_ext_proc.client_headers_duration | Histogram | s | | Time between when the ext_proc filter sees the client's headers and when it allows those headers to continue on to the next filter. | -| grpc.server_ext_proc.client_half_close_duration | Histogram | s | | Time between when the ext_proc filter sees the client's half-close and when it allows that half-close to continue on to the next filter. | -| grpc.server_ext_proc.server_headers_duration | Histogram | s | | Time between when the ext_proc filter sees the server's headers and when it allows those headers to continue on to the next filter. | -| grpc.server_ext_proc.server_trailers_duration | Histogram | s | | Time between when the ext_proc filter sees the server's trailers and when it allows those trailers to continue on to the next filter. | - ### Filter Configuration The filter supports both a top-level configuration and an override @@ -646,6 +608,44 @@ for each data plane RPC. The processing mode for an RPC will be initialized based on the filter's config, but it may be modified later by subsequent overrides. +### Metrics + +The ext_authz filter will export metrics using the non-per-call metrics +architecture defined in [A79]. There will be a separate set of metrics +on client side and server side, because (a) there are additional labels +that are relevant on the client but not on the server, and (b) it may be +useful to differentiate between authorization behavior on the client vs. +the server. + +#### Client-Side Metrics + +The client-side metrics will have the following labels: + +| Name | Disposition | Description | +| ----------- | ----------- | ----------- | +| grpc.target | required | The target of the gRPC channel in which ext_authz is used, as the defined in [A66]. | +| grpc.lb.backend_service | optional | The backend service to which the traffic is being sent, as defined in [A89]. This will be populated from the xDS cluster name, which will be passed to the ext_authz filter as described in [A60]. | + +The following client-side metrics will be exported: + +| Name | Type | Unit | Labels | Description | +| ------------- | ----- | ----- | ------- | ----------- | +| grpc.client_ext_proc.client_headers_duration | Histogram | s | grpc.target, grpc.lb.backend_service | Time between when the ext_proc filter sees the client's headers and when it allows those headers to continue on to the next filter. | +| grpc.client_ext_proc.client_half_close_duration | Histogram | s | grpc.target, grpc.lb.backend_service | Time between when the ext_proc filter sees the client's half-close and when it allows that half-close to continue on to the next filter. | +| grpc.client_ext_proc.server_headers_duration | Histogram | s | grpc.target, grpc.lb.backend_service | Time between when the ext_proc filter sees the server's headers and when it allows those headers to continue on to the next filter. | +| grpc.client_ext_proc.server_trailers_duration | Histogram | s | grpc.target, grpc.lb.backend_service | Time between when the ext_proc filter sees the server's trailers and when it allows those trailers to continue on to the next filter. | + +#### Server-Side Metrics + +The following server-side metrics will be exported: + +| Name | Type | Unit | Labels | Description | +| ------------- | ----- | ----- | ------- | ----------- | +| grpc.server_ext_proc.client_headers_duration | Histogram | s | | Time between when the ext_proc filter sees the client's headers and when it allows those headers to continue on to the next filter. | +| grpc.server_ext_proc.client_half_close_duration | Histogram | s | | Time between when the ext_proc filter sees the client's half-close and when it allows that half-close to continue on to the next filter. | +| grpc.server_ext_proc.server_headers_duration | Histogram | s | | Time between when the ext_proc filter sees the server's headers and when it allows those headers to continue on to the next filter. | +| grpc.server_ext_proc.server_trailers_duration | Histogram | s | | Time between when the ext_proc filter sees the server's trailers and when it allows those trailers to continue on to the next filter. | + ### Temporary environment variable protection TODO: do we need separate env vars for client and server sides? From 51add0a306f4228b61ccb675cdc09dd30b9b2184 Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Tue, 16 Sep 2025 23:47:47 +0000 Subject: [PATCH 19/27] spell out drain workflow --- A93-xds-ext-proc.md | 111 +++++++++++++++++++++++++++----------------- 1 file changed, 68 insertions(+), 43 deletions(-) diff --git a/A93-xds-ext-proc.md b/A93-xds-ext-proc.md index b14b49098..c7cb25280 100644 --- a/A93-xds-ext-proc.md +++ b/A93-xds-ext-proc.md @@ -67,16 +67,6 @@ plane RPC will be done on that ext_proc stream. The filter will pass the trace context from the data plane RPC to the ext_proc RPC, so that the ext_proc RPC appears as a child span on the data plane RPC's trace. -Note that the stream to the ext_proc server may be terminated at any time. -If the stream terminates with OK status, that indicates to the filter that -it no longer needs to send any more events to the ext_proc server for that -data plane RPC; all remaining events may proceed on the data plane RPC -without any further action taken by the ext_proc filter. If the stream -terminates with a non-OK status, then by default the data plane RPC will -be failed with UNAVAILABLE status. However, if the `failure_mode_allow` -config field is set to true, then the data plane RPC will instead be -allowed to continue, with no further action taken by the ext_proc filter. - #### Events on the ext_proc Stream On the ext_proc stream, the events sent and received must be in the same @@ -112,7 +102,7 @@ pass messages back unmodified, or completely replace all of the messages on the stream. Note that the number of messages produced by the ext_proc server may not match the number of messages originally sent to it. -A client half-close is typically represented to the ext_proc server as an +A client half-close is represented to the ext_proc server as an end-of-stream indicator on a header or message event, rather than being a discrete event. @@ -123,41 +113,40 @@ waiting for a response for client headers, if it sees a client message, it can send the client message event on the ext_proc stream immediately (assuming it is configured to send client messages). -#### Observability Mode +#### Early Termination of ext_proc Stream -If the -[`observability_mode`](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L283) -config field is set to true, the contents of the data plane RPC events -are sent to the ext_proc server, but the ext_proc server is not expected -to make any modifications to the data plane RPC, so the data plane -RPC proceeds without waiting for a response from the ext_proc server. -Note that in this mode, all messages on the ext_proc stream will have the -[`observability_mode`](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L126) -field set. - -One challenge in this mode is flow control: if we are blocked sending -a message to the ext_proc server by flow control, then we need some -push-back on the data plane RPC, or else we would have to buffer messages -to be sent to the ext_proc server, which can cause OOMs. (Envoy has -noted this problem in https://github.com/envoyproxy/envoy/issues/33319 -but has not proposed a solution yet.) +The stream to the ext_proc server may be terminated at any time. -For gRPC, we will address this problem by requiring the message to the -ext_proc server to pass flow control before we allow the message to -proceed on the data plane RPC. Due to differences in flow control -semantics across languages, this will look a little different in each -one: +If the stream terminates with a non-OK status, then by default the +data plane RPC will be failed with UNAVAILABLE status. However, if the +`failure_mode_allow` config field is set to true, then the data plane +RPC will instead be allowed to continue, with no further action taken +by the ext_proc filter. -- In C-core, we will wait for the write to complete on the ext_proc - stream before we allow the message to continue on the data plane RPC. -- In Java, the application has to explicitly check whether there is flow - control push-back before doing a write. For the purposes of that API, - Java will not consider the flow control available until it passes flow - control on both the ext_proc stream and on the data plane stream. -- In Go, a write is blocking and doesn't return until the write passes - flow control. So observability mode doesn't need to do anything - special to handle flow control; it will simply not wait for the - ext_proc response before sending the message on the data plane stream. +If the stream terminates with OK status, that indicates to the filter that +it no longer needs to send any more events to the ext_proc server for that +data plane RPC; all remaining events may proceed on the data plane RPC +without any further action taken by the ext_proc filter. However, in +streaming RPCs, it may be necessary for the ext_proc server to drain the +stream before terminating. This is because the filter may have already sent +messages on the stream that the server has not yet seen, and if the +server never echoes them back, then the messages will simply be dropped +from the stream. In order to avoid that, the following drain procedure +will be used: + +1. The ext_proc server will send an ext_proc response with the + `request_drain` field (see https://github.com/envoyproxy/envoy/pull/38753) + set to true. The filter will react by sending a half-close on the + ext_proc stream. Note that at this point, the filter must stop + reading messages from the data plane stream, so that flow control + push-back occurs. +2. The ext_proc server will echo all messages received from the filter + back to the filter without modification, until it sees the half close + from the filter. It will then terminate the ext_proc stream with OK + status. +3. When the filter sees the ext_proc stream terminate with OK status, it + will resume reading messages from the data plane stream, and all such + messages will proceed on the data plane stream without modification. #### Payload Handling @@ -209,6 +198,42 @@ It is be desirable for Envoy to implement the same mode, so that users can switch back and forth between proxy and proxyless data planes without breaking their ext_proc servers. +#### Observability Mode + +If the +[`observability_mode`](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L283) +config field is set to true, the contents of the data plane RPC events +are sent to the ext_proc server, but the ext_proc server is not expected +to make any modifications to the data plane RPC, so the data plane +RPC proceeds without waiting for a response from the ext_proc server. +Note that in this mode, all messages on the ext_proc stream will have the +[`observability_mode`](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L126) +field set. + +One challenge in this mode is flow control: if we are blocked sending +a message to the ext_proc server by flow control, then we need some +push-back on the data plane RPC, or else we would have to buffer messages +to be sent to the ext_proc server, which can cause OOMs. (Envoy has +noted this problem in https://github.com/envoyproxy/envoy/issues/33319 +but has not proposed a solution yet.) + +For gRPC, we will address this problem by requiring the message to the +ext_proc server to pass flow control before we allow the message to +proceed on the data plane RPC. Due to differences in flow control +semantics across languages, this will look a little different in each +one: + +- In C-core, we will wait for the write to complete on the ext_proc + stream before we allow the message to continue on the data plane RPC. +- In Java, the application has to explicitly check whether there is flow + control push-back before doing a write. For the purposes of that API, + Java will not consider the flow control available until it passes flow + control on both the ext_proc stream and on the data plane stream. +- In Go, a write is blocking and doesn't return until the write passes + flow control. So observability mode doesn't need to do anything + special to handle flow control; it will simply not wait for the + ext_proc response before sending the message on the data plane stream. + ### Filter Configuration The filter supports both a top-level configuration and an override From f4831252fd96bf86a1af60c34d6fc0a9614d61d3 Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Wed, 17 Sep 2025 19:54:05 +0000 Subject: [PATCH 20/27] document filter state retention --- A93-xds-ext-proc.md | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-) diff --git a/A93-xds-ext-proc.md b/A93-xds-ext-proc.md index c7cb25280..708e91fd7 100644 --- a/A93-xds-ext-proc.md +++ b/A93-xds-ext-proc.md @@ -4,7 +4,7 @@ A93: xDS ExtProc Support * Approver: @ejona86, @dfawley * Status: {Draft, In Review, Ready for Implementation, Implemented} * Implemented in: -* Last updated: 2025-09-16 +* Last updated: 2025-09-17 * Discussion at: (filled after thread exists) ## Abstract @@ -28,6 +28,7 @@ the bootstrap config, described in [A102]. It will also make use of the ### Related Proposals: * [A39: xDS HTTP Filter Support][A39] * [A81: xDS Authority Rewriting][A81] +* [A83: xDS GCP Authentication Filter][A83] * [A102: xDS GrpcService Support][A102] (pending) * [A60: xDS-Based Stateful Session Affinity for Weighted Clusters][A60] * [A79: Non-Per-Call Metrics Architecture][A79] @@ -36,6 +37,7 @@ the bootstrap config, described in [A102]. It will also make use of the [A39]: A39-xds-http-filters.md [A81]: A81-xds-authority-rewriting.md +[A83]: A83-xds-gcp-authn-filter.md [A102]: https://github.com/grpc/proposal/pull/510 [A60]: A60-xds-stateful-session-affinity-weighted-clusters.md [A79]: A79-non-per-call-metrics-architecture.md @@ -49,9 +51,6 @@ server side. ### Filter Behavior -TODO: ExtProc channel retention (simple approach for now, probably -globally shared channel is fine?) - For every event in the data plane RPC (client headers, client message, client half-close, server headers, server message, and server trailers), the filter can be configured to send the contents of that event to the @@ -67,6 +66,19 @@ plane RPC will be done on that ext_proc stream. The filter will pass the trace context from the data plane RPC to the ext_proc RPC, so that the ext_proc RPC appears as a child span on the data plane RPC's trace. +#### ExtProc Side Channel + +The ext_proc filter will create a gRPC channel to the ext_proc server. +It will use the mechanism described in [A102] to determine the server to +talk to and the channel credentials to use. + +We do not want to recreate this channel on LDS updates, unless the +target URI or channel credentials changes. The ext_proc filter will +use the filter state retention mechanism described in [A83] to retain +the channel across updates. + +TODO: stats plugin propagation? + #### Events on the ext_proc Stream On the ext_proc stream, the events sent and received must be in the same From 3404ced40af13fa65490d5afd2a8b4af3a21c1b5 Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Thu, 18 Sep 2025 22:58:13 +0000 Subject: [PATCH 21/27] add mailing list link --- A93-xds-ext-proc.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/A93-xds-ext-proc.md b/A93-xds-ext-proc.md index 708e91fd7..53d72a9e9 100644 --- a/A93-xds-ext-proc.md +++ b/A93-xds-ext-proc.md @@ -5,7 +5,7 @@ A93: xDS ExtProc Support * Status: {Draft, In Review, Ready for Implementation, Implemented} * Implemented in: * Last updated: 2025-09-17 -* Discussion at: (filled after thread exists) +* Discussion at: https://groups.google.com/g/grpc-io/c/AqqG4kkUc08 ## Abstract From 17a8fe4df37159d7a862d64dd5e5a36d036d9a9e Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Mon, 29 Sep 2025 23:37:04 +0000 Subject: [PATCH 22/27] improve description of body send mode --- A93-xds-ext-proc.md | 23 +++++++++-------------- 1 file changed, 9 insertions(+), 14 deletions(-) diff --git a/A93-xds-ext-proc.md b/A93-xds-ext-proc.md index 53d72a9e9..5df5461fe 100644 --- a/A93-xds-ext-proc.md +++ b/A93-xds-ext-proc.md @@ -4,7 +4,7 @@ A93: xDS ExtProc Support * Approver: @ejona86, @dfawley * Status: {Draft, In Review, Ready for Implementation, Implemented} * Implemented in: -* Last updated: 2025-09-17 +* Last updated: 2025-09-29 * Discussion at: https://groups.google.com/g/grpc-io/c/AqqG4kkUc08 ## Abstract @@ -181,13 +181,6 @@ work for the ext_proc server to do, so it seems better for the ext_proc client to do the work of deframing the gRPC messages, and sending only complete deframed messages to the ext_proc server, one at a time. -More seriously, ext_proc currently assumes that all of the contents of -the DATA frames are a single payload, and the ext_proc server is only -allowed to send a single response to modify that payload. That model is -incompatible with gRPC streaming semantics, where the ext_proc server -may need to modify each message individually and cannot wait until the -end of the stream to do so. - Furthermore, the ext_proc filter in gRPC will not actually have access to the raw HTTP/2 DATA frames in the first place. In our architecture, the framing/deframing is handled in the transport layer, and filters @@ -198,12 +191,14 @@ the gRPC server side will see the messages after they have been received and deframed by the transport. Therefore, we propose adding a new ext_proc `BodySendMode` called `GRPC` -(see https://github.com/envoyproxy/envoy/pull/38753). In this mode, the -ext_proc client would handle deframing the gRPC messages. It will send -each gRPC message to the ext_proc server as a separate `request_body`, -and the ext_proc server will send a separate response for each one. -This is a streaming mode similar to the existing `FULL_DUPLEX_STREAMED` -mode. +(see https://github.com/envoyproxy/envoy/pull/38753). This mode will +be similar to the existing `FULL_DUPLEX_STREAMED` mode, in which the +ext_proc client sends body chunks from the data plane stream to the +ext_proc server as it sees them, and the ext_proc server sends back body +chunks to be used on the data plane stream. However, in the new `GRPC` +mode, the ext_proc client will handle the framing and deframing of gRPC +messages, so each body chunk sent to or from the ext_proc server will +be a complete, deframed gRPC message. The new `GRPC` mode will be the only processing mode supported in gRPC. It is be desirable for Envoy to implement the same mode, so that users From 376ecd7530d57679b6b209aaca90ef6ea823d36e Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Tue, 7 Oct 2025 19:12:24 +0000 Subject: [PATCH 23/27] document interaction with compression --- A93-xds-ext-proc.md | 42 +++++++++++++++++++++++++++++++++++++++++- 1 file changed, 41 insertions(+), 1 deletion(-) diff --git a/A93-xds-ext-proc.md b/A93-xds-ext-proc.md index 5df5461fe..8945b007d 100644 --- a/A93-xds-ext-proc.md +++ b/A93-xds-ext-proc.md @@ -4,7 +4,7 @@ A93: xDS ExtProc Support * Approver: @ejona86, @dfawley * Status: {Draft, In Review, Ready for Implementation, Implemented} * Implemented in: -* Last updated: 2025-09-29 +* Last updated: 2025-10-07 * Discussion at: https://groups.google.com/g/grpc-io/c/AqqG4kkUc08 ## Abstract @@ -205,6 +205,40 @@ It is be desirable for Envoy to implement the same mode, so that users can switch back and forth between proxy and proxyless data planes without breaking their ext_proc servers. +#### Interaction With Compression + +There are a couple of unfortunate interactions between ext_proc and +compression. + +In gRPC, the ext_proc filter will be above the compression code, which +means that messages sent to the ext_proc server will always be +uncompressed: for sent messages, compression will not happen until after +the ext_proc filter runs, and for received messages, decompression will +happen before the ext_proc filter runs. However, in Envoy, the messages +may be either compressed or uncompressed, depending on whether the +sender compressed them before sending them to Envoy. To expose this +difference to the ext_proc server, we have added a new field called +`grpc_message_compressed` in both the ext_proc request and response +message (see https://github.com/envoyproxy/envoy/pull/38753). Envoy +will set this bit in the ext_authz request based on the corresponding +bit in the gRPC framing header, and it will propagate the bit from the +ext_proc response back to the gRPC framing header. In gRPC, if an +ext_proc response sets this bit, the ext_proc filter will cancel the +ext_proc stream and treat it as if it failed with a non-OK status. + +Note that there are security reasons why some messages should not be +compressed: it is insecure to compress sensitive payloads, as it provides +a side-channel to determine the unencrypted contents. To address this, +gRPC's API allows applications to indicate that a given message should not +be compressed. However, because the messages sent back by the ext_proc +server are not required to map one-to-one to the messages sent to the +ext_proc server, there is currently no way to preserve the bit set +by the application. For now, the ext_proc server will be required to +disable compression for messages received back from the ext_proc server. +If this becomes a problem in the future, we can consider adding yet +another knob to the ext_proc protocol to pass this bit from the data +plane to the ext_proc server and back. + #### Observability Mode If the @@ -416,6 +450,8 @@ sent to the server will be populated as follows: (i.e., if the client never sent any message on the stream, or if the half-close is sent after the filter has already sent the last message to the ext_proc server). + - grpc_message_compressed (new field being added in + https://github.com/envoyproxy/envoy/pull/38753): Never set. - [response_trailers](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L103). Populated when sending server trailers. Inside of it: - [trailers](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L247): @@ -509,6 +545,10 @@ as follows: - end_of_stream_without_message (new field being added in https://github.com/envoyproxy/envoy/pull/38753): Will be set to true to indicate a half-close with no message to send. + - grpc_message_compressed (new field being added in + https://github.com/envoyproxy/envoy/pull/38753): If set to + true, the filter will cancel the ext_proc stream and treat it + as if it failed with a non-OK status (see above for details). - Note: We do not support body or clear_body. - Note: We do not support header_mutation in response to a body. This field will be ignored in this context. From e2da6bd3acd59203c457a41084ac45129701d421 Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Tue, 7 Oct 2025 20:39:49 +0000 Subject: [PATCH 24/27] fix typo --- A93-xds-ext-proc.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/A93-xds-ext-proc.md b/A93-xds-ext-proc.md index 8945b007d..3f6b5add9 100644 --- a/A93-xds-ext-proc.md +++ b/A93-xds-ext-proc.md @@ -233,7 +233,7 @@ gRPC's API allows applications to indicate that a given message should not be compressed. However, because the messages sent back by the ext_proc server are not required to map one-to-one to the messages sent to the ext_proc server, there is currently no way to preserve the bit set -by the application. For now, the ext_proc server will be required to +by the application. For now, the ext_proc filter will be required to disable compression for messages received back from the ext_proc server. If this becomes a problem in the future, we can consider adding yet another knob to the ext_proc protocol to pass this bit from the data From ef27648dbfe6c5c2a3b3a130998bb82bde26cb63 Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Fri, 10 Oct 2025 19:27:19 +0000 Subject: [PATCH 25/27] refer back to A92 for header rewriting --- A93-xds-ext-proc.md | 22 ++++------------------ 1 file changed, 4 insertions(+), 18 deletions(-) diff --git a/A93-xds-ext-proc.md b/A93-xds-ext-proc.md index 3f6b5add9..f66edc4fd 100644 --- a/A93-xds-ext-proc.md +++ b/A93-xds-ext-proc.md @@ -4,7 +4,7 @@ A93: xDS ExtProc Support * Approver: @ejona86, @dfawley * Status: {Draft, In Review, Ready for Implementation, Implemented} * Implemented in: -* Last updated: 2025-10-07 +* Last updated: 2025-10-10 * Discussion at: https://groups.google.com/g/grpc-io/c/AqqG4kkUc08 ## Abstract @@ -34,6 +34,7 @@ the bootstrap config, described in [A102]. It will also make use of the * [A79: Non-Per-Call Metrics Architecture][A79] * [A66: OpenTelemetry Metrics][A66] * [A89: Backend Service Metric Label][A89] +* [A92: xDS ExtAuthz][A92] (WIP) [A39]: A39-xds-http-filters.md [A81]: A81-xds-authority-rewriting.md @@ -43,6 +44,7 @@ the bootstrap config, described in [A102]. It will also make use of the [A79]: A79-non-per-call-metrics-architecture.md [A89]: A89-backend-service-metric-label.md [A66]: A66-otel-stats.md +[A92]: https://github.com/grpc/proposal/pull/481 ## Proposal @@ -607,23 +609,7 @@ event, the ext_proc server can return a message that says how to mutate the headers. That message will be handled as follows: - [set_headers](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L375): - Headers to set or mutate. Inside this message: - - [header](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/config/core/v3/base.proto#L458): - Required. Within it: - - [key](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/config/core/v3/base.proto#L404): - The header name. Must be non-empty and all lower-case. Length - must not exceed 16384. The entry will be ignored if the key is - `host` or starts with a `:`. - - [raw_value](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/config/core/v3/base.proto#L422): - The header value. Length must not exceed 16384. - - The value field will be ignored. - - [append_action](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/config/core/v3/base.proto#L476): - We honor the 4 enum values as described in the proto file. - - [keep_empty_value](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/config/core/v3/base.proto#L480): - By default, any header mutation that results in a header with an - empty value will cause the header key to be removed. If this field - is set to true, then such empty headers will be kept. - - We do not support the deprecated append field. + Headers to set or mutate. This will be handled as described in [A92]. - [remove_headers](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L379): Header names to remove. The filter will ignore `host` and any entry that starts with `:`. From 36fb214870a27ead4fe20fe741644161bf750ed2 Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Fri, 10 Oct 2025 23:54:39 +0000 Subject: [PATCH 26/27] refer to A92 also for header mutation rules --- A93-xds-ext-proc.md | 36 ++++++++++++++++-------------------- 1 file changed, 16 insertions(+), 20 deletions(-) diff --git a/A93-xds-ext-proc.md b/A93-xds-ext-proc.md index f66edc4fd..ce7d2a75f 100644 --- a/A93-xds-ext-proc.md +++ b/A93-xds-ext-proc.md @@ -324,20 +324,7 @@ proto](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4f any unsupported attribute name will be ignored. See [Attributes Sent to ext_proc Server](#attributes-sent-to-the-ext_proc-server) below for details. - [mutation_rules](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L225): - Optional. Inside of it: - - [disallow_all](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/config/common/mutation_rules/v3/mutation_rules.proto#L70): - If true, disallows all header mutations from the ext_proc server. - - [allow_expression](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/config/common/mutation_rules/v3/mutation_rules.proto#L75): - If set, specifically allows any matching header that is not also - matched by `disallow_expression`. Note that regexes should be checked - for validity as part of resource validation. - - [disallow_expression](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/config/common/mutation_rules/v3/mutation_rules.proto#L79): - If set, specifically disallows any matching header. Note that regexes - should be checked for validity as part of resource validation. - - [disallow_is_error](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/config/common/mutation_rules/v3/mutation_rules.proto#L87): - If true, a disallowed header will cause the filter to fail the data - plane RPC with INTERNAL status. - - allow_all_routing, disallow_system, allow_envoy: These fields will be ignored. + Optional. See [Header Mutations](#header-mutations) for details. - [forward_rules](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto#L237): Configures which headers from the data plane RPC will be included in the message to the ext_proc server. In this message: @@ -518,7 +505,7 @@ as follows: the filter will cancel the ext_proc stream and treat it as if it failed with a non-OK status (see above for details). - [header_mutation](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L308): - Header mutations. See [Header rewriting](#header-rewriting) below. + Header mutations. See [Header Mutations](#header-mutations) below. - Note: We do not support body_mutation in response to headers. This field will be ignored in this context. - Note: We do not support trailers, since that works only with @@ -560,7 +547,7 @@ as follows: - [response_trailers](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L159): Sent in response to server trailers. Inside of it: - [header_mutation](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L273): - Header mutations. See [Header rewriting](#header-rewriting) below. + Header mutations. See [Header Mutations](#header-mutations) below. - [immediate_response](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L168): May be sent in response to any message from the data plane. If sent in response to a server trailers event, sets the status and optionally @@ -601,7 +588,10 @@ ext_proc server responds to the client message event first, that is considered a protocol error. The filter will treat that as if the ext_proc stream failed with a non-OK status. -#### Header Rewriting +#### Header Mutations + +The ext_proc filter will leverage the same header mutation types as +defined in the ext_authz filter in [A92]. When responding to a client headers, server headers, or server trailers event, the ext_proc server can return a @@ -609,10 +599,16 @@ event, the ext_proc server can return a message that says how to mutate the headers. That message will be handled as follows: - [set_headers](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L375): - Headers to set or mutate. This will be handled as described in [A92]. + Headers to add or modify. This is a repeated + `envoy.config.core.v3.HeaderValueOption` message, which will be handled + as described in [A92]. - [remove_headers](https://github.com/envoyproxy/envoy/blob/cdd19052348f7f6d85910605d957ba4fe0538aec/api/envoy/service/ext_proc/v3/external_processor.proto#L379): - Header names to remove. The filter will ignore `host` and any entry - that starts with `:`. + Headers to remove. + +All mutations (additions, modifications, and removals) are subject to the +rules in the `mutation_rules` field in the filter config. This field is +an `envoy.config.common.mutation_rules.v3.HeaderMutationRules` message, +which will be handled as described in [A92]. #### Processing Mode Override From 9f7014327341fd5d61f2343a7da8ab7aab93cbb6 Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Wed, 24 Dec 2025 01:22:59 +0000 Subject: [PATCH 27/27] separate env var guards on client and server side --- A93-xds-ext-proc.md | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/A93-xds-ext-proc.md b/A93-xds-ext-proc.md index ce7d2a75f..626691f3c 100644 --- a/A93-xds-ext-proc.md +++ b/A93-xds-ext-proc.md @@ -4,7 +4,7 @@ A93: xDS ExtProc Support * Approver: @ejona86, @dfawley * Status: {Draft, In Review, Ready for Implementation, Implemented} * Implemented in: -* Last updated: 2025-10-10 +* Last updated: 2025-12-23 * Discussion at: https://groups.google.com/g/grpc-io/c/AqqG4kkUc08 ## Abstract @@ -702,11 +702,15 @@ The following server-side metrics will be exported: ### Temporary environment variable protection -TODO: do we need separate env vars for client and server sides? +Because implementations are likely to add support for ext_proc on the +client and server sides at different times, we will have a separate +environment variable guarding the functionality on each side. -Support for the ext_proc filter will be guarded by the -`GRPC_EXPERIMENTAL_XDS_EXT_PROC` environment variable. This guard will -be removed once the feature passes interop tests. +Support for the ext_proc filter will be guarded +by the `GRPC_EXPERIMENTAL_XDS_EXT_PROC_ON_CLIENT` and +`GRPC_EXPERIMENTAL_XDS_EXT_PROC_ON_SERVER` environment variables on the +client and server sides respectively. These guards will be removed once +the feature passes interop tests. ## Rationale