Skip to content

Agent Engine: Support traceparent header propagation for distributed tracing #6341

@nino-robotfutures-co

Description

@nino-robotfutures-co

Is your feature request related to a problem? Please describe.

When calling Agent Engine from a service instrumented with OpenTelemetry (e.g., Cloud Run), distributed traces break at the Agent Engine boundary. The traceparent header is sent but Agent Engine does not observe or propagate it.

This makes it impossible to correlate the calling service's spans with Agent Engine's internal operations in Cloud Trace. I can't see end-to-end latency, diagnose where time is spent (model vs. tools vs. queue), or correlate Agent Engine errors with the calling request context.

Describe the solution you'd like

Agent Engine should participate in distributed tracing by:

  1. Extracting the incoming traceparent header per the W3C TraceContext spec
  2. Creating internal spans (model invocation, tool execution) as children of the caller's trace
  3. Propagating trace context to any downstream calls (webhooks, tool endpoints)

This would allow full visibility in Cloud Trace: Cloud Run → Agent Engine → model → tools → response as a single correlated trace.

Describe alternatives you've considered

  • Client-side timing: I can measure round-trip time from the caller, but this gives no visibility into Agent Engine internals (model latency vs. tool execution vs. queuing).
  • Log correlation: I could try to correlate logs by timestamp, but this is fragile and doesn't provide the span-level granularity that tracing offers.
  • Custom instrumentation in tools: I can propagate trace context to my own tool endpoints, but this doesn't help with Agent Engine's internal spans or model calls.

None of these provide the observability that proper trace context propagation would enable.

Additional context

  • Environment: Cloud Run (Python) → Agent Engine via google-adk / REST API, with OpenTelemetry exporting to Cloud Trace
  • Also filed on Google Issue Tracker for visibility with the Agent Engine team: https://issuetracker.google.com/u/4/issues/484226257 – cross-posting here for broader community visibility, and gathering thumbs up to encourage prioritization of this feature.

Thumbs up if you want/need this in Agent Engine.

Metadata

Metadata

Assignees

No one assigned

    Labels

    api: vertex-aiIssues related to the googleapis/python-aiplatform API.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions