-
Notifications
You must be signed in to change notification settings - Fork 431
Description
Is your feature request related to a problem? Please describe.
When calling Agent Engine from a service instrumented with OpenTelemetry (e.g., Cloud Run), distributed traces break at the Agent Engine boundary. The traceparent header is sent but Agent Engine does not observe or propagate it.
This makes it impossible to correlate the calling service's spans with Agent Engine's internal operations in Cloud Trace. I can't see end-to-end latency, diagnose where time is spent (model vs. tools vs. queue), or correlate Agent Engine errors with the calling request context.
Describe the solution you'd like
Agent Engine should participate in distributed tracing by:
- Extracting the incoming traceparent header per the W3C TraceContext spec
- Creating internal spans (model invocation, tool execution) as children of the caller's trace
- Propagating trace context to any downstream calls (webhooks, tool endpoints)
This would allow full visibility in Cloud Trace: Cloud Run → Agent Engine → model → tools → response as a single correlated trace.
Describe alternatives you've considered
- Client-side timing: I can measure round-trip time from the caller, but this gives no visibility into Agent Engine internals (model latency vs. tool execution vs. queuing).
- Log correlation: I could try to correlate logs by timestamp, but this is fragile and doesn't provide the span-level granularity that tracing offers.
- Custom instrumentation in tools: I can propagate trace context to my own tool endpoints, but this doesn't help with Agent Engine's internal spans or model calls.
None of these provide the observability that proper trace context propagation would enable.
Additional context
- Environment: Cloud Run (Python) → Agent Engine via google-adk / REST API, with OpenTelemetry exporting to Cloud Trace
- Also filed on Google Issue Tracker for visibility with the Agent Engine team: https://issuetracker.google.com/u/4/issues/484226257 – cross-posting here for broader community visibility, and gathering thumbs up to encourage prioritization of this feature.
Thumbs up if you want/need this in Agent Engine.