-
Notifications
You must be signed in to change notification settings - Fork 3
feat: add Grafana Tempo for tracing #79
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Conversation
overlays/monitoring/config/grafana/provisioning/datasources/datasources.yml
Show resolved
Hide resolved
Add profiling overlay with Grafana Tempo and OpenTelemetry Java Agent for distributed tracing of DHIS2 requests. Changes include: - Add Tempo service for trace storage and querying - Add OpenTelemetry Java Agent download with SHA256 verification - Configure trace context in log4j2.xml for log correlation - Add httpMethod: POST to Prometheus datasource for long queries - Add Tempo datasource with trace-to-log and trace-to-metric links - Create detailed README for the profiling overlay - Document JAVA_TOOL_OPTIONS override behavior Signed-off-by: Morten Svanaes <msvanaes@dhis2.org>
The SHA256 checksum for opentelemetry-javaagent.jar v2.11.0 was incorrect, causing the otel-init container to fail verification. Correct hash: 4cff4ab46179260a61fc0d884f3f170cfbd9d2962dd260be2cff31262d0c7618 Signed-off-by: Morten Svanaes <msvanaes@dhis2.org>
radnov
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good! Minor comment added.
bobjolliffe
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Few small comments on tempo config.
| docker compose run --rm compose-docs > docs/environment-variables.md | ||
|
|
||
| COMPOSE_CMD = docker compose -f docker-compose.yml -f overlays/traefik-dashboard/docker-compose.yml -f overlays/monitoring/docker-compose.yml -f overlays/glowroot/docker-compose.yml | ||
| COMPOSE_CMD = docker compose -f docker-compose.yml -f overlays/traefik-dashboard/docker-compose.yml -f overlays/monitoring/docker-compose.yml -f overlays/profiling/docker-compose.yml -f overlays/glowroot/docker-compose.yml |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For profiling you will probably not want to have both tempo and glowroot. Better to pick one (or none)?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
maybe it's better to have a separate make command for this, like "make profiling"?
Added details on Tempo default overrides for DHIS2's high trace volume, including settings and their purposes.
Updated README to include error message for high trace volume.
Add Distributed Tracing with Grafana Tempo and OpenTelemetry
Summary
Architecture
Changes
New Services:
tempo- Grafana Tempo for trace storage (OTLP receiver on port 4318)tempo-init- Volume permission initializationotel-init- Downloads OpenTelemetry Java Agent JARDHIS2 Instrumentation:
JAVA_TOOL_OPTIONStrace_idandspan_idfor correlationWhat This Enables
Test Plan