-
Notifications
You must be signed in to change notification settings - Fork 831
Verify and document native histograms with custom buckets (NHCB) support #1846
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
zeitlinger
wants to merge
4
commits into
main
Choose a base branch
from
custom-buckets
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,170 @@ | ||
| # Native Histograms with Custom Buckets (NHCB) Example | ||
|
|
||
| This example demonstrates how to use native histograms with custom bucket boundaries (NHCB) in | ||
| Prometheus Java client. It shows three different types of custom bucket configurations and how | ||
| Prometheus converts them to native histograms with schema -53. | ||
|
|
||
| ## What are Native Histograms with Custom Buckets? | ||
|
|
||
| Native Histograms with Custom Buckets (NHCB) is a Prometheus feature that combines the benefits of: | ||
|
|
||
| - **Custom bucket boundaries**: Precisely defined buckets optimized for your specific use case | ||
| - **Native histograms**: Efficient storage and querying capabilities of native histograms | ||
|
|
||
| When you configure Prometheus with `convert_classic_histograms_to_nhcb: true`, it converts classic | ||
| histograms with custom buckets into native histograms using schema -53, preserving the custom | ||
| bucket boundaries. | ||
|
|
||
| ## Example Metrics | ||
|
|
||
| This example application generates three different histogram metrics demonstrating different | ||
| bucket configuration strategies: | ||
|
|
||
| ### 1. API Latency - Arbitrary Custom Boundaries | ||
|
|
||
| ```java | ||
| Histogram apiLatency = Histogram.builder() | ||
| .name("api_request_duration_seconds") | ||
| .classicUpperBounds(0.01, 0.05, 0.1, 0.5, 1.0, 5.0, 10.0) | ||
| .register(); | ||
| ``` | ||
|
|
||
| **Use case**: Optimized for typical API response times in seconds. | ||
|
|
||
| ### 2. Queue Size - Linear Boundaries | ||
|
|
||
| ```java | ||
| Histogram queueSize = Histogram.builder() | ||
| .name("message_queue_size") | ||
| .classicLinearUpperBounds(10, 10, 10) // 10, 20, 30, ..., 100 | ||
| .register(); | ||
| ``` | ||
|
|
||
| **Use case**: Equal-width buckets for monitoring queue depth or other discrete values. | ||
|
|
||
| ### 3. Response Size - Exponential Boundaries | ||
|
|
||
| ```java | ||
| Histogram responseSize = Histogram.builder() | ||
| .name("http_response_size_bytes") | ||
| .classicExponentialUpperBounds(100, 10, 6) // 100, 1k, 10k, 100k, 1M, 10M | ||
| .register(); | ||
| ``` | ||
|
|
||
| **Use case**: Data spanning multiple orders of magnitude (bytes, milliseconds, etc). | ||
|
|
||
| ## Build | ||
|
|
||
| This example is built as part of the `client_java` project: | ||
|
|
||
| ```shell | ||
| ./mvnw package | ||
| ``` | ||
|
|
||
| This creates `./examples/example-custom-buckets/target/example-custom-buckets.jar`. | ||
|
|
||
| ## Run | ||
|
|
||
| With the JAR file present, run: | ||
|
|
||
| ```shell | ||
| cd ./examples/example-custom-buckets/ | ||
| docker-compose up | ||
| ``` | ||
|
|
||
| This starts three Docker containers: | ||
|
|
||
| - **[http://localhost:9400/metrics](http://localhost:9400/metrics)** - Example application | ||
| - **[http://localhost:9090](http://localhost:9090)** - Prometheus server (with NHCB enabled) | ||
| - **[http://localhost:3000](http://localhost:3000)** - Grafana (user: _admin_, password: _admin_) | ||
|
|
||
| You might need to replace `localhost` with `host.docker.internal` on macOS or Windows. | ||
|
|
||
| ## Verify NHCB Conversion | ||
|
|
||
| ### 1. Check Prometheus Configuration | ||
|
|
||
| The Prometheus configuration enables NHCB conversion: | ||
|
|
||
| ```yaml | ||
| scrape_configs: | ||
| - job_name: "custom-buckets-demo" | ||
| scrape_protocols: ["PrometheusProto"] | ||
| convert_classic_histograms_to_nhcb: true | ||
| scrape_classic_histograms: true | ||
| ``` | ||
|
|
||
| ### 2. Verify in Prometheus | ||
|
|
||
| Visit [http://localhost:9090](http://localhost:9090) and run queries: | ||
|
|
||
| ```promql | ||
| # View histogram metadata (should show schema -53 for NHCB) | ||
| prometheus_tsdb_head_series | ||
|
|
||
| # Calculate quantiles from custom buckets | ||
| histogram_quantile(0.95, rate(api_request_duration_seconds[1m])) | ||
|
|
||
| # View raw histogram structure | ||
| api_request_duration_seconds | ||
| ``` | ||
|
|
||
| ### 3. View in Grafana | ||
|
|
||
| The Grafana dashboard at [http://localhost:3000](http://localhost:3000) shows: | ||
|
|
||
| - p95 and p50 latencies for API endpoints (arbitrary custom buckets) | ||
| - Queue size distribution (linear buckets) | ||
| - Response size distribution (exponential buckets) | ||
|
|
||
| ## Key Observations | ||
|
|
||
| 1. **Custom Buckets Preserved**: The custom bucket boundaries you define are preserved when | ||
| converted to NHCB (schema -53). | ||
|
|
||
| 2. **Dual Representation**: By default, histograms maintain both classic and native | ||
| representations, allowing gradual migration. | ||
|
|
||
| 3. **Efficient Storage**: Native histograms provide more efficient storage than classic histograms | ||
| while preserving your custom bucket boundaries. | ||
|
|
||
| 4. **Flexible Bucket Strategies**: You can choose arbitrary, linear, or exponential buckets based | ||
| on your specific monitoring needs. | ||
|
|
||
| ## When to Use Custom Buckets | ||
|
|
||
| Consider using custom buckets (and NHCB) when: | ||
|
|
||
| - **Precise boundaries needed**: You know the expected distribution and want specific bucket edges | ||
| - **Migrating from classic histograms**: You want to preserve existing bucket boundaries | ||
| - **Specific use cases**: Default exponential bucketing doesn't fit your distribution well | ||
| - Temperature ranges (might include negative values) | ||
| - Queue depths (discrete values with linear growth) | ||
| - File sizes (exponential growth but with specific thresholds) | ||
| - API latencies (specific SLA boundaries) | ||
|
|
||
| ## Differences from Standard Native Histograms | ||
|
|
||
| | Feature | Standard Native Histograms | NHCB (Schema -53) | | ||
| | ----------------- | ------------------------------- | --------------------------------- | | ||
| | Bucket boundaries | Exponential (base 2^(2^-scale)) | Custom boundaries | | ||
| | Use case | General-purpose | Specific distributions | | ||
| | Mergeability | Can merge with same schema | Cannot merge different boundaries | | ||
| | Configuration | Schema level (0-8) | Explicit boundary list | | ||
|
|
||
| ## Cleanup | ||
|
|
||
| Stop the containers: | ||
|
|
||
| ```shell | ||
| docker-compose down | ||
| ``` | ||
|
|
||
| ## Further Reading | ||
|
|
||
| <!-- editorconfig-checker-disable --> | ||
| <!-- markdownlint-disable MD013 --> | ||
|
|
||
| - [Prometheus Native Histograms Specification](https://prometheus.io/docs/specs/native_histograms/) | ||
| - [Prometheus Java Client Documentation](https://prometheus.github.io/client_java/) | ||
| - [OpenTelemetry Exponential Histograms](https://opentelemetry.io/docs/specs/otel/metrics/data-model/#exponentialhistogram) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,26 @@ | ||
| version: "3" | ||
| services: | ||
| example-application: | ||
| image: eclipse-temurin:25.0.1_8-jre@sha256:9d1d3068b16f2c4127be238ca06439012ff14a8fdf38f8f62472160f9058464a | ||
| network_mode: host | ||
| volumes: | ||
| - ./target/example-custom-buckets.jar:/example-custom-buckets.jar | ||
| command: | ||
| - /opt/java/openjdk/bin/java | ||
| - -jar | ||
| - /example-custom-buckets.jar | ||
| prometheus: | ||
| image: prom/prometheus:v3.9.1@sha256:1f0f50f06acaceb0f5670d2c8a658a599affe7b0d8e78b898c1035653849a702 | ||
| network_mode: host | ||
| volumes: | ||
| - ./docker-compose/prometheus.yml:/prometheus.yml | ||
| command: | ||
| - --enable-feature=native-histograms | ||
| - --config.file=/prometheus.yml | ||
| grafana: | ||
| image: grafana/grafana:12.3.2@sha256:ba93c9d192e58b23e064c7f501d453426ccf4a85065bf25b705ab1e98602bfb1 | ||
| network_mode: host | ||
| volumes: | ||
| - ./docker-compose/grafana-datasources.yaml:/etc/grafana/provisioning/datasources/grafana-datasources.yaml | ||
| - ./docker-compose/grafana-dashboards.yaml:/etc/grafana/provisioning/dashboards/grafana-dashboards.yaml | ||
| - ./docker-compose/grafana-dashboard-custom-buckets.json:/etc/grafana/grafana-dashboard-custom-buckets.json |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The description of
mise run buildhere says it ensures that all tests pass, but thebuildtask inmise.tomlruns Maven with-DskipTests, so tests are not actually executed. Please either adjust this text to call out that tests are skipped, or point to a task/command (such asmise run test-all) that really runs the full test suite.