From ff07dbb90beb13fd60729b6b684f5812900c1e97 Mon Sep 17 00:00:00 2001 From: Gregor Zeitlinger Date: Wed, 4 Feb 2026 14:07:08 +0100 Subject: [PATCH 1/5] Add support verification and documentation for native histograms with custom buckets (NHCB) Implements issue #1838 by verifying and documenting that client_java fully supports native histograms with custom buckets (NHCB, schema -53). According to the Prometheus specification, NHCB is handled by exposing classic histograms with custom bucket boundaries, which Prometheus servers convert to native histograms (schema -53) when configured with convert_classic_histograms_to_nhcb. Changes: - Add comprehensive test suite (11 tests) verifying custom bucket support for arbitrary, linear, and exponential boundaries - Add documentation section on custom buckets and NHCB to metric-types.md - Create complete working example with Docker Compose (Prometheus + Grafana) - Add verification report documenting findings Test coverage: - Arbitrary custom boundaries - Linear boundaries (equal-width buckets) - Exponential boundaries - Classic-only and dual-mode histograms - Text and protobuf format serialization - Labeled histograms and edge cases All 11 tests pass successfully. Signed-off-by: Gregor Zeitlinger --- CUSTOM_BUCKETS_VERIFICATION.md | 142 ++++++ docs/content/getting-started/metric-types.md | 88 ++++ examples/example-custom-buckets/README.md | 165 ++++++ .../docker-compose.yaml | 26 + .../grafana-dashboard-custom-buckets.json | 349 +++++++++++++ .../docker-compose/grafana-dashboards.yaml | 8 + .../docker-compose/grafana-datasources.yaml | 7 + .../docker-compose/prometheus.yml | 14 + examples/example-custom-buckets/pom.xml | 62 +++ .../metrics/examples/custombuckets/Main.java | 108 ++++ examples/pom.xml | 1 + .../metrics/CustomBucketsHistogramTest.java | 470 ++++++++++++++++++ 12 files changed, 1440 insertions(+) create mode 100644 CUSTOM_BUCKETS_VERIFICATION.md create mode 100644 examples/example-custom-buckets/README.md create mode 100644 examples/example-custom-buckets/docker-compose.yaml create mode 100644 examples/example-custom-buckets/docker-compose/grafana-dashboard-custom-buckets.json create mode 100644 examples/example-custom-buckets/docker-compose/grafana-dashboards.yaml create mode 100644 examples/example-custom-buckets/docker-compose/grafana-datasources.yaml create mode 100644 examples/example-custom-buckets/docker-compose/prometheus.yml create mode 100644 examples/example-custom-buckets/pom.xml create mode 100644 examples/example-custom-buckets/src/main/java/io/prometheus/metrics/examples/custombuckets/Main.java create mode 100644 prometheus-metrics-core/src/test/java/io/prometheus/metrics/core/metrics/CustomBucketsHistogramTest.java diff --git a/CUSTOM_BUCKETS_VERIFICATION.md b/CUSTOM_BUCKETS_VERIFICATION.md new file mode 100644 index 000000000..e566a5383 --- /dev/null +++ b/CUSTOM_BUCKETS_VERIFICATION.md @@ -0,0 +1,142 @@ +# Native Histograms with Custom Buckets - Verification Report + +## Issue #1838: Verify that client_java supports native histograms with custom buckets + +### Summary + +This report documents the verification that the Prometheus Java client library properly supports native histograms with custom bucket configurations (NHCB - Native Histograms with Custom Buckets). + +### Background + +According to the [Prometheus Native Histograms specification](https://prometheus.io/docs/specs/native_histograms/), native histograms with custom buckets (schema -53) are a feature that allows representing classic histograms as native histograms with explicit bucket boundaries. + +**Key findings from the specification:** +- Schema -53 is used for custom bucket boundaries +- There is currently no dedicated protobuf field for custom bucket boundaries +- Custom-bucket histograms are exposed as **classic histograms** with custom boundaries +- Prometheus servers convert these to NHCB upon ingestion when configured with `convert_classic_histograms_to_nhcb` + +### Verification Approach + +The Java client library already supports custom bucket configurations through the `classicUpperBounds()`, `classicLinearUpperBounds()`, and `classicExponentialUpperBounds()` builder methods. These methods allow users to define custom bucket boundaries for histograms. + +Since NHCB is handled by Prometheus servers during ingestion (not by client libraries), our verification focuses on ensuring that: + +1. Histograms with custom bucket boundaries can be created +2. Custom buckets are correctly exposed in both text and protobuf formats +3. Both classic-only and dual (classic+native) histograms work with custom buckets +4. Various custom bucket configurations work correctly + +### Test Implementation + +Created comprehensive test suite: `CustomBucketsHistogramTest.java` + +The test suite includes 11 tests covering: + +#### 1. **Custom Buckets with Arbitrary Boundaries** (`testCustomBucketsWithArbitraryBoundaries`) +- Tests histogram with arbitrary custom bucket boundaries +- Verifies observations are distributed correctly across buckets +- Validates count and sum calculations + +#### 2. **Custom Buckets with Linear Boundaries** (`testCustomBucketsWithLinearBoundaries`) +- Tests histogram with linear custom bucket boundaries (equal-width buckets) +- Use case: Queue size monitoring with fixed intervals + +#### 3. **Custom Buckets with Exponential Boundaries** (`testCustomBucketsWithExponentialBoundaries`) +- Tests histogram with exponential custom bucket boundaries +- Use case: Metrics spanning multiple orders of magnitude (e.g., response sizes) + +#### 4. **Classic-Only Histogram with Custom Buckets** (`testCustomBucketsClassicOnlyHistogram`) +- Verifies custom buckets work when using `.classicOnly()` +- Confirms no native histogram representation is maintained + +#### 5. **Dual-Mode Histogram with Custom Buckets** (`testCustomBucketsDualModeHistogram`) +- Tests the default mode (both classic and native representations) +- Verifies custom classic buckets coexist with native histogram representation +- **This is the most relevant test for NHCB support** + +#### 6. **Text Format Output** (`testCustomBucketsTextFormatOutput`) +- Verifies custom buckets are correctly serialized in Prometheus text format +- Validates bucket labels (le) and counts + +#### 7. **Protobuf Format Output** (`testCustomBucketsProtobufFormatOutput`) +- Verifies custom buckets are correctly serialized in Prometheus protobuf format +- Validates bucket upper bounds and cumulative counts +- Confirms native histogram fields are present (for dual-mode) + +#### 8. **Custom Buckets with Negative Values** (`testCustomBucketsWithNegativeValues`) +- Tests custom buckets with negative boundary values +- Use case: Temperature or other metrics with negative ranges + +#### 9. **Custom Buckets with Labels** (`testCustomBucketsWithLabels`) +- Verifies custom buckets work correctly with labeled histograms +- Tests multiple label combinations + +#### 10. **Boundary Edge Cases** (`testCustomBucketsBoundaryEdgeCases`) +- Tests observations exactly on bucket boundaries +- Verifies buckets are inclusive of their upper bound + +#### 11. **Fine-Grained Custom Buckets** (`testCustomBucketsFineBoundaries`) +- Tests with very precise custom bucket boundaries +- Use case: High-precision measurements + +### Test Results + +All 11 tests pass successfully: + +``` +[INFO] Tests run: 11, Failures: 0, Errors: 0, Skipped: 0 +[INFO] BUILD SUCCESS +``` + +### Key Findings + +1. **Custom bucket support is fully functional**: The Java client library correctly handles histograms with custom bucket boundaries. + +2. **Dual-mode operation**: By default, histograms maintain both classic (with custom buckets) and native representations, which is ideal for NHCB support. + +3. **Correct serialization**: Custom buckets are properly serialized in both: + - Text format (with `le` labels) + - Protobuf format (with bucket upper bounds and cumulative counts) + +4. **Native histogram fields present**: When using dual-mode (default), the protobuf output includes native histogram fields (schema, zero_count, etc.) alongside the classic buckets. + +5. **Flexible bucket configurations**: The library supports: + - Arbitrary custom boundaries + - Linear boundaries (equal-width) + - Exponential boundaries + - Negative values + - Very fine-grained precision + +### Conclusion + +**The Prometheus Java client library (client_java) fully supports native histograms with custom buckets.** + +The library correctly: +- Allows users to define custom bucket boundaries +- Maintains both classic and native histogram representations by default +- Exposes custom buckets in the classic histogram format +- Serializes correctly in both text and protobuf formats + +Prometheus servers can convert these histograms to NHCB (schema -53) upon ingestion when configured with the `convert_classic_histograms_to_nhcb` option. + +### Recommendations + +1. **Documentation**: Consider documenting the NHCB support in the user-facing documentation, explaining that: + - Custom buckets are supported via the existing `classicUpperBounds()` API + - Prometheus servers handle the conversion to NHCB (schema -53) + - The default dual-mode is recommended for NHCB compatibility + +2. **Example**: Consider adding an example demonstrating custom bucket usage for NHCB scenarios. + +3. **Close issue #1838**: This verification confirms that custom bucket support is working correctly. + +### Test File Location + +- `prometheus-metrics-core/src/test/java/io/prometheus/metrics/core/metrics/CustomBucketsHistogramTest.java` + +### References + +- [Prometheus Native Histograms Specification](https://prometheus.io/docs/specs/native_histograms/) +- [GitHub Issue #1838](https://github.com/prometheus/client_java/issues/1838) +- [Prometheus client_model Repository](https://github.com/prometheus/client_model) diff --git a/docs/content/getting-started/metric-types.md b/docs/content/getting-started/metric-types.md index 46d53ece1..844d63a9c 100644 --- a/docs/content/getting-started/metric-types.md +++ b/docs/content/getting-started/metric-types.md @@ -121,6 +121,94 @@ for [Histogram.Builder](/client_java/api/io/prometheus/metrics/core/metrics/Hist for a complete list of options. Some options can be configured at runtime, see [config]({{< relref "../config/config.md" >}}). +### Custom Bucket Boundaries + +The default bucket boundaries are designed for measuring request durations in seconds. For other +use cases, you may want to define custom bucket boundaries. The histogram builder provides three +methods for this: + +**1. Arbitrary Custom Boundaries** + +Use `classicUpperBounds(...)` to specify arbitrary bucket boundaries: + +```java +Histogram responseSize = Histogram.builder() + .name("http_response_size_bytes") + .help("HTTP response size in bytes") + .classicUpperBounds(100, 1000, 10000, 100000, 1000000) // bytes + .register(); +``` + +**2. Linear Boundaries** + +Use `classicLinearUpperBounds(start, width, count)` for equal-width buckets: + +```java +Histogram queueSize = Histogram.builder() + .name("queue_size") + .help("Number of items in queue") + .classicLinearUpperBounds(10, 10, 10) // 10, 20, 30, ..., 100 + .register(); +``` + +**3. Exponential Boundaries** + +Use `classicExponentialUpperBounds(start, factor, count)` for exponential growth: + +```java +Histogram dataSize = Histogram.builder() + .name("data_size_bytes") + .help("Data size in bytes") + .classicExponentialUpperBounds(100, 10, 5) // 100, 1k, 10k, 100k, 1M + .register(); +``` + +### Native Histograms with Custom Buckets (NHCB) + +Prometheus supports a special mode called Native Histograms with Custom Buckets (NHCB) that uses +schema -53. In this mode, custom bucket boundaries from classic histograms are preserved when +converting to native histograms. + +The Java client library automatically supports NHCB: + +1. By default, histograms maintain both classic (with custom buckets) and native representations +2. The classic representation with custom buckets is exposed to Prometheus +3. Prometheus servers can convert these to NHCB upon ingestion when configured with the + `convert_classic_histograms_to_nhcb` scrape option + +Example: + +```java +// This histogram will work seamlessly with NHCB +Histogram apiLatency = Histogram.builder() + .name("api_request_duration_seconds") + .help("API request duration") + .classicUpperBounds(0.01, 0.05, 0.1, 0.5, 1.0, 5.0, 10.0) // custom boundaries + .register(); +``` + +On the Prometheus side, configure the scrape job: + +```yaml +scrape_configs: + - job_name: "my-app" + scrape_protocols: ["PrometheusProto"] + convert_classic_histograms_to_nhcb: true + static_configs: + - targets: ["localhost:9400"] +``` + +{{< hint type=note >}} +NHCB is useful when: + +- You need precise bucket boundaries for your specific use case +- You're migrating from classic histograms and want to preserve bucket boundaries +- Exponential bucketing from standard native histograms isn't a good fit for your distribution + {{< /hint >}} + +See [examples/example-custom-buckets](https://github.com/prometheus/client_java/tree/main/examples/example-custom-buckets) +for a complete example with Prometheus and Grafana. + Histograms and summaries are both used for observing distributions. Therefore, the both implement the `DistributionDataPoint` interface. Using the `DistributionDataPoint` interface directly gives you the option to switch between histograms and summaries later with minimal code changes. diff --git a/examples/example-custom-buckets/README.md b/examples/example-custom-buckets/README.md new file mode 100644 index 000000000..feeab13c9 --- /dev/null +++ b/examples/example-custom-buckets/README.md @@ -0,0 +1,165 @@ +# Native Histograms with Custom Buckets (NHCB) Example + +This example demonstrates how to use native histograms with custom bucket boundaries (NHCB) in +Prometheus Java client. It shows three different types of custom bucket configurations and how +Prometheus converts them to native histograms with schema -53. + +## What are Native Histograms with Custom Buckets? + +Native Histograms with Custom Buckets (NHCB) is a Prometheus feature that combines the benefits of: +- **Custom bucket boundaries**: Precisely defined buckets optimized for your specific use case +- **Native histograms**: Efficient storage and querying capabilities of native histograms + +When you configure Prometheus with `convert_classic_histograms_to_nhcb: true`, it converts classic +histograms with custom buckets into native histograms using schema -53, preserving the custom +bucket boundaries. + +## Example Metrics + +This example application generates three different histogram metrics demonstrating different +bucket configuration strategies: + +### 1. API Latency - Arbitrary Custom Boundaries + +```java +Histogram apiLatency = Histogram.builder() + .name("api_request_duration_seconds") + .classicUpperBounds(0.01, 0.05, 0.1, 0.5, 1.0, 5.0, 10.0) + .register(); +``` + +**Use case**: Optimized for typical API response times in seconds. + +### 2. Queue Size - Linear Boundaries + +```java +Histogram queueSize = Histogram.builder() + .name("message_queue_size") + .classicLinearUpperBounds(10, 10, 10) // 10, 20, 30, ..., 100 + .register(); +``` + +**Use case**: Equal-width buckets for monitoring queue depth or other discrete values. + +### 3. Response Size - Exponential Boundaries + +```java +Histogram responseSize = Histogram.builder() + .name("http_response_size_bytes") + .classicExponentialUpperBounds(100, 10, 6) // 100, 1k, 10k, 100k, 1M, 10M + .register(); +``` + +**Use case**: Data spanning multiple orders of magnitude (bytes, milliseconds, etc). + +## Build + +This example is built as part of the `client_java` project: + +```shell +./mvnw package +``` + +This creates `./examples/example-custom-buckets/target/example-custom-buckets.jar`. + +## Run + +With the JAR file present, run: + +```shell +cd ./examples/example-custom-buckets/ +docker-compose up +``` + +This starts three Docker containers: + +- **[http://localhost:9400/metrics](http://localhost:9400/metrics)** - Example application +- **[http://localhost:9090](http://localhost:9090)** - Prometheus server (with NHCB enabled) +- **[http://localhost:3000](http://localhost:3000)** - Grafana (user: _admin_, password: _admin_) + +You might need to replace `localhost` with `host.docker.internal` on macOS or Windows. + +## Verify NHCB Conversion + +### 1. Check Prometheus Configuration + +The Prometheus configuration enables NHCB conversion: + +```yaml +scrape_configs: + - job_name: "custom-buckets-demo" + scrape_protocols: ['PrometheusProto'] + convert_classic_histograms_to_nhcb: true + scrape_classic_histograms: true +``` + +### 2. Verify in Prometheus + +Visit [http://localhost:9090](http://localhost:9090) and run queries: + +```promql +# View histogram metadata (should show schema -53 for NHCB) +prometheus_tsdb_head_series + +# Calculate quantiles from custom buckets +histogram_quantile(0.95, rate(api_request_duration_seconds[1m])) + +# View raw histogram structure +api_request_duration_seconds +``` + +### 3. View in Grafana + +The Grafana dashboard at [http://localhost:3000](http://localhost:3000) shows: +- p95 and p50 latencies for API endpoints (arbitrary custom buckets) +- Queue size distribution (linear buckets) +- Response size distribution (exponential buckets) + +## Key Observations + +1. **Custom Buckets Preserved**: The custom bucket boundaries you define are preserved when + converted to NHCB (schema -53). + +2. **Dual Representation**: By default, histograms maintain both classic and native + representations, allowing gradual migration. + +3. **Efficient Storage**: Native histograms provide more efficient storage than classic histograms + while preserving your custom bucket boundaries. + +4. **Flexible Bucket Strategies**: You can choose arbitrary, linear, or exponential buckets based + on your specific monitoring needs. + +## When to Use Custom Buckets + +Consider using custom buckets (and NHCB) when: + +- **Precise boundaries needed**: You know the expected distribution and want specific bucket edges +- **Migrating from classic histograms**: You want to preserve existing bucket boundaries +- **Specific use cases**: Default exponential bucketing doesn't fit your distribution well + - Temperature ranges (might include negative values) + - Queue depths (discrete values with linear growth) + - File sizes (exponential growth but with specific thresholds) + - API latencies (specific SLA boundaries) + +## Differences from Standard Native Histograms + +| Feature | Standard Native Histograms | NHCB (Schema -53) | +|---------|---------------------------|-------------------| +| Bucket boundaries | Exponential (base 2^(2^-scale)) | Custom boundaries | +| Use case | General-purpose | Specific distributions | +| Mergeability | Can merge histograms with same schema | Cannot merge with different boundaries | +| Configuration | Schema level (0-8) | Explicit boundary list | + +## Cleanup + +Stop the containers: + +```shell +docker-compose down +``` + +## Further Reading + +- [Prometheus Native Histograms Specification](https://prometheus.io/docs/specs/native_histograms/) +- [Prometheus Java Client Documentation](https://prometheus.github.io/client_java/) +- [OpenTelemetry Exponential Histograms](https://opentelemetry.io/docs/specs/otel/metrics/data-model/#exponentialhistogram) diff --git a/examples/example-custom-buckets/docker-compose.yaml b/examples/example-custom-buckets/docker-compose.yaml new file mode 100644 index 000000000..7579faa3f --- /dev/null +++ b/examples/example-custom-buckets/docker-compose.yaml @@ -0,0 +1,26 @@ +version: "3" +services: + example-application: + image: eclipse-temurin:25.0.1_8-jre@sha256:9d1d3068b16f2c4127be238ca06439012ff14a8fdf38f8f62472160f9058464a + network_mode: host + volumes: + - ./target/example-custom-buckets.jar:/example-custom-buckets.jar + command: + - /opt/java/openjdk/bin/java + - -jar + - /example-custom-buckets.jar + prometheus: + image: prom/prometheus:v3.9.1@sha256:1f0f50f06acaceb0f5670d2c8a658a599affe7b0d8e78b898c1035653849a702 + network_mode: host + volumes: + - ./docker-compose/prometheus.yml:/prometheus.yml + command: + - --enable-feature=native-histograms + - --config.file=/prometheus.yml + grafana: + image: grafana/grafana:12.3.2@sha256:ba93c9d192e58b23e064c7f501d453426ccf4a85065bf25b705ab1e98602bfb1 + network_mode: host + volumes: + - ./docker-compose/grafana-datasources.yaml:/etc/grafana/provisioning/datasources/grafana-datasources.yaml + - ./docker-compose/grafana-dashboards.yaml:/etc/grafana/provisioning/dashboards/grafana-dashboards.yaml + - ./docker-compose/grafana-dashboard-custom-buckets.json:/etc/grafana/grafana-dashboard-custom-buckets.json diff --git a/examples/example-custom-buckets/docker-compose/grafana-dashboard-custom-buckets.json b/examples/example-custom-buckets/docker-compose/grafana-dashboard-custom-buckets.json new file mode 100644 index 000000000..11ae25775 --- /dev/null +++ b/examples/example-custom-buckets/docker-compose/grafana-dashboard-custom-buckets.json @@ -0,0 +1,349 @@ +{ + "annotations": { + "list": [] + }, + "editable": true, + "fiscalYearStartMonth": 0, + "graphTooltip": 0, + "id": null, + "links": [], + "panels": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "description": "API request duration with custom bucket boundaries (0.01, 0.05, 0.1, 0.5, 1.0, 5.0, 10.0 seconds). Shows how custom buckets are preserved in NHCB (schema -53).", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "tooltip": false, + "viz": false, + "legend": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "s" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 24, + "x": 0, + "y": 0 + }, + "id": 1, + "options": { + "legend": { + "calcs": ["mean", "max"], + "displayMode": "table", + "placement": "right", + "showLegend": true + }, + "tooltip": { + "mode": "single", + "sort": "none" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.95, rate(api_request_duration_seconds[1m]))", + "instant": false, + "legendFormat": "{{endpoint}} {{status}} (p95)", + "range": true, + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.5, rate(api_request_duration_seconds[1m]))", + "hide": false, + "instant": false, + "legendFormat": "{{endpoint}} {{status}} (p50)", + "range": true, + "refId": "B" + } + ], + "title": "API Latency - Custom Buckets (Arbitrary Boundaries)", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "description": "Queue size with linear bucket boundaries (10, 20, 30, ..., 100). Demonstrates equal-width buckets for discrete values.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "tooltip": false, + "viz": false, + "legend": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 8 + }, + "id": 2, + "options": { + "legend": { + "calcs": ["mean", "max"], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "single", + "sort": "none" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.95, rate(message_queue_size[1m]))", + "instant": false, + "legendFormat": "{{queue_name}} (p95)", + "range": true, + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.5, rate(message_queue_size[1m]))", + "hide": false, + "instant": false, + "legendFormat": "{{queue_name}} (p50)", + "range": true, + "refId": "B" + } + ], + "title": "Queue Size - Linear Buckets", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "description": "HTTP response size with exponential bucket boundaries (100, 1k, 10k, 100k, 1M, 10M bytes). Shows exponential growth for data spanning multiple orders of magnitude.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "tooltip": false, + "viz": false, + "legend": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "bytes" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 8 + }, + "id": 3, + "options": { + "legend": { + "calcs": ["mean", "max"], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "single", + "sort": "none" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.95, rate(http_response_size_bytes[1m]))", + "instant": false, + "legendFormat": "{{endpoint}} (p95)", + "range": true, + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.5, rate(http_response_size_bytes[1m]))", + "hide": false, + "instant": false, + "legendFormat": "{{endpoint}} (p50)", + "range": true, + "refId": "B" + } + ], + "title": "Response Size - Exponential Buckets", + "type": "timeseries" + } + ], + "refresh": "5s", + "schemaVersion": 39, + "tags": ["custom-buckets", "nhcb", "native-histogram"], + "templating": { + "list": [] + }, + "time": { + "from": "now-5m", + "to": "now" + }, + "timepicker": {}, + "timezone": "browser", + "title": "Native Histograms with Custom Buckets (NHCB)", + "uid": "custom-buckets-nhcb", + "version": 1, + "weekStart": "" +} diff --git a/examples/example-custom-buckets/docker-compose/grafana-dashboards.yaml b/examples/example-custom-buckets/docker-compose/grafana-dashboards.yaml new file mode 100644 index 000000000..3225b88ae --- /dev/null +++ b/examples/example-custom-buckets/docker-compose/grafana-dashboards.yaml @@ -0,0 +1,8 @@ +apiVersion: 1 + +providers: + - name: "Custom Buckets (NHCB) Example" + type: file + options: + path: /etc/grafana/grafana-dashboard-custom-buckets.json + foldersFromFilesStructure: false diff --git a/examples/example-custom-buckets/docker-compose/grafana-datasources.yaml b/examples/example-custom-buckets/docker-compose/grafana-datasources.yaml new file mode 100644 index 000000000..d442d28d2 --- /dev/null +++ b/examples/example-custom-buckets/docker-compose/grafana-datasources.yaml @@ -0,0 +1,7 @@ +apiVersion: 1 + +datasources: + - name: Prometheus + type: prometheus + uid: prometheus + url: http://localhost:9090 diff --git a/examples/example-custom-buckets/docker-compose/prometheus.yml b/examples/example-custom-buckets/docker-compose/prometheus.yml new file mode 100644 index 000000000..d056e4096 --- /dev/null +++ b/examples/example-custom-buckets/docker-compose/prometheus.yml @@ -0,0 +1,14 @@ +--- +global: + scrape_interval: 5s # very short interval for demo purposes + +scrape_configs: + - job_name: "custom-buckets-demo" + # Use protobuf format to receive native histogram data + scrape_protocols: ['PrometheusProto'] + # Convert classic histograms with custom buckets to NHCB (schema -53) + convert_classic_histograms_to_nhcb: true + # Also scrape classic histograms for comparison + scrape_classic_histograms: true + static_configs: + - targets: ["localhost:9400"] diff --git a/examples/example-custom-buckets/pom.xml b/examples/example-custom-buckets/pom.xml new file mode 100644 index 000000000..b7e104e5a --- /dev/null +++ b/examples/example-custom-buckets/pom.xml @@ -0,0 +1,62 @@ + + + 4.0.0 + + + io.prometheus + examples + 1.5.0-SNAPSHOT + + + example-custom-buckets + + Example - Custom Buckets + + End-to-End example of Native Histograms with Custom Buckets (NHCB): Java app -> Prometheus -> Grafana + + + + + io.prometheus + prometheus-metrics-core + ${project.version} + + + io.prometheus + prometheus-metrics-instrumentation-jvm + ${project.version} + + + io.prometheus + prometheus-metrics-exporter-httpserver + ${project.version} + + + + + ${project.artifactId} + + + org.apache.maven.plugins + maven-shade-plugin + + + package + + shade + + + + + io.prometheus.metrics.examples.custombuckets.Main + + + + + + + + + diff --git a/examples/example-custom-buckets/src/main/java/io/prometheus/metrics/examples/custombuckets/Main.java b/examples/example-custom-buckets/src/main/java/io/prometheus/metrics/examples/custombuckets/Main.java new file mode 100644 index 000000000..3d286fdf0 --- /dev/null +++ b/examples/example-custom-buckets/src/main/java/io/prometheus/metrics/examples/custombuckets/Main.java @@ -0,0 +1,108 @@ +package io.prometheus.metrics.examples.custombuckets; + +import io.prometheus.metrics.core.metrics.Histogram; +import io.prometheus.metrics.exporter.httpserver.HTTPServer; +import io.prometheus.metrics.instrumentation.jvm.JvmMetrics; +import io.prometheus.metrics.model.snapshots.Unit; +import java.io.IOException; +import java.util.Random; + +/** + * Example demonstrating native histograms with custom buckets (NHCB). + * + *

This example shows three different types of custom bucket configurations: + * + *

+ * + *

These histograms maintain both classic (with custom buckets) and native representations. When + * Prometheus is configured with {@code convert_classic_histograms_to_nhcb: true}, the custom bucket + * boundaries are preserved in the native histogram format (schema -53). + */ +public class Main { + + public static void main(String[] args) throws IOException, InterruptedException { + + JvmMetrics.builder().register(); + + // Example 1: API latency with arbitrary custom boundaries + // Optimized for typical API response times in seconds + Histogram apiLatency = + Histogram.builder() + .name("api_request_duration_seconds") + .help("API request duration with custom buckets") + .unit(Unit.SECONDS) + .classicUpperBounds(0.01, 0.05, 0.1, 0.5, 1.0, 5.0, 10.0) + .labelNames("endpoint", "status") + .register(); + + // Example 2: Queue size with linear boundaries + // Equal-width buckets for monitoring queue depth + Histogram queueSize = + Histogram.builder() + .name("message_queue_size") + .help("Number of messages in queue with linear buckets") + .classicLinearUpperBounds(10, 10, 10) // 10, 20, 30, ..., 100 + .labelNames("queue_name") + .register(); + + // Example 3: Response size with exponential boundaries + // Exponential growth for data spanning multiple orders of magnitude + Histogram responseSize = + Histogram.builder() + .name("http_response_size_bytes") + .help("HTTP response size in bytes with exponential buckets") + .classicExponentialUpperBounds(100, 10, 6) // 100, 1k, 10k, 100k, 1M, 10M + .labelNames("endpoint") + .register(); + + HTTPServer server = HTTPServer.builder().port(9400).buildAndStart(); + + System.out.println( + "HTTPServer listening on port http://localhost:" + server.getPort() + "/metrics"); + System.out.println("\nGenerating metrics with custom bucket configurations:"); + System.out.println("1. API latency: custom boundaries optimized for response times"); + System.out.println("2. Queue size: linear boundaries (10, 20, 30, ..., 100)"); + System.out.println("3. Response size: exponential boundaries (100, 1k, 10k, ..., 10M)"); + System.out.println("\nPrometheus will convert these to NHCB (schema -53) when configured.\n"); + + Random random = new Random(0); + + while (true) { + // Simulate API latency observations + // Fast endpoint: mostly < 100ms, occasionally slow + double fastLatency = Math.abs(random.nextGaussian() * 0.03 + 0.05); + String status = random.nextInt(100) < 95 ? "200" : "500"; + apiLatency.labelValues("/api/fast", status).observe(fastLatency); + + // Slow endpoint: typically 1-3 seconds + double slowLatency = Math.abs(random.nextGaussian() * 0.5 + 2.0); + apiLatency.labelValues("/api/slow", status).observe(slowLatency); + + // Simulate queue size observations + // Queue oscillates between 20-80 items + int queueDepth = 50 + (int) (random.nextGaussian() * 15); + queueDepth = Math.max(0, Math.min(100, queueDepth)); + queueSize.labelValues("default").observe(queueDepth); + + // Priority queue: usually smaller + int priorityQueueDepth = 10 + (int) (random.nextGaussian() * 5); + priorityQueueDepth = Math.max(0, Math.min(50, priorityQueueDepth)); + queueSize.labelValues("priority").observe(priorityQueueDepth); + + // Simulate response size observations + // Small responses: mostly < 10KB + double smallResponse = Math.abs(random.nextGaussian() * 2000 + 5000); + responseSize.labelValues("/api/summary").observe(smallResponse); + + // Large responses: can be up to several MB + double largeResponse = Math.abs(random.nextGaussian() * 200000 + 500000); + responseSize.labelValues("/api/download").observe(largeResponse); + + Thread.sleep(1000); + } + } +} diff --git a/examples/pom.xml b/examples/pom.xml index 5b93c068f..d0c364067 100644 --- a/examples/pom.xml +++ b/examples/pom.xml @@ -30,6 +30,7 @@ example-exporter-opentelemetry example-simpleclient-bridge example-native-histogram + example-custom-buckets example-prometheus-properties diff --git a/prometheus-metrics-core/src/test/java/io/prometheus/metrics/core/metrics/CustomBucketsHistogramTest.java b/prometheus-metrics-core/src/test/java/io/prometheus/metrics/core/metrics/CustomBucketsHistogramTest.java new file mode 100644 index 000000000..347f775cf --- /dev/null +++ b/prometheus-metrics-core/src/test/java/io/prometheus/metrics/core/metrics/CustomBucketsHistogramTest.java @@ -0,0 +1,470 @@ +package io.prometheus.metrics.core.metrics; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.data.Offset.offset; + +import io.prometheus.metrics.config.EscapingScheme; +import io.prometheus.metrics.expositionformats.OpenMetricsTextFormatWriter; +import io.prometheus.metrics.expositionformats.generated.com_google_protobuf_4_33_5.Metrics; +import io.prometheus.metrics.expositionformats.internal.PrometheusProtobufWriterImpl; +import io.prometheus.metrics.model.snapshots.ClassicHistogramBucket; +import io.prometheus.metrics.model.snapshots.ClassicHistogramBuckets; +import io.prometheus.metrics.model.snapshots.HistogramSnapshot; +import io.prometheus.metrics.model.snapshots.Labels; +import io.prometheus.metrics.model.snapshots.MetricSnapshots; +import java.io.ByteArrayOutputStream; +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.util.List; +import java.util.stream.Collectors; +import org.junit.jupiter.api.Test; + +/** + * Comprehensive tests to verify that client_java supports native histograms with custom buckets + * (NHCB). + * + *

According to the Prometheus specification + * (https://prometheus.io/docs/specs/native_histograms/), native histograms with custom buckets + * (schema -53) are exposed as classic histograms with custom bucket boundaries. Prometheus servers + * can then convert these to NHCB upon ingestion when configured with + * convert_classic_histograms_to_nhcb. + * + *

These tests verify that: + * + *

+ * + *

See issue #1838 for more context. + */ +class CustomBucketsHistogramTest { + + @Test + void testCustomBucketsWithArbitraryBoundaries() { + // Create a histogram with arbitrary custom bucket boundaries + Histogram histogram = + Histogram.builder() + .name("http_request_duration_seconds") + .help("HTTP request duration with custom buckets") + .classicUpperBounds(0.01, 0.05, 0.1, 0.5, 1.0, 5.0, 10.0) + .build(); + + // Observe some values + histogram.observe(0.008); + histogram.observe(0.045); + histogram.observe(0.3); + histogram.observe(2.5); + histogram.observe(7.8); + + HistogramSnapshot snapshot = histogram.collect(); + HistogramSnapshot.HistogramDataPointSnapshot data = snapshot.getDataPoints().get(0); + + // Verify custom bucket boundaries are set correctly + List upperBounds = + data.getClassicBuckets().stream() + .map(ClassicHistogramBucket::getUpperBound) + .collect(Collectors.toList()); + + assertThat(upperBounds) + .containsExactly(0.01, 0.05, 0.1, 0.5, 1.0, 5.0, 10.0, Double.POSITIVE_INFINITY); + + // Verify observations are distributed correctly across buckets + // Note: counts are non-cumulative (count for that specific bucket only) + ClassicHistogramBuckets buckets = data.getClassicBuckets(); + assertThat(buckets.getCount(0)).isEqualTo(1); // <= 0.01: (0.008) + assertThat(buckets.getCount(1)).isEqualTo(1); // (0.01, 0.05]: (0.045) + assertThat(buckets.getCount(2)).isEqualTo(0); // (0.05, 0.1]: none + assertThat(buckets.getCount(3)).isEqualTo(1); // (0.1, 0.5]: (0.3) + assertThat(buckets.getCount(4)).isEqualTo(0); // (0.5, 1.0]: none + assertThat(buckets.getCount(5)).isEqualTo(1); // (1.0, 5.0]: (2.5) + assertThat(buckets.getCount(6)).isEqualTo(1); // (5.0, 10.0]: (7.8) + assertThat(buckets.getCount(7)).isEqualTo(0); // (10.0, +Inf]: none + + // Verify count and sum + assertThat(data.getCount()).isEqualTo(5); + assertThat(data.getSum()).isCloseTo(10.653, offset(0.01)); + } + + @Test + void testCustomBucketsWithLinearBoundaries() { + // Create a histogram with linear custom bucket boundaries + // This represents a use case where equal-width buckets are needed + Histogram histogram = + Histogram.builder() + .name("queue_size") + .help("Queue size with linear buckets") + .classicLinearUpperBounds(10.0, 10.0, 10) // start=10, width=10, count=10 + .build(); + + // Observe some values + for (int i = 5; i <= 95; i += 10) { + histogram.observe(i); + } + + HistogramSnapshot snapshot = histogram.collect(); + HistogramSnapshot.HistogramDataPointSnapshot data = snapshot.getDataPoints().get(0); + + // Verify linear bucket boundaries + List upperBounds = + data.getClassicBuckets().stream() + .map(ClassicHistogramBucket::getUpperBound) + .collect(Collectors.toList()); + + assertThat(upperBounds) + .containsExactly( + 10.0, 20.0, 30.0, 40.0, 50.0, 60.0, 70.0, 80.0, 90.0, 100.0, Double.POSITIVE_INFINITY); + + // Verify observations + assertThat(data.getCount()).isEqualTo(10); + } + + @Test + void testCustomBucketsWithExponentialBoundaries() { + // Create a histogram with exponential custom bucket boundaries + // This is useful for metrics that span multiple orders of magnitude + Histogram histogram = + Histogram.builder() + .name("response_size_bytes") + .help("Response size with exponential buckets") + .classicExponentialUpperBounds(100.0, 10.0, 5) // start=100, factor=10, count=5 + .build(); + + // Observe some values across different magnitudes + histogram.observe(50); + histogram.observe(500); + histogram.observe(5000); + histogram.observe(50000); + histogram.observe(500000); + + HistogramSnapshot snapshot = histogram.collect(); + HistogramSnapshot.HistogramDataPointSnapshot data = snapshot.getDataPoints().get(0); + + // Verify exponential bucket boundaries + List upperBounds = + data.getClassicBuckets().stream() + .map(ClassicHistogramBucket::getUpperBound) + .collect(Collectors.toList()); + + assertThat(upperBounds) + .containsExactly(100.0, 1000.0, 10000.0, 100000.0, 1000000.0, Double.POSITIVE_INFINITY); + + // Verify observations + assertThat(data.getCount()).isEqualTo(5); + } + + @Test + void testCustomBucketsClassicOnlyHistogram() { + // Verify that custom buckets work with classic-only histograms + Histogram histogram = + Histogram.builder() + .name("test_classic_only") + .help("Classic-only histogram with custom buckets") + .classicOnly() + .classicUpperBounds(1.0, 5.0, 10.0) + .build(); + + histogram.observe(2.0); + histogram.observe(7.0); + + HistogramSnapshot snapshot = histogram.collect(); + HistogramSnapshot.HistogramDataPointSnapshot data = snapshot.getDataPoints().get(0); + + // Verify it's a classic-only histogram + assertThat(data.getNativeSchema()).isEqualTo(HistogramSnapshot.CLASSIC_HISTOGRAM); + + // Verify custom buckets + List upperBounds = + data.getClassicBuckets().stream() + .map(ClassicHistogramBucket::getUpperBound) + .collect(Collectors.toList()); + + assertThat(upperBounds).containsExactly(1.0, 5.0, 10.0, Double.POSITIVE_INFINITY); + } + + @Test + void testCustomBucketsDualModeHistogram() { + // Verify that custom buckets work with dual-mode (classic+native) histograms + // This is the default mode and most relevant for NHCB support + Histogram histogram = + Histogram.builder() + .name("test_dual_mode") + .help("Dual-mode histogram with custom buckets") + .classicUpperBounds(0.1, 1.0, 10.0) + .build(); + + histogram.observe(0.5); + histogram.observe(5.0); + + HistogramSnapshot snapshot = histogram.collect(); + HistogramSnapshot.HistogramDataPointSnapshot data = snapshot.getDataPoints().get(0); + + // Verify it has both classic and native representations + assertThat(data.getClassicBuckets().size()).isGreaterThan(0); + assertThat(data.getNativeSchema()).isNotEqualTo(HistogramSnapshot.CLASSIC_HISTOGRAM); + + // Verify custom classic buckets + List upperBounds = + data.getClassicBuckets().stream() + .map(ClassicHistogramBucket::getUpperBound) + .collect(Collectors.toList()); + + assertThat(upperBounds).containsExactly(0.1, 1.0, 10.0, Double.POSITIVE_INFINITY); + + // Verify native histogram is also populated + long nativeTotalCount = + data.getNativeBucketsForPositiveValues().stream() + .mapToLong(bucket -> bucket.getCount()) + .sum(); + assertThat(nativeTotalCount).isEqualTo(2); + } + + @Test + void testCustomBucketsTextFormatOutput() throws IOException { + // Verify that custom buckets are correctly serialized in text format + Histogram histogram = + Histogram.builder() + .name("test_custom_buckets") + .help("Test histogram with custom buckets") + .classicUpperBounds(0.5, 1.0, 2.0) + .build(); + + histogram.observe(0.3); + histogram.observe(0.7); + histogram.observe(1.5); + + ByteArrayOutputStream out = new ByteArrayOutputStream(); + OpenMetricsTextFormatWriter writer = new OpenMetricsTextFormatWriter(false, true); + writer.write(out, MetricSnapshots.of(histogram.collect()), EscapingScheme.ALLOW_UTF8); + + String output = out.toString(StandardCharsets.UTF_8); + + // Verify the output contains the custom bucket boundaries + assertThat(output).contains("le=\"0.5\""); + assertThat(output).contains("le=\"1.0\""); + assertThat(output).contains("le=\"2.0\""); + assertThat(output).contains("le=\"+Inf\""); + + // Verify bucket counts + assertThat(output).containsPattern("le=\"0.5\".*1"); // 1 observation <= 0.5 + assertThat(output).containsPattern("le=\"1.0\".*2"); // 2 observations <= 1.0 + assertThat(output).containsPattern("le=\"2.0\".*3"); // 3 observations <= 2.0 + assertThat(output).containsPattern("le=\"\\+Inf\".*3"); // 3 observations total + } + + @Test + void testCustomBucketsProtobufFormatOutput() { + // Verify that custom buckets are correctly serialized in Prometheus protobuf format + Histogram histogram = + Histogram.builder() + .name("test_custom_buckets_protobuf") + .help("Test histogram with custom buckets for protobuf") + .classicUpperBounds(1.0, 5.0, 10.0) + .build(); + + histogram.observe(0.5); + histogram.observe(3.0); + histogram.observe(7.0); + + HistogramSnapshot snapshot = histogram.collect(); + Metrics.MetricFamily metricFamily = + new PrometheusProtobufWriterImpl().convert(snapshot, EscapingScheme.ALLOW_UTF8); + + assertThat(metricFamily).isNotNull(); + assertThat(metricFamily.getName()).isEqualTo("test_custom_buckets_protobuf"); + assertThat(metricFamily.getType()).isEqualTo(Metrics.MetricType.HISTOGRAM); + + Metrics.Histogram protoHistogram = metricFamily.getMetric(0).getHistogram(); + + // Verify classic buckets in protobuf + assertThat(protoHistogram.getBucketCount()).isEqualTo(4); // 3 custom + +Inf + + // Verify bucket upper bounds + assertThat(protoHistogram.getBucket(0).getUpperBound()).isEqualTo(1.0); + assertThat(protoHistogram.getBucket(1).getUpperBound()).isEqualTo(5.0); + assertThat(protoHistogram.getBucket(2).getUpperBound()).isEqualTo(10.0); + assertThat(protoHistogram.getBucket(3).getUpperBound()).isEqualTo(Double.POSITIVE_INFINITY); + + // Verify bucket counts (cumulative) + assertThat(protoHistogram.getBucket(0).getCumulativeCount()).isEqualTo(1); // <= 1.0 + assertThat(protoHistogram.getBucket(1).getCumulativeCount()).isEqualTo(2); // <= 5.0 + assertThat(protoHistogram.getBucket(2).getCumulativeCount()).isEqualTo(3); // <= 10.0 + assertThat(protoHistogram.getBucket(3).getCumulativeCount()).isEqualTo(3); // +Inf + + // Verify native histogram fields are also present (for dual-mode) + assertThat(protoHistogram.hasSchema()).isTrue(); + assertThat(protoHistogram.getSchema()).isNotEqualTo(HistogramSnapshot.CLASSIC_HISTOGRAM); + } + + @Test + void testCustomBucketsWithNegativeValues() { + // Verify that custom buckets work correctly with negative values + Histogram histogram = + Histogram.builder() + .name("temperature_celsius") + .help("Temperature readings with custom buckets") + .classicUpperBounds(-20.0, -10.0, 0.0, 10.0, 20.0, 30.0) + .build(); + + histogram.observe(-15.0); + histogram.observe(-5.0); + histogram.observe(5.0); + histogram.observe(15.0); + histogram.observe(25.0); + + HistogramSnapshot snapshot = histogram.collect(); + HistogramSnapshot.HistogramDataPointSnapshot data = snapshot.getDataPoints().get(0); + + // Verify bucket boundaries + List upperBounds = + data.getClassicBuckets().stream() + .map(ClassicHistogramBucket::getUpperBound) + .collect(Collectors.toList()); + + assertThat(upperBounds) + .containsExactly(-20.0, -10.0, 0.0, 10.0, 20.0, 30.0, Double.POSITIVE_INFINITY); + + // Verify observations are distributed correctly + // Note: counts are non-cumulative + ClassicHistogramBuckets buckets = data.getClassicBuckets(); + assertThat(buckets.getCount(0)).isEqualTo(0); // <= -20: none + assertThat(buckets.getCount(1)).isEqualTo(1); // (-20, -10]: (-15.0) + assertThat(buckets.getCount(2)).isEqualTo(1); // (-10, 0]: (-5.0) + assertThat(buckets.getCount(3)).isEqualTo(1); // (0, 10]: (5.0) + assertThat(buckets.getCount(4)).isEqualTo(1); // (10, 20]: (15.0) + assertThat(buckets.getCount(5)).isEqualTo(1); // (20, 30]: (25.0) + + assertThat(data.getCount()).isEqualTo(5); + } + + @Test + void testCustomBucketsWithLabels() { + // Verify that custom buckets work correctly with labeled histograms + Histogram histogram = + Histogram.builder() + .name("api_request_duration_seconds") + .help("API request duration with custom buckets") + .classicUpperBounds(0.01, 0.1, 1.0, 10.0) + .labelNames("method", "endpoint") + .build(); + + histogram.labelValues("GET", "/users").observe(0.05); + histogram.labelValues("GET", "/users").observe(0.5); + histogram.labelValues("POST", "/users").observe(2.0); + + HistogramSnapshot snapshot = histogram.collect(); + + // Verify we have 2 data points (one for each unique label combination) + assertThat(snapshot.getDataPoints()).hasSize(2); + + // Verify both data points have the correct custom buckets + for (HistogramSnapshot.HistogramDataPointSnapshot data : snapshot.getDataPoints()) { + List upperBounds = + data.getClassicBuckets().stream() + .map(ClassicHistogramBucket::getUpperBound) + .collect(Collectors.toList()); + + assertThat(upperBounds).containsExactly(0.01, 0.1, 1.0, 10.0, Double.POSITIVE_INFINITY); + } + + // Verify GET /users data point + HistogramSnapshot.HistogramDataPointSnapshot getData = + getData(histogram, "method", "GET", "endpoint", "/users"); + + assertThat(getData.getCount()).isEqualTo(2); + + // Verify POST /users data point + HistogramSnapshot.HistogramDataPointSnapshot postData = + getData(histogram, "method", "POST", "endpoint", "/users"); + + assertThat(postData.getCount()).isEqualTo(1); + } + + private HistogramSnapshot.HistogramDataPointSnapshot getData( + Histogram histogram, String... labels) { + return histogram.collect().getDataPoints().stream() + .filter(d -> d.getLabels().equals(Labels.of(labels))) + .findAny() + .orElseThrow(() -> new RuntimeException("histogram with labels not found")); + } + + @Test + void testCustomBucketsBoundaryEdgeCases() { + // Test edge cases: observations exactly on bucket boundaries + Histogram histogram = + Histogram.builder() + .name("test_boundaries") + .help("Test bucket boundary edge cases") + .classicUpperBounds(1.0, 5.0, 10.0) + .build(); + + // Observe values exactly on the boundaries + histogram.observe(1.0); + histogram.observe(5.0); + histogram.observe(10.0); + + HistogramSnapshot snapshot = histogram.collect(); + HistogramSnapshot.HistogramDataPointSnapshot data = snapshot.getDataPoints().get(0); + + // Values on boundaries should be included in their respective buckets + // Buckets are inclusive of upper bound + // Note: counts are non-cumulative + ClassicHistogramBuckets buckets = data.getClassicBuckets(); + assertThat(buckets.getCount(0)).isEqualTo(1); // <= 1.0: (1.0) + assertThat(buckets.getCount(1)).isEqualTo(1); // (1.0, 5.0]: (5.0) + assertThat(buckets.getCount(2)).isEqualTo(1); // (5.0, 10.0]: (10.0) + + assertThat(data.getCount()).isEqualTo(3); + } + + @Test + void testCustomBucketsFineBoundaries() { + // Test with very fine-grained custom bucket boundaries + // This simulates a use case where precise bucket boundaries are needed + Histogram histogram = + Histogram.builder() + .name("precise_measurement") + .help("Histogram with fine-grained custom buckets") + .classicUpperBounds(0.001, 0.002, 0.003, 0.004, 0.005, 0.006, 0.007, 0.008, 0.009, 0.01) + .build(); + + histogram.observe(0.0015); + histogram.observe(0.0045); + histogram.observe(0.0075); + + HistogramSnapshot snapshot = histogram.collect(); + HistogramSnapshot.HistogramDataPointSnapshot data = snapshot.getDataPoints().get(0); + + // Verify fine-grained buckets are set correctly + List upperBounds = + data.getClassicBuckets().stream() + .map(ClassicHistogramBucket::getUpperBound) + .collect(Collectors.toList()); + + assertThat(upperBounds) + .containsExactly( + 0.001, + 0.002, + 0.003, + 0.004, + 0.005, + 0.006, + 0.007, + 0.008, + 0.009, + 0.01, + Double.POSITIVE_INFINITY); + + // Verify observations are in correct buckets + // Note: counts are non-cumulative + ClassicHistogramBuckets buckets = data.getClassicBuckets(); + assertThat(buckets.getCount(0)).isEqualTo(0); // <= 0.001: none + assertThat(buckets.getCount(1)).isEqualTo(1); // (0.001, 0.002]: (0.0015) + assertThat(buckets.getCount(4)).isEqualTo(1); // (0.004, 0.005]: (0.0045) + assertThat(buckets.getCount(7)).isEqualTo(1); // (0.007, 0.008]: (0.0075) + + assertThat(data.getCount()).isEqualTo(3); + } +} From 6ec01eab6e698c52705a41572a89e2ec6dc81d09 Mon Sep 17 00:00:00 2001 From: Gregor Zeitlinger Date: Wed, 4 Feb 2026 14:25:15 +0100 Subject: [PATCH 2/5] fix instructions Signed-off-by: Gregor Zeitlinger --- CLAUDE.md | 37 ++++- CUSTOM_BUCKETS_VERIFICATION.md | 142 ------------------ examples/example-custom-buckets/README.md | 16 +- .../docker-compose/prometheus.yml | 2 +- 4 files changed, 42 insertions(+), 155 deletions(-) delete mode 100644 CUSTOM_BUCKETS_VERIFICATION.md diff --git a/CLAUDE.md b/CLAUDE.md index 492b11dc0..74a9676e8 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -69,19 +69,46 @@ Pre-built instrumentations: `prometheus-metrics-instrumentation-jvm`, `-caffeine ## Code Style - **Formatter**: Google Java Format (enforced via Spotless) -- **Line length**: 100 characters +- **Line length**: 100 characters (enforced for ALL files including Markdown, Java, YAML, etc.) - **Indentation**: 2 spaces - **Static analysis**: Error Prone with NullAway (`io.prometheus.metrics` package) - **Logger naming**: Logger fields must be named `logger` (not `log`, `LOG`, or `LOGGER`) - **Assertions in tests**: Use static imports from AssertJ (`import static org.assertj.core.api.Assertions.assertThat`) - **Empty catch blocks**: Use `ignored` as the exception variable name +- **Markdown code blocks**: Always specify language (e.g., ` ```java`, ` ```bash`, ` ```text`) ## Linting and Validation -- **IMPORTANT**: Always run `mise run build` after modifying Java files to ensure all lints, code formatting (Spotless), static analysis (Error Prone), and checkstyle checks pass -- **IMPORTANT**: Always run `mise run lint:super-linter` after modifying non-Java files (YAML, Markdown, shell scripts, JSON, etc.) -- Super-linter is configured to only show ERROR-level messages via `LOG_LEVEL=ERROR` in `.github/super-linter.env` -- Local super-linter version is pinned to match CI (see `.mise/tasks/lint/super-linter.sh`) +**CRITICAL**: These checks MUST be run before creating any commits. CI will fail if these checks fail. + +### Java Files + +- **ALWAYS** run `mise run build` after modifying Java files to ensure: + - Code formatting (Spotless with Google Java Format) + - Static analysis (Error Prone with NullAway) + - Checkstyle validation + - All tests pass + +### Non-Java Files (Markdown, YAML, JSON, shell scripts, etc.) + +- **ALWAYS** run `mise run lint:super-linter` after modifying non-Java files +- Super-linter will **auto-fix** many issues (formatting, trailing whitespace, etc.) +- It only reports ERROR-level issues (configured via `LOG_LEVEL=ERROR` in `.github/super-linter.env`) +- Common issues caught: + - Lines exceeding 100 characters in Markdown files + - Missing language tags in fenced code blocks + - Table formatting issues + - YAML/JSON syntax errors + +### Running Linters + +```bash +# After modifying Java files (run BEFORE committing) +mise run build + +# After modifying non-Java files (run BEFORE committing) +mise run lint:super-linter +``` ## Testing diff --git a/CUSTOM_BUCKETS_VERIFICATION.md b/CUSTOM_BUCKETS_VERIFICATION.md deleted file mode 100644 index e566a5383..000000000 --- a/CUSTOM_BUCKETS_VERIFICATION.md +++ /dev/null @@ -1,142 +0,0 @@ -# Native Histograms with Custom Buckets - Verification Report - -## Issue #1838: Verify that client_java supports native histograms with custom buckets - -### Summary - -This report documents the verification that the Prometheus Java client library properly supports native histograms with custom bucket configurations (NHCB - Native Histograms with Custom Buckets). - -### Background - -According to the [Prometheus Native Histograms specification](https://prometheus.io/docs/specs/native_histograms/), native histograms with custom buckets (schema -53) are a feature that allows representing classic histograms as native histograms with explicit bucket boundaries. - -**Key findings from the specification:** -- Schema -53 is used for custom bucket boundaries -- There is currently no dedicated protobuf field for custom bucket boundaries -- Custom-bucket histograms are exposed as **classic histograms** with custom boundaries -- Prometheus servers convert these to NHCB upon ingestion when configured with `convert_classic_histograms_to_nhcb` - -### Verification Approach - -The Java client library already supports custom bucket configurations through the `classicUpperBounds()`, `classicLinearUpperBounds()`, and `classicExponentialUpperBounds()` builder methods. These methods allow users to define custom bucket boundaries for histograms. - -Since NHCB is handled by Prometheus servers during ingestion (not by client libraries), our verification focuses on ensuring that: - -1. Histograms with custom bucket boundaries can be created -2. Custom buckets are correctly exposed in both text and protobuf formats -3. Both classic-only and dual (classic+native) histograms work with custom buckets -4. Various custom bucket configurations work correctly - -### Test Implementation - -Created comprehensive test suite: `CustomBucketsHistogramTest.java` - -The test suite includes 11 tests covering: - -#### 1. **Custom Buckets with Arbitrary Boundaries** (`testCustomBucketsWithArbitraryBoundaries`) -- Tests histogram with arbitrary custom bucket boundaries -- Verifies observations are distributed correctly across buckets -- Validates count and sum calculations - -#### 2. **Custom Buckets with Linear Boundaries** (`testCustomBucketsWithLinearBoundaries`) -- Tests histogram with linear custom bucket boundaries (equal-width buckets) -- Use case: Queue size monitoring with fixed intervals - -#### 3. **Custom Buckets with Exponential Boundaries** (`testCustomBucketsWithExponentialBoundaries`) -- Tests histogram with exponential custom bucket boundaries -- Use case: Metrics spanning multiple orders of magnitude (e.g., response sizes) - -#### 4. **Classic-Only Histogram with Custom Buckets** (`testCustomBucketsClassicOnlyHistogram`) -- Verifies custom buckets work when using `.classicOnly()` -- Confirms no native histogram representation is maintained - -#### 5. **Dual-Mode Histogram with Custom Buckets** (`testCustomBucketsDualModeHistogram`) -- Tests the default mode (both classic and native representations) -- Verifies custom classic buckets coexist with native histogram representation -- **This is the most relevant test for NHCB support** - -#### 6. **Text Format Output** (`testCustomBucketsTextFormatOutput`) -- Verifies custom buckets are correctly serialized in Prometheus text format -- Validates bucket labels (le) and counts - -#### 7. **Protobuf Format Output** (`testCustomBucketsProtobufFormatOutput`) -- Verifies custom buckets are correctly serialized in Prometheus protobuf format -- Validates bucket upper bounds and cumulative counts -- Confirms native histogram fields are present (for dual-mode) - -#### 8. **Custom Buckets with Negative Values** (`testCustomBucketsWithNegativeValues`) -- Tests custom buckets with negative boundary values -- Use case: Temperature or other metrics with negative ranges - -#### 9. **Custom Buckets with Labels** (`testCustomBucketsWithLabels`) -- Verifies custom buckets work correctly with labeled histograms -- Tests multiple label combinations - -#### 10. **Boundary Edge Cases** (`testCustomBucketsBoundaryEdgeCases`) -- Tests observations exactly on bucket boundaries -- Verifies buckets are inclusive of their upper bound - -#### 11. **Fine-Grained Custom Buckets** (`testCustomBucketsFineBoundaries`) -- Tests with very precise custom bucket boundaries -- Use case: High-precision measurements - -### Test Results - -All 11 tests pass successfully: - -``` -[INFO] Tests run: 11, Failures: 0, Errors: 0, Skipped: 0 -[INFO] BUILD SUCCESS -``` - -### Key Findings - -1. **Custom bucket support is fully functional**: The Java client library correctly handles histograms with custom bucket boundaries. - -2. **Dual-mode operation**: By default, histograms maintain both classic (with custom buckets) and native representations, which is ideal for NHCB support. - -3. **Correct serialization**: Custom buckets are properly serialized in both: - - Text format (with `le` labels) - - Protobuf format (with bucket upper bounds and cumulative counts) - -4. **Native histogram fields present**: When using dual-mode (default), the protobuf output includes native histogram fields (schema, zero_count, etc.) alongside the classic buckets. - -5. **Flexible bucket configurations**: The library supports: - - Arbitrary custom boundaries - - Linear boundaries (equal-width) - - Exponential boundaries - - Negative values - - Very fine-grained precision - -### Conclusion - -**The Prometheus Java client library (client_java) fully supports native histograms with custom buckets.** - -The library correctly: -- Allows users to define custom bucket boundaries -- Maintains both classic and native histogram representations by default -- Exposes custom buckets in the classic histogram format -- Serializes correctly in both text and protobuf formats - -Prometheus servers can convert these histograms to NHCB (schema -53) upon ingestion when configured with the `convert_classic_histograms_to_nhcb` option. - -### Recommendations - -1. **Documentation**: Consider documenting the NHCB support in the user-facing documentation, explaining that: - - Custom buckets are supported via the existing `classicUpperBounds()` API - - Prometheus servers handle the conversion to NHCB (schema -53) - - The default dual-mode is recommended for NHCB compatibility - -2. **Example**: Consider adding an example demonstrating custom bucket usage for NHCB scenarios. - -3. **Close issue #1838**: This verification confirms that custom bucket support is working correctly. - -### Test File Location - -- `prometheus-metrics-core/src/test/java/io/prometheus/metrics/core/metrics/CustomBucketsHistogramTest.java` - -### References - -- [Prometheus Native Histograms Specification](https://prometheus.io/docs/specs/native_histograms/) -- [GitHub Issue #1838](https://github.com/prometheus/client_java/issues/1838) -- [Prometheus client_model Repository](https://github.com/prometheus/client_model) diff --git a/examples/example-custom-buckets/README.md b/examples/example-custom-buckets/README.md index feeab13c9..82664a3b9 100644 --- a/examples/example-custom-buckets/README.md +++ b/examples/example-custom-buckets/README.md @@ -7,6 +7,7 @@ Prometheus converts them to native histograms with schema -53. ## What are Native Histograms with Custom Buckets? Native Histograms with Custom Buckets (NHCB) is a Prometheus feature that combines the benefits of: + - **Custom bucket boundaries**: Precisely defined buckets optimized for your specific use case - **Native histograms**: Efficient storage and querying capabilities of native histograms @@ -88,7 +89,7 @@ The Prometheus configuration enables NHCB conversion: ```yaml scrape_configs: - job_name: "custom-buckets-demo" - scrape_protocols: ['PrometheusProto'] + scrape_protocols: ["PrometheusProto"] convert_classic_histograms_to_nhcb: true scrape_classic_histograms: true ``` @@ -111,6 +112,7 @@ api_request_duration_seconds ### 3. View in Grafana The Grafana dashboard at [http://localhost:3000](http://localhost:3000) shows: + - p95 and p50 latencies for API endpoints (arbitrary custom buckets) - Queue size distribution (linear buckets) - Response size distribution (exponential buckets) @@ -143,12 +145,12 @@ Consider using custom buckets (and NHCB) when: ## Differences from Standard Native Histograms -| Feature | Standard Native Histograms | NHCB (Schema -53) | -|---------|---------------------------|-------------------| -| Bucket boundaries | Exponential (base 2^(2^-scale)) | Custom boundaries | -| Use case | General-purpose | Specific distributions | -| Mergeability | Can merge histograms with same schema | Cannot merge with different boundaries | -| Configuration | Schema level (0-8) | Explicit boundary list | +| Feature | Standard Native Histograms | NHCB (Schema -53) | +| ----------------- | ------------------------------------- | -------------------------------------- | +| Bucket boundaries | Exponential (base 2^(2^-scale)) | Custom boundaries | +| Use case | General-purpose | Specific distributions | +| Mergeability | Can merge histograms with same schema | Cannot merge with different boundaries | +| Configuration | Schema level (0-8) | Explicit boundary list | ## Cleanup diff --git a/examples/example-custom-buckets/docker-compose/prometheus.yml b/examples/example-custom-buckets/docker-compose/prometheus.yml index d056e4096..5c5782023 100644 --- a/examples/example-custom-buckets/docker-compose/prometheus.yml +++ b/examples/example-custom-buckets/docker-compose/prometheus.yml @@ -5,7 +5,7 @@ global: scrape_configs: - job_name: "custom-buckets-demo" # Use protobuf format to receive native histogram data - scrape_protocols: ['PrometheusProto'] + scrape_protocols: ["PrometheusProto"] # Convert classic histograms with custom buckets to NHCB (schema -53) convert_classic_histograms_to_nhcb: true # Also scrape classic histograms for comparison From 0f045727779b0fc2fd167b9b56721e2e6db59f17 Mon Sep 17 00:00:00 2001 From: Gregor Zeitlinger Date: Wed, 4 Feb 2026 14:29:56 +0100 Subject: [PATCH 3/5] Fix markdown lint errors in custom buckets README - Compress table columns to fit 100 char line limit - Break long URL link across lines Signed-off-by: Gregor Zeitlinger --- examples/example-custom-buckets/README.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/examples/example-custom-buckets/README.md b/examples/example-custom-buckets/README.md index 82664a3b9..a1d2c10bb 100644 --- a/examples/example-custom-buckets/README.md +++ b/examples/example-custom-buckets/README.md @@ -145,12 +145,12 @@ Consider using custom buckets (and NHCB) when: ## Differences from Standard Native Histograms -| Feature | Standard Native Histograms | NHCB (Schema -53) | -| ----------------- | ------------------------------------- | -------------------------------------- | -| Bucket boundaries | Exponential (base 2^(2^-scale)) | Custom boundaries | -| Use case | General-purpose | Specific distributions | -| Mergeability | Can merge histograms with same schema | Cannot merge with different boundaries | -| Configuration | Schema level (0-8) | Explicit boundary list | +| Feature | Standard Native Histograms | NHCB (Schema -53) | +| ----------------- | ------------------------------- | --------------------------------- | +| Bucket boundaries | Exponential (base 2^(2^-scale)) | Custom boundaries | +| Use case | General-purpose | Specific distributions | +| Mergeability | Can merge with same schema | Cannot merge different boundaries | +| Configuration | Schema level (0-8) | Explicit boundary list | ## Cleanup From 4f770b8b163ab5ba66c984a35e818be7acd5bf06 Mon Sep 17 00:00:00 2001 From: Gregor Zeitlinger Date: Wed, 4 Feb 2026 15:50:02 +0100 Subject: [PATCH 4/5] fix Signed-off-by: Gregor Zeitlinger --- examples/example-custom-buckets/README.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/examples/example-custom-buckets/README.md b/examples/example-custom-buckets/README.md index a1d2c10bb..a7a6a8564 100644 --- a/examples/example-custom-buckets/README.md +++ b/examples/example-custom-buckets/README.md @@ -162,6 +162,9 @@ docker-compose down ## Further Reading + + + - [Prometheus Native Histograms Specification](https://prometheus.io/docs/specs/native_histograms/) - [Prometheus Java Client Documentation](https://prometheus.github.io/client_java/) - [OpenTelemetry Exponential Histograms](https://opentelemetry.io/docs/specs/otel/metrics/data-model/#exponentialhistogram) From 3d8adb0e97a7526bda450dd77eb6026b9eeda7fe Mon Sep 17 00:00:00 2001 From: Gregor Zeitlinger Date: Thu, 5 Feb 2026 11:42:45 +0100 Subject: [PATCH 5/5] Update CLAUDE.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Gregor Zeitlinger --- CLAUDE.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CLAUDE.md b/CLAUDE.md index 74a9676e8..9c1c60571 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -87,7 +87,7 @@ Pre-built instrumentations: `prometheus-metrics-instrumentation-jvm`, `-caffeine - Code formatting (Spotless with Google Java Format) - Static analysis (Error Prone with NullAway) - Checkstyle validation - - All tests pass + - Build succeeds (tests are skipped; run `mise run test` or `mise run test-all` to execute tests) ### Non-Java Files (Markdown, YAML, JSON, shell scripts, etc.)