Skip to content

Allow multiple exemplars on counters and histograms#2838

Open
dashpole wants to merge 1 commit intoprometheus:mainfrom
dashpole:multiple_exemplars
Open

Allow multiple exemplars on counters and histograms#2838
dashpole wants to merge 1 commit intoprometheus:mainfrom
dashpole:multiple_exemplars

Conversation

@dashpole
Copy link
Contributor

Fixes prometheus/OpenMetrics#311

Remove the restriction on having a single exemplar on counters. This matches PRW 2.0, OpenTelemetry, and makes exemplars more consistent between types.

While reading through exemplar language, I noticed the MUST requirement for histogram-bucket-aligned exemplars on fixed-bucket histograms. This relaxes (to SHOULD) the alignment of exemplars with histogram buckets. This is done because:

  • Consistency between types (counters, histograms, and native histograms support multiple exemplars without restrictions.
  • Reduces the amount of validation required by clients.
  • Allows native histograms and classic histograms to share exemplar sampling in Prometheus clients (when storing both types internally), without confining exemplars to the classic buckets.

Signed-off-by: David Ashpole <dashpole@google.com>
If the NaN value is allowed, it MUST be counted in the +Inf bucket, and MUST NOT be counted in any other bucket. The rationale is that NaN does not belong to any bucket mathematically, however instrumentation libraries traditionally put it into the +Inf bucket.

Classic Bucket values MAY have exemplars. The value of the exemplar MUST be within the Classic Bucket. Exemplars SHOULD be put into the Classic Bucket with the lowest threshold that includes the exemplar value. A Classic Bucket MUST NOT have more than one exemplar.
Classic Bucket values MAY have exemplars. The value of the exemplar SHOULD be within the Classic Bucket. Exemplars SHOULD be put into the Classic Bucket with the lowest threshold that includes the exemplar value. A Classic Bucket SHOULD NOT have more than one exemplar.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WG notes

  • It's typical in Otel to go outside
  • How this works for native histograms? Separate storages? We could switch to the language native histograms uses - "evenly distributed". As long as it's evenly distributed, it's fine if it's not within buckets.
  • Krajo: For classic histograms are by design it's inside bucket. For classic histograms we can have only one exemplar per bucket. We can refine this to say similar thing to native histograms. In Go SDK exemplars storages are different for native vs classic (although on proto it's one field - to be checked).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • How this will work with 1.0 compatible storage

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • K: Maybe we should say one exemplar per bucket? We could change over time.
  • D: What's value of saying MUST
  • K: Either be strict (same as before), or be relaxed and lose exemplars for 1.0 - it's likely fine

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

K: We need to rewrite this rule as this applies in a different way (composite) - should be one per bucket / align wording

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

OM 2.0: All types MUST support multiple exemplars

2 participants