Add Guidance wrt Labelling to Naming and Rules Best Practices

[conallob]

Add Guidance wrt Labelling to Naming and Rules Best Practices to docs/practices/naming.md and docs/practices/rules.md, specifically:

• The primary purposes of job and instance
• Include WARNINGS about accidentally stripping the job label, especially in multi-tenant systems

This Fixes #2690

[conallob]

Friendly ping @SuperQ @beorn7 ?

[conallob]

Obligatory post-it note reminder: https://photos.app.goo.gl/Bkfir4wRiLtNVG4W8

[beorn7]

With my current patchy availability, there is little chance I get to this anytime soon. Maybe @juliusv has a qualified opinion here?

[conallob]

Friendly ping @SuperQ @juliusv

[andrechalella]

Hey @conallob, congrats for the nice PR!

Just one thing: maybe you could fix the eaach typo in

• The job label is a primary key to differentiate metrics from eaach other.

There is a small confusion I would love to see fixed, in a paragraph just below one of your edits. It is:

To keep the operations clean, _sum is omitted if there are other operations,
as sum().

I don't understand the as sum() part. Like, "x is omitted if there are other operations such as x"? It doesn't make sense to me, in a very basic way. I know it's out of the scope of this PR, but maybe you could touch it to clarify.

[conallob]

Hey @conallob, congrats for the nice PR!

Just one thing: maybe you could fix the eaach typo in

• The job label is a primary key to differentiate metrics from eaach other.

There is a small confusion I would love to see fixed, in a paragraph just below one of your edits. It is:

Fixed the typo.

To keep the operations clean, _sum is omitted if there are other operations,
as sum().

I don't understand the as sum() part. Like, "x is omitted if there are other operations such as x"? It doesn't make sense to me, in a very basic way. I know it's out of the scope of this PR, but maybe you could touch it to clarify.

I'm afraid that best practice is unrelated.

It also makes sense as written, once you've written enough rules. It's weighing up the trade-off between tracking the chain of operations across a pipeline of rules vs the rule name growing unwieldy. Many of these best practices trace back to specific philosophies from Prometheus' predecessor.

If you still think it needs a polish, please a separate doc bug.

[conallob]

Ping @juliusv , since @SuperQ is currently unavailable for life reasons

[SuperQ]

This is misleading. Prometheus doesn't have the concept of "primary key". Not even metric names are a "primary key".

[conallob]

Fair, especially since folks used to SQL DBs will jump to the conclusion that it's a SQL DB, which it isn't.

Iterated on the language to avoid creating ambiguity

[conallob]

PTAL

[conallob]

Friendly ping?

[conallob]

Friendly, you're not at SRECon EMEA this week, ping?

[SuperQ]

Suggested change

	* The `instance` label will include the `ip:port` what was scraped, providing a crucial breadcrumb for debugging scrape time issues
	* The `instance` label by default will include the `ip:port` what was scraped.

[SuperQ]

Suggested change

	* The `job` label is one of the few ubiquitious labels, set at scrape time, and is used to identify metrics scraped from the same target/exporter.
	* The `job` is a default target label set by the scrape configs and is used to identify metrics scraped from the same target/exporter.

[SuperQ]

I'm not sure this is really a useful note here, as this applies to all label matching.

Suggested change

* If not specified in PromQL expressions, they will match unrelated metrics with the same name. This is especially true in a multi system or multi tenant installation

[conallob]

It applies to all labels. But job and instance are two uniform labels found on every metric, including ubiquitous synthetic metrics such as up

[SuperQ]

Yes, but it's not related to job, but related to "target labels" and discovery. That is a different thing and related to querying, not creating labels.

[SuperQ]

This warning doesn't make a lot of sense to me. It has a high probability of being quoted as copy-pasta without being understood. Let's just drop it.

Suggested change

WARNING: When using `without`, be careful not to strip out the `job` label accidentally.

[conallob]

I don't want to encourage copy-pasta, but this is an important point.

If using alerting expressions like up{job=bla} > 0 for 3m , you need to be careful not to accidentally strip the job label. If you do, your alert no longer works as intended.

I'll polish the wording here

[SuperQ]

This doesn't seem related to naming practices, which is what this guide is about.

[conallob]

For perspective, one of the motivations behind this PR is the anti-patterm of writing alert expressions intended for a single tenant system, which has evolved into a multi-tenant system.

e.g up{} for 5m without defining a job label works for one job.

Once you start adding additional jobs that match on the same labels (e.g Daemonsets, fleet-wide node_exporter, etc), teams start getting paged for systems they don't own or care about