(fleet/prometheus-alerts) set up gitops for alert deployment

This allows for prometheus rules to be automagically applied through rancher
fleet. Simply adding some rules to the fleet directory will deploy them into the
cluster, ready for the prometheus operator to pick them up and deploy.

Author: fbegyn

fleet/lib/prometheus-alerts/Chart.yaml

@@ -0,0 +1,5 @@
+apiVersion: v2
+appVersion: 0.0.0
+description: Prometheus LSST rules GitOps
+name: prometheus-alerts
+version: 0.0.0

fleet/lib/prometheus-alerts/README.md

@@ -0,0 +1,32 @@
+# Prometheus rules GitOps
+
+Any Prometheus rules file defined in the `/rules` directory will be deployed to
+the cluster. It's possible to define a default namespace in the `values.yaml`
+file with the `rules.namespace` key.
+
+## Adding Prometheus rules
+
+1. Write the Prometheus rules in a yaml file according to the [prometheus
+   specification](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/).
+1. Add the YAML file to the `/rules` directory
+1. Commit
+
+## Prometheus rule AURA standards
+
+* `summary` annotation: The `summary` annotation is used to be able to describe a
+  group of alerts incomming. This annotation DOES NOT contain any templated
+  variables and provides a simple single sentence summary of what the alert is
+  about. For example "Disk space full in 24h". When a cluster triggers several
+  alerts, it can be hany to group these alerts into a single notification, this
+  is when the `summary` can be used.
+* `discription` annotation: This provides a detailed overview of the alert
+  specifically to this instance of the alert. It MAY contain templated variables
+  to enrich the message.
+* `receiver` label: The receiver label is used by alertmanager to decide on the
+  routing of the notification for the alert. It exists out of `,` seperated list
+  of receivers, pre- and suffixed with `,` to make regex matching easier in the
+  alertmanager. For example: `,slack,squadcast,email,` The receivers are defined
+  in the alertmanager configuration.
+  Currently (20240503) the following receivers are configured:
+   * `slack-test`
+   * `squadcast-test`

fleet/lib/prometheus-alerts/fleet.yaml

@@ -0,0 +1,14 @@
+---
+defaultNamespace: &name prometheus-alerts
+labels:
+  bundle: *name
+namespaceLabels:
+  lsst.io/discover: "true"
+helm:
+  releaseName: *name
+  takeOwnership: true
+  waitForJobs: true
+dependsOn:
+  - selector:
+      matchLabels:
+        bundle: prometheus-operator-crds

fleet/lib/prometheus-alerts/templates/prometheusrule.yaml

@@ -0,0 +1,34 @@
+# yamllint disable-file
+{{- if .Values.rules.enabled }}
+{{- $files := .Files.Glob "rules/**.yaml" }}
+{{- range $rawpath, $content := $files }}
+{{- $path := ($rawpath | lower | replace " " "-") }}
+{{- $ruleDir := dir $path }}
+{{- $ruleFile := base $path }}
+{{- $namespaceSplit := regexSplit "\\/+" $ruleDir -1 }}
+{{- $namespace := $.Values.rules.namespace | default $.Release.Namespace }}
+{{- if (eq (len $namespaceSplit) 2) }}
+{{- $namespace = (index $namespaceSplit 1) }}
+{{- end }}
+{{- $alertName := lower (index (regexSplit "\\.yaml" $ruleFile -1) 0) }}
+---
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: {{ printf "%s-%s" "alert" $alertName | trunc 63 | trimSuffix "-" }}
+  namespace: {{ $namespace }}
+  labels:
+    lsst.io/component: "prometheus-rules"
+    lsst.io/dir: {{ $ruleDir | quote }}
+    lsst.io/file: {{ $ruleFile | quote }}
+    {{- with $.Values.rules.additionalLabels }}
+    {{- toYaml . | nindent 4 }}
+    {{- end }}
+  {{- with $.Values.rules.additionalAnnotations }}
+  annotations:
+    {{- toYaml . | nindent 4 }}
+  {{- end }}
+spec:
+  {{ $content | toString }}
+{{ end }}
+{{ end }}

fleet/lib/prometheus-alerts/values.yaml

@@ -0,0 +1,7 @@
+---
+rules:
+  enabled: true
+  namespace: ~
+  additionalAnnotations: {}
+  additionalLabels:
+    lsst.io/rule: "true"