Alert Cleanup at ISAR

[rbo]

Alerts

Name	Severity	State	Total	Solution
NTOPodsNotReady	warning	firing	2	✅ All nodes online
ClusterOperatorDegraded	warning	firing	1	✅ All nodes online
DeprecatedMachineType	warning	firing	1	✅ Comments below
UnsupportedHCOModification	info	firing	1	✅ Comments below
KubeVirtDeprecatedAPIRequested	info	firing	1	✅ Solved alone..
ArgoCDSyncAlert	warning	firing	5	✅ Comments below
__ArgoCDSyncAlert cluster-configuration	OutOfSync	Healthy		✅ - Debug log still enabled
__ArgoCDSyncAlert operator-acs	OutOfSync	Healthy		✅ Fixed
__ArgoCDSyncAlert operator-mtv	OutOfSync	Healthy		✅ Comment bellow
__ArgoCDSyncAlert operator-nmstate	OutOfSync	Healthy		✅ synced, namespace labels
__ArgoCDSyncAlert operator-ocp-v	OutOfSync	Healthy		✅ synced
__ArgoCDSyncAlert operator-openshift-storage	OutOfSync	Healthy		✅ Fixed
__ArgoCDSyncAlert operator-quay	OutOfSync	Healthy		✅ synced
__ArgoCDSyncAlert stormshift-app	OutOfSync	Healthy		✅ Fixed
InsightsRecommendationActive	info	firing	2
DeschedulerPSIDisabled	critical	firing	1	✅ All nodes online
LvSetDeletionAlert	warning	firing	1	✅ #345
Watchdog	none	firing	1	✅ Default
PrometheusPossibleNarrowSelectors	warning	firing	1
TargetDown	warning	firing	9	✅ All nodes online
KubeDaemonSetMisScheduled	warning	firing	11	✅ All nodes online
KubeDaemonSetRolloutStuck	warning	firing	15	✅ All nodes online
KubeJobFailed	warning	firing	1	✅ Comments below
KubeNodeUnreachable	warning	firing	2	✅ All nodes online
PrometheusOperatorRejectedResources	warning	firing	1	✅ Comments below
TelemeterClientFailures	warning	firing	1

[rbo]

Booted all Machines #338 to solve some unreachable & co

[rbo]

KubeJobFailed

% oc get jobs -n openshift-operator-lifecycle-manager

NAME                        STATUS     COMPLETIONS   DURATION   AGE
collect-profiles-29431650   Failed     0/1           6d19h      6d19h
collect-profiles-29441445   Complete   1/1           5s         36m
collect-profiles-29441460   Complete   1/1           5s         21m
collect-profiles-29441475   Complete   1/1           5s         6m6s

% oc delete jobs -n openshift-operator-lifecycle-manager collect-profiles-29431650
job.batch "collect-profiles-29431650" deleted

Long time ago. Failed job deleted.

[rbo]

PrometheusOperatorRejectedResources

https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/PrometheusOperatorRejectedResources.md

% oc get events --field-selector involvedObject.kind=ServiceMonitor --all-namespaces

NAMESPACE       LAST SEEN   TYPE      REASON                 OBJECT                                                  MESSAGE
openshift-nfd   12m         Warning   InvalidConfiguration   servicemonitor/nfd-controller-manager-metrics-monitor   "openshift-nfd/nfd-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits

% oc get -n openshift-nfd ServiceMonitor  nfd-controller-manager-metrics-monitor -o yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  creationTimestamp: "2025-11-24T12:25:11Z"
  generation: 1
  labels:
    control-plane: controller-manager
    olm.managed: "true"
  name: nfd-controller-manager-metrics-monitor
  namespace: openshift-nfd
  ownerReferences:
  - apiVersion: operators.coreos.com/v1alpha1
    blockOwnerDeletion: false
    controller: false
    kind: ClusterServiceVersion
    name: nfd.4.20.0-202512081147
    uid: e03b8950-afcb-410f-98f7-8707b7cda550
  resourceVersion: "3455470289"
  uid: 7ded2124-f490-417a-97d2-73acb13586f0
spec:
  endpoints:
  - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
    interval: 30s
    path: /metrics
    port: https
    scheme: https
    tlsConfig:
      caFile: /etc/prometheus/configmaps/serving-certs-ca-bundle/service-ca.crt
      serverName: nfd-controller-manager-metrics-service.openshift-nfd.svc
  selector:
    matchLabels:
      control-plane: controller-manager


% oc get -n openshift-nfd svc nfd-controller-manager-metrics-service -o yaml
apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.openshift.io/serving-cert-secret-name: node-feature-discovery-operator-tls
  creationTimestamp: "2025-11-24T12:25:11Z"
  labels:
    control-plane: controller-manager
    olm.managed: "true"
    operators.coreos.com/nfd.openshift-nfd: ""
  name: nfd-controller-manager-metrics-service
  namespace: openshift-nfd
  ownerReferences:
  - apiVersion: operators.coreos.com/v1alpha1
    blockOwnerDeletion: false
    controller: false
    kind: ClusterServiceVersion
    name: nfd.4.20.0-202512081147
    uid: e03b8950-afcb-410f-98f7-8707b7cda550
  resourceVersion: "3455472574"
  uid: 1c87b2aa-a1fa-487f-b386-461320e253f4
spec:
  clusterIP: 172.30.97.219
  clusterIPs:
  - 172.30.97.219
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: https
    port: 8443
    protocol: TCP
    targetPort: https
  selector:
    control-plane: controller-manager
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

%  oc get pods -n openshift-nfd -l control-plane=controller-manager
NAME                                      READY   STATUS    RESTARTS       AGE
nfd-controller-manager-7c4f5dc6fb-z68t7   1/1     Running   24 (10h ago)   3d15h

OpenShift monitoring label is missing:

% oc get namespace openshift-nfd -o yaml
apiVersion: v1
kind: Namespace
metadata:
  annotations:
    openshift.io/sa.scc.mcs: s0:c42,c9
    openshift.io/sa.scc.supplemental-groups: 1001740000/10000
    openshift.io/sa.scc.uid-range: 1001740000/10000
    security.openshift.io/MinimallySufficientPodSecurityStandard: privileged
  creationTimestamp: "2025-11-24T12:24:59Z"
  labels:
    kubernetes.io/metadata.name: openshift-nfd
    olm.operatorgroup.uid/95f3d280-45d5-452e-9dd1-3eb7516d2fa9: ""
    pod-security.kubernetes.io/audit: privileged
    pod-security.kubernetes.io/audit-version: latest
    pod-security.kubernetes.io/warn: privileged
    pod-security.kubernetes.io/warn-version: latest
    security.openshift.io/scc.podSecurityLabelSync: "true"
  name: openshift-nfd
  resourceVersion: "3353483127"
  uid: 03b7b831-e45c-44fd-b52b-7f136ef678ba
spec:
  finalizers:
  - kubernetes
status:
  phase: Active

oc label namespace openshift-nfd openshift.io/cluster-monitoring=true

[rbo]

PrometheusPossibleNarrowSelectors

• https://docs.redhat.com/en/documentation/monitoring_stack_for_red_hat_openshift/4.19/html/release_notes_for_openshift_monitoring/monitoring-release-notes#monitoring-4-19-changes-alerting-rules_monitoring-release-notes
• https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/PrometheusPossibleNarrowSelectors.md

%  oc -n openshift-monitoring edit configmap cluster-monitoring-config
Add log level debug 

Wait for prom restart:

% export NAMESPACE='openshift-monitoring'
% export NAME='prometheus'
% oc get pods -l app.kubernetes.io/name=$NAME
NAME               READY   STATUS    RESTARTS   AGE
prometheus-k8s-0   6/6     Running   0          3d16h
prometheus-k8s-1   5/6     Running   0          110s

After the restart, the ALL alert was gone for a while. Now it's back:

% oc -n $NAMESPACE logs -l app.kubernetes.io/name=$NAME --tail=-1 \
| grep 'narrow_matcher_label' | sort | uniq -c | sort -n
   1 time=2025-12-23T11:39:06.571Z level=DEBUG source=parse.go:102 msg="selector set to explicitly match an integer, but values could be floats" component=parser narrow_matcher_label=le integer=1 matchers="[job=\"apiserver\" service=\"kubernetes\" le=\"1\" verb=~\"POST|PUT|DELETE|PATCH\" __name__=\"apiserver_request_duration_seconds_bucket\"]"
   1 time=2025-12-23T11:40:05.422Z level=DEBUG source=parse.go:102 msg="selector set to explicitly match an integer, but values could be floats" component=parser narrow_matcher_label=le integer=1 matchers="[job=\"apiserver\" service=\"kubernetes\" le=\"1\" verb=~\"POST|PUT|DELETE|PATCH\" __name__=\"apiserver_request_duration_seconds_bucket\"]"
   1 time=2025-12-23T11:41:05.682Z level=DEBUG source=parse.go:102 msg="selector set to explicitly match an integer, but values could be floats" component=parser narrow_matcher_label=le integer=1 matchers="[job=\"apiserver\" service=\"kubernetes\" le=\"1\" verb=~\"POST|PUT|DELETE|PATCH\" __name__=\"apiserver_request_duration_seconds_bucket\"]"
   1 time=2025-12-23T11:42:05.246Z level=DEBUG source=parse.go:102 msg="selector set to explicitly match an integer, but values could be floats" component=parser narrow_matcher_label=le integer=1 matchers="[job=\"apiserver\" service=\"kubernetes\" le=\"1\" verb=~\"POST|PUT|DELETE|PATCH\" __name__=\"apiserver_request_duration_seconds_bucket\"]"

[rbo]

UnsupportedHCOModification

Let's remove the following annotations form HCO

kubevirt.kubevirt.io/jsonpatch: '[ {"op": "add", "path": "/spec/configuration/developerConfiguration/featureGates/-",
      "value": "VolumesUpdateStrategy"}, {"op": "add", "path": "/spec/configuration/developerConfiguration/featureGates/-",
      "value": "VolumeMigration"} ]'

% oc edit hco -n openshift-cnv kubevirt-hyperconverged
Warning: spec.featureGates.withHostPassthroughCPU is deprecated and ignored. It will be removed in a future version;
Warning: spec.featureGates.deployTektonTaskResources is deprecated and ignored. It will be removed in a future version;
Warning: spec.featureGates.nonRoot is deprecated and ignored. It will be removed in a future version;
Warning: spec.featureGates.enableManagedTenantQuota is deprecated and ignored. It will be removed in a future version;
hyperconverged.hco.kubevirt.io/kubevirt-hyperconverged edited

[rbo]

TelemeterClientFailures

• https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/TelemeterClientFailures.md

% oc logs -n openshift-monitoring deployment.apps/telemeter-client -c telemeter-client -f
...
level=error caller=forwarder.go:296 ts=2025-12-23T11:45:11.452840465Z component=forwarder/worker msg="unable to forward results" err="unable to authorize to server: unable to exchange initial token for a long lived token: 409:\nthe provided cluster identifier is already in use under a different account or is not sufficiently random\n"
...

401 and 403 codes indicate a misconfiguration of the client. A typical reason is the global cluster pull secret.

Lets check the pull secret

Pull secret updated, with a new one from rh-ee-rbohne with proper OpenShift subscriptions

[rbo]

DeprecatedMachineType

Virtual Machine 'windows-rdp' in namespace 'rbohne-mgmt' is using machine type 'pc-q35-rhel8.6.0', which is deprecated. Current status: 'non_running'.

We can delete the VM

[rbo]

ArgoCDSyncAlert operator-mtv

• Subscription Live: installPlanApproval: Automatic, Git: installPlanApproval: Manual
  • Switch installPlanApproval from Automatic to Manuel switch back immediately.
• Subscription has following annotation: operatorpolicy.policy.open-cluster-management.io/managed: open-cluster-management-policies.mtv-operator 🤔

oc get OperatorPolicy -n open-cluster-management-policies mtv-operator
NAME           AGE
mtv-operator   21h

OwnerReference:

ownerReferences:
  - apiVersion: work.open-cluster-management.io/v1
    kind: AppliedManifestWork
    name: 0dae03d5867a066358318689195a2ee96803079bf46e37f4fd3def4a47173d85-addon-mtv-operator-deploy-0
    uid: 3a2575f8-d965-448f-83f3-b7255f914402

AppliedManifestWork/0dae03d5867a066358318689195a2ee96803079bf46e37f4fd3def4a47173d85-addon-mtv-operator-deploy-0

% oc get AppliedManifestWork 0dae03d5867a066358318689195a2ee96803079bf46e37f4fd3def4a47173d85-addon-mtv-operator-deploy-0 --show-labels
NAME                                                                                           AGE   LABELS
0dae03d5867a066358318689195a2ee96803079bf46e37f4fd3def4a47173d85-addon-mtv-operator-deploy-0   21h   open-cluster-management.io/addon-name=mtv-operator

Maybe because of stormshift/clusters@ee4fbb1