LvSetDeletionAlert - LocalVolumeSet has had a deletion timestamp older than 72 hours

[rbo]

@DanielFroehlich accidentally deleted the LocalVolumeSet for ODF

apiVersion: local.storage.openshift.io/v1alpha1
kind: LocalVolumeSet
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"local.storage.openshift.io/v1alpha1","kind":"LocalVolumeSet","metadata":{"annotations":{},"name":"local-odf-nvme","namespace":"openshift-local-storage"},"spec":{"deviceInclusionSpec":{"deviceMechanicalProperties":["NonRotational"],"deviceTypes":["disk"],"minSize":"1Gi"},"nodeSelector":{"nodeSelectorTerms":[{"matchExpressions":[{"key":"kubernetes.io/hostname","operator":"In","values":["ceph10","ceph11","ceph12"]}]}]},"storageClassName":"local-odf-nvme","tolerations":[{"effect":"NoSchedule","key":"node.ocs.openshift.io/storage","operator":"Equal","value":"true"}],"volumeMode":"Block"}}
  creationTimestamp: "2024-10-01T18:09:00Z"
  deletionGracePeriodSeconds: 0
  deletionTimestamp: "2025-12-19T18:59:52Z"
  finalizers:
  - storage.openshift.com/local-volume-protection
  generation: 2
  name: local-odf-nvme
  namespace: openshift-local-storage
  resourceVersion: "3476177475"
  uid: 08c14012-20cb-4409-8f10-6c20c97def44
spec:
  deviceInclusionSpec:
    deviceMechanicalProperties:
    - NonRotational
    deviceTypes:
    - disk
    minSize: 1Gi
  nodeSelector:
    nodeSelectorTerms:
    - matchExpressions:
      - key: kubernetes.io/hostname
        operator: In
        values:
        - ceph10
        - ceph11
        - ceph12
  storageClassName: local-odf-nvme
  tolerations:
  - effect: NoSchedule
    key: node.ocs.openshift.io/storage
    operator: Equal
    value: "true"
  volumeMode: Block
status:
  conditions:
  - lastTransitionTime: "2024-10-01T18:09:00Z"
    message: 'DiskMaker: Available'
    status: "True"
    type: DaemonSetsAvailable
  - lastTransitionTime: "2024-10-01T18:09:00Z"
    message: 'Operator error: failed to update localvolumeset finalizer: localvolumeset
      openshift-local-storage/local-odf-nvme has owned persistentvolumes in use'
    status: "False"
    type: Available
  observedGeneration: 1
  totalProvisionedDeviceCount: 9

[rbo]

ODF is still available:

oc rsh deployment/rook-ceph-tools  sh -c 'ceph -s; ceph osd status'
  cluster:
    id:     26331761-7b92-4d3d-bbc8-f0da9765fb21
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum a,g,h (age 4h)
    mgr: a(active, since 12m), standbys: b
    mds: 1/1 daemons up, 1 hot standby
    osd: 6 osds: 6 up (since 5h), 6 in (since 6w)
    rgw: 1 daemon active (1 hosts, 1 zones)

  data:
    volumes: 1/1 healthy
    pools:   15 pools, 424 pgs
    objects: 637.37k objects, 2.4 TiB
    usage:   4.8 TiB used, 6.1 TiB / 11 TiB avail
    pgs:     424 active+clean

  io:
    client:   1.6 KiB/s rd, 1.5 MiB/s wr, 2 op/s rd, 115 op/s wr

ID  HOST     USED  AVAIL  WR OPS  WR DATA  RD OPS  RD DATA  STATE
 0  ceph11   788G  1074G     25      200k      0       11   exists,up
 2  ceph11   864G   998G     45      295k      1       16   exists,up
 3  ceph12   791G  1071G     24      692k      1        0   exists,up
 5  ceph12   860G  1002G     26      185k      0        0   exists,up
 6  ceph10   842G  1020G     18      146k      0       11   exists,up
 7  ceph10   755G  1107G     21      416k      1      101   exists,up

[rbo]

oc get pv -l storage.openshift.com/owner-name=local-odf-nvme
NAME                CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                                          STORAGECLASS     VOLUMEATTRIBUTESCLASS   REASON   AGE
local-pv-459bdc39   1863Gi     RWO            Delete           Bound    openshift-storage/ocs-deviceset-local-odf-nvme-2-data-1tkh6p   local-odf-nvme   <unset>                          143d
local-pv-71db5eff   1863Gi     RWO            Delete           Bound    openshift-storage/ocs-deviceset-local-odf-nvme-0-data-0xtf58   local-odf-nvme   <unset>                          447d
local-pv-7a2ab8cf   1863Gi     RWO            Delete           Bound    openshift-storage/ocs-deviceset-local-odf-nvme-2-data-0wqzqn   local-odf-nvme   <unset>                          447d
local-pv-8719b445   1863Gi     RWO            Delete           Bound    openshift-storage/ocs-deviceset-local-odf-nvme-1-data-2rmcz9   local-odf-nvme   <unset>                          143d
local-pv-ce2fef6d   1863Gi     RWO            Delete           Bound    openshift-storage/ocs-deviceset-local-odf-nvme-0-data-1jmcp4   local-odf-nvme   <unset>                          447d
local-pv-ef2dac4c   1863Gi     RWO            Delete           Bound    openshift-storage/ocs-deviceset-local-odf-nvme-1-data-1z8gbw   local-odf-nvme   <unset>                          447d

% oc get pv -l storage.openshift.com/owner-name=local-odf-nvme -o custom-columns=NAME:.metadata.name,persistentVolumeReclaimPolicy:.spec.persistentVolumeReclaimPolicy
NAME                persistentVolumeReclaimPolicy
local-pv-459bdc39   Delete
local-pv-71db5eff   Delete
local-pv-7a2ab8cf   Delete
local-pv-8719b445   Delete
local-pv-ce2fef6d   Delete
local-pv-ef2dac4c   Delete

Let's change the persistentVolumeReclaimPolicy to Retain. Try to don't loose the data.

[rbo]

% oc get pv -l storage.openshift.com/owner-name=local-odf-nvme -o name | xargs -t -n1 oc patch --type merge -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'
oc patch --type merge -p {"spec":{"persistentVolumeReclaimPolicy":"Retain"}} persistentvolume/local-pv-459bdc39
persistentvolume/local-pv-459bdc39 patched
oc patch --type merge -p {"spec":{"persistentVolumeReclaimPolicy":"Retain"}} persistentvolume/local-pv-71db5eff
persistentvolume/local-pv-71db5eff patched
oc patch --type merge -p {"spec":{"persistentVolumeReclaimPolicy":"Retain"}} persistentvolume/local-pv-7a2ab8cf
persistentvolume/local-pv-7a2ab8cf patched
oc patch --type merge -p {"spec":{"persistentVolumeReclaimPolicy":"Retain"}} persistentvolume/local-pv-8719b445
persistentvolume/local-pv-8719b445 patched
oc patch --type merge -p {"spec":{"persistentVolumeReclaimPolicy":"Retain"}} persistentvolume/local-pv-ce2fef6d
persistentvolume/local-pv-ce2fef6d patched
oc patch --type merge -p {"spec":{"persistentVolumeReclaimPolicy":"Retain"}} persistentvolume/local-pv-ef2dac4c
persistentvolume/local-pv-ef2dac4c patched

% oc get pv -l storage.openshift.com/owner-name=local-odf-nvme -o custom-columns=NAME:.metadata.name,persistentVolumeReclaimPolicy:.spec.persistentVolumeReclaimPolicy
NAME                persistentVolumeReclaimPolicy
local-pv-459bdc39   Retain
local-pv-71db5eff   Retain
local-pv-7a2ab8cf   Retain
local-pv-8719b445   Retain
local-pv-ce2fef6d   Retain
local-pv-ef2dac4c   Retain

[rbo]

Try to replace LocalVolumeSet:

apiVersion: local.storage.openshift.io/v1alpha1
kind: LocalVolumeSet
metadata:
  annotations:
  name: local-odf-nvme
  namespace: openshift-local-storage
spec:
  deviceInclusionSpec:
    deviceMechanicalProperties:
    - NonRotational
    deviceTypes:
    - disk
    minSize: 1Gi
  nodeSelector:
    nodeSelectorTerms:
    - matchExpressions:
      - key: kubernetes.io/hostname
        operator: In
        values:
        - ceph10
        - ceph11
        - ceph12
  storageClassName: local-odf-nvme
  tolerations:
  - effect: NoSchedule
    key: node.ocs.openshift.io/storage
    operator: Equal
    value: "true"
  volumeMode: Block

% oc replace -f Backup-LocalVolumeSet-local-odf-nvme.yaml
localvolumeset.local.storage.openshift.io/local-odf-nvme replaced

[rbo]

🫨

% oc get localvolumeset.local.storage.openshift.io -A
No resources found

% oc create -f Backup-LocalVolumeSet-local-odf-nvme.yaml
localvolumeset.local.storage.openshift.io/local-odf-nvme created
% oc get localvolumeset.local.storage.openshift.io -A
NAMESPACE                 NAME             AGE
openshift-local-storage   local-odf-nvme   25s

[rbo]

% oc describe LocalVolumeSet
Name:         local-odf-nvme
Namespace:    openshift-local-storage
Labels:       <none>
Annotations:  <none>
API Version:  local.storage.openshift.io/v1alpha1
Kind:         LocalVolumeSet
Metadata:
  Creation Timestamp:  2025-12-23T10:37:05Z
  Finalizers:
    storage.openshift.com/local-volume-protection
  Generation:        1
  Resource Version:  3476423808
  UID:               4863f4e7-0c55-41f2-8e50-5cc19ba8f489
Spec:
  Device Inclusion Spec:
    Device Mechanical Properties:
      NonRotational
    Device Types:
      disk
    Min Size:  1Gi
  Node Selector:
    Node Selector Terms:
      Match Expressions:
        Key:       kubernetes.io/hostname
        Operator:  In
        Values:
          ceph10
          ceph11
          ceph12
  Storage Class Name:  local-odf-nvme
  Tolerations:
    Effect:     NoSchedule
    Key:        node.ocs.openshift.io/storage
    Operator:   Equal
    Value:      true
  Volume Mode:  Block
Status:
  Conditions:
    Last Transition Time:          2025-12-23T10:37:06Z
    Message:                       DiskMaker: Available
    Status:                        True
    Type:                          DaemonSetsAvailable
    Last Transition Time:          2025-12-23T10:37:06Z
    Message:                       Operator reconciled successfully.
    Status:                        True
    Type:                          Available
  Observed Generation:             1
  Total Provisioned Device Count:  6
Events:
  Type    Reason               Age                 From                               Message
  ----    ------               ----                ----                               -------
  Normal  DiscoveredNewDevice  89s (x16 over 89s)  localvolumeset-symlink-controller  ceph11 - found possible matching disk, waiting 1m0s to claim
  Normal  DiscoveredNewDevice  88s (x19 over 88s)  localvolumeset-symlink-controller  ceph12 - found possible matching disk, waiting 1m0s to claim
  Normal  DiscoveredNewDevice  87s (x16 over 87s)  localvolumeset-symlink-controller  ceph10 - found possible matching disk, waiting 1m0s to claim

[rbo]

I don't know if this will cause trouble in the future...

[rbo]

With pull secret update the ceph nodes are rebooting.. #346 (comment)

[DanielFroehlich]

Hm, uptime still shows dates from last week, not sure if there was actually a reboot of the ceph nodes.
I am forcing one now...
... after reboot of ceph10, we are in trouble:
osd1 and osd4 not starting:

failed to reconcile CephCluster "openshift-storage/ocs-storagecluster-cephcluster". failed to reconcile cluster "ocs-storagecluster-cephcluster": failed to configure local ceph cluster: failed to create cluster: failed to start ceph osds: 2 failures encountered while running osds on nodes in namespace "openshift-storage". failed to update OSD 1: failed to generate config for OSD 1 on PVC "ocs-deviceset-local-odf-nvme-1-data-079scm": failed to find valid VolumeSource for PVC "ocs-deviceset-local-odf-nvme-1-data-079scm" failed to update OSD 4: failed to generate config for OSD 4 on PVC "ocs-deviceset-local-odf-nvme-2-data-1pg7qg": failed to find valid VolumeSource for PVC "ocs-deviceset-local-odf-nvme-2-data-1pg7qg"