csiaddonsnode: Add retry with exponential backoff for connections

[black-dragon74]

This patch adds the functionality to retry for a maximum of
maxRetries to connect to the sidecar.

If the connection attempt is not successful, the object is considered
obsolete and is deleted.

The retry is tracked inside an annotation(connRetryAnnotation) and also
reflected in object's status.

These transient artifacts are cleaned up once a connection is
established.

[nixpanic]

What is the process to get a deleted CSIAddonsNode back in case of a longer network interruption? Can that be automated too?

[black-dragon74]

What is the process to get a deleted CSIAddonsNode back in case of a longer network interruption? Can that be automated too?

#765 takes care of such cases, if a CSIAddonsNode should exist, it will be ensured that it exists.

[Madhu-1]

Why do we need to extract or store the details in the annotation, cant we make use of the status/message?

[black-dragon74]

Why do we need to extract or store the details in the annotation, cant we make use of the status/message?

We could. A lot of manual parsing would be required without API changes (status messages should be human readable). Are there any downsides of having a transient annotation (keeps the changes simple)?

[nixpanic]

Why do we need to extract or store the details in the annotation, cant we make use of the status/message?

Hmm, yes, I agree that .Status.Conditions[] is cleaner for this.

[black-dragon74]

Why do we need to extract or store the details in the annotation, cant we make use of the status/message?

Hmm, yes, I agree that .Status.Conditions[] is cleaner for this.

Not that willing for API changes. What about the assumption that if status == retrying, message holds some form of info about the retry. A bit of manual parsing but can be done. WDYT?

[black-dragon74]

Test results

2025-12-15T10:39:27.331Z        INFO    Adding finalizer        {"controller": "csiaddonsnode", "controllerGroup": "csiaddons.openshift.io", "controllerKind": "CSIAddonsNode", "CSIAddonsNode": {"name":"test-it-out","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "test-it-out", "reconcileID": "b6a2c4df-fef2-46f8-bd73-5e8f81b94460", "NodeID": "minikube", "DriverName": "rook-ceph.rbd.csi.ceph.com", "EndPoint": ""}
2025-12-15T10:39:27.368Z        INFO    Connecting to sidecar   {"controller": "csiaddonsnode", "controllerGroup": "csiaddons.openshift.io", "controllerKind": "CSIAddonsNode", "CSIAddonsNode": {"name":"test-it-out","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "test-it-out", "reconcileID": "b6a2c4df-fef2-46f8-bd73-5e8f81b94460", "NodeID": "minikube", "DriverName": "rook-ceph.rbd.csi.ceph.com", "EndPoint": ""}
2025-12-15T10:39:27.379Z        ERROR   Failed to establish connection with sidecar     {"controller": "csiaddonsnode", "controllerGroup": "csiaddons.openshift.io", "controllerKind": "CSIAddonsNode", "CSIAddonsNode": {"name":"test-it-out","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "test-it-out", "reconcileID": "b6a2c4df-fef2-46f8-bd73-5e8f81b94460", "NodeID": "minikube", "DriverName": "rook-ceph.rbd.csi.ceph.com", "EndPoint": "", "attempt": 1, "error": "failed to exit idle mode: delegating_resolver: invalid target address \"\": missing address"}
...
2025-12-15T10:39:27.405Z        INFO    Requeuing request for attempting the connection again   {"controller": "csiaddonsnode", "controllerGroup": "csiaddons.openshift.io", "controllerKind": "CSIAddonsNode", "CSIAddonsNode": {"name":"test-it-out","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "test-it-out", "reconcileID": "b6a2c4df-fef2-46f8-bd73-5e8f81b94460", "NodeID": "minikube", "DriverName": "rook-ceph.rbd.csi.ceph.com", "EndPoint": "", "backoff": "2s"}
....
2025-12-15T10:39:41.655Z        INFO    Failed to establish connection with sidecar after 3 attempts, deleting the object       {"controller": "csiaddonsnode", "controllerGroup": "csiaddons.openshift.io", "controllerKind": "CSIAddonsNode", "CSIAddonsNode": {"name":"test-it-out","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "test-it-out", "reconcileID": "18c5fdde-df4c-4d95-93c8-767c928928ec", "NodeID": "minikube", "DriverName": "rook-ceph.rbd.csi.ceph.com", "EndPoint": ""}
2025-12-15T10:39:41.696Z        INFO    successfully deleted CSIAddonsNode object due to reaching max reconnection attempts     {"controller": "csiaddonsnode", "controllerGroup": "csiaddons.openshift.io", "controllerKind": "CSIAddonsNode", "CSIAddonsNode": {"name":"test-it-out","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "test-it-out", "reconcileID": "18c5fdde-df4c-4d95-93c8-767c928928ec", "NodeID": "minikube", "DriverName": "rook-ceph.rbd.csi.ceph.com", "EndPoint": ""}
2025-12-15T10:39:41.715Z        INFO    Deleting connection     {"controller": "csiaddonsnode", "controllerGroup": "csiaddons.openshift.io", "controllerKind": "CSIAddonsNode", "CSIAddonsNode": {"name":"test-it-out","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "test-it-out", "reconcileID": "257ae5a8-4401-4d30-8141-a6d45029d74a", "NodeID": "minikube", "DriverName": "rook-ceph.rbd.csi.ceph.com", "EndPoint": "", "Key": "rook-ceph/csi-rbdplugin-noent"}
2025-12-15T10:39:41.716Z        INFO    Removing finalizer      {"controller": "csiaddonsnode", "controllerGroup": "csiaddons.openshift.io", "controllerKind": "CSIAddonsNode", "CSIAddonsNode": {"name":"test-it-out","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "test-it-out", "reconcileID": "257ae5a8-4401-4d30-8141-a6d45029d74a", "NodeID": "minikube", "DriverName": "rook-ceph.rbd.csi.ceph.com", "EndPoint": ""}
2025-12-15T10:39:41.749Z        INFO    CSIAddonsNode resource not found        {"controller": "csiaddonsnode", "controllerGroup": "csiaddons.openshift.io", "controllerKind": "CSIAddonsNode", "CSIAddonsNode": {"name":"test-it-out","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "test-it-out", "reconcileID": "0d7228cb-6da9-4395-81e7-e758d89f402c"}

[Madhu-1]

I see only one problem, if the controller restarts multiple times we will delete the csiaddons node object immediately right?

[black-dragon74]

I see only one problem, if the controller restarts multiple times we will delete the csiaddons node object immediately right?

Even if we do that, the object will be recreated by the sidecar almost immediately. But we only delete and update the status if the actual connection attempt is unsuccessful.

Upon testing further, I am going to revert the change that returns an empty podname in case a pod is not found. If we keep it so, the reconciler will keep on retrying again and again and we will never reach the backoff section.

Better to let it try and fail than to keep retrying infinitely? WDYT?

@iPraveenParihar @Rakshith-R ^^

[Rakshith-R]

I see only one problem, if the controller restarts multiple times we will delete the csiaddons node object immediately right?

Even if we do that, the object will be recreated by the sidecar almost immediately. But we only delete and update the status if the actual connection attempt is unsuccessful.

Upon testing further, I am going to revert the change that returns an empty podname in case a pod is not found. If we keep it so, the reconciler will keep on retrying again and again and we will never reach the backoff section.

Better to let it try and fail than to keep retrying infinitely? WDYT?

@iPraveenParihar @Rakshith-R ^^

Let's move the entire retry logic to a helper method and call it at

• the current spot
• when podname emtpy too ?

[black-dragon74]

Let's move the entire retry logic to a helper method and call it at

• the current spot
• when podname emtpy too ?

I was mid refactor and then....

It's not that simple and would complicate things because, we need to add additional logic to decide between stopping the reconcile, requeueing after a backoff or returning an error. And we would need to do that at multiple places (two at-least).

A pod uniquely identifies a csiaddonsnode object and if that pod is not found (stale), it is not going to come back under normal circumstances. If it does come back, the sidecar will always create a csiaddonsnode for it.

Let's keep it simple; we have quantifiable number of worker nodes (out of which a select few will be in this state requiring cleanup) and we retry for 3 times only.

From my POV, it is worth retrying only when the connection fails (network can be flaky), not when something is out of the ordinary. If we want to be optimistic and assume that the pod WILL come back, let's keep the current code which will use its own retry-backoff and keep delaying the reconcile due to missing pod.