OCPEDGE-2166: test(two-node): stablize tnf recovery suite by jaypoulz · Pull Request #30716 · openshift/origin

jaypoulz · 2026-01-22T20:27:25Z

Individual recovery tests now skip when cluster preconditions aren't met, but an AfterSuite hook ensures the suite fails with diagnostic information about which tests were skipped and why. This makes precondition failures visible to CI analysis services while maintaining test stability.

openshift-ci-robot · 2026-01-22T20:27:29Z

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: automatic mode

openshift-ci-robot · 2026-01-22T20:27:31Z

@jaypoulz: This pull request references OCPEDGE-2011 which is a valid jira issue.

Details

In response to this:

Replace skip behavior with fail for cluster health checks to make precondition failures visible to test analysis services. Consolidate health check functions and use consistent 5-minute timeouts for pre-checks across recovery and node replacement tests.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

eggfoobar · 2026-01-22T20:41:26Z

/payload-job periodic-ci-openshift-release-master-nightly-4.22-e2e-metal-ovn-two-node-fencing-recovery-techpreview

openshift-ci · 2026-01-22T20:41:31Z

@eggfoobar: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

periodic-ci-openshift-release-master-nightly-4.22-e2e-metal-ovn-two-node-fencing-recovery-techpreview

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/b98661b0-f7d2-11f0-965f-3eb360421fed-0

eggfoobar · 2026-01-22T21:09:35Z

/test verify

Checking if fluke

openshift-ci-robot · 2026-01-22T21:54:05Z

Scheduling required tests:
/test e2e-aws-csi
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi

eggfoobar · 2026-01-22T23:16:20Z

/payload-job pull-ci-openshift-release-master-nightly-4.22-e2e-metal-ovn-two-node-fencing-recovery-techpreview

openshift-ci · 2026-01-22T23:16:25Z

@eggfoobar: trigger 0 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

eggfoobar · 2026-01-22T23:16:50Z

/payload-job periodic-ci-openshift-release-master-nightly-4.22-e2e-metal-ovn-two-node-fencing-recovery-techpreview

openshift-ci · 2026-01-22T23:16:53Z

@eggfoobar: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

periodic-ci-openshift-release-master-nightly-4.22-e2e-metal-ovn-two-node-fencing-recovery-techpreview

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/6edd8510-f7e8-11f0-9c54-2a9b6caa1fe3-0

clobrano

I just left one comment, otherwise lgtm

test/extended/two_node/tnf_node_replacement.go

jaypoulz · 2026-01-26T12:20:19Z

/hold
I've thought of a different way to do this

openshift-ci-robot · 2026-01-26T14:21:34Z

@jaypoulz: This pull request references OCPEDGE-2011 which is a valid jira issue.

Details

In response to this:

Individual recovery tests now skip when cluster preconditions aren't met, but an AfterSuite hook ensures the suite fails with diagnostic information about which tests were skipped and why. This makes precondition failures visible to CI analysis services while maintaining test stability.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

jaypoulz · 2026-01-26T14:21:46Z

/payload-job periodic-ci-openshift-release-master-nightly-4.22-e2e-metal-ovn-two-node-fencing-recovery-techpreview

openshift-ci · 2026-01-26T14:22:06Z

@jaypoulz: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

periodic-ci-openshift-release-master-nightly-4.22-e2e-metal-ovn-two-node-fencing-recovery-techpreview

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/590785b0-fac2-11f0-93aa-392383318691-0

clobrano

Great rework, I like the idea to track skipped tests!

/lgtm

jaypoulz · 2026-01-27T13:54:35Z

/payload-job periodic-ci-openshift-release-master-nightly-4.22-e2e-metal-ovn-two-node-fencing-recovery-techpreview

openshift-ci · 2026-01-27T13:54:43Z

@jaypoulz: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

periodic-ci-openshift-release-master-nightly-4.22-e2e-metal-ovn-two-node-fencing-recovery-techpreview

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/b77b6740-fb87-11f0-8719-0016cab4e65f-0

jaypoulz · 2026-02-04T16:05:55Z

/unhold

openshift-ci-robot · 2026-02-04T16:12:32Z

Scheduling required tests:
/test e2e-aws-csi
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi

jaypoulz · 2026-02-04T18:59:50Z

/retest-required

… precondition failures - Increase precondition health check timeout from 10 to 15 minutes - Add SetupCompleted flag to short-circuit AfterEach when BeforeEach skips - Apply pattern to both tnf_node_replacement.go and tnf_kubelet_disruption.go This prevents wasteful cleanup operations when tests skip due to precondition failures (e.g., unhealthy cluster operators still progressing). Co-authored-by: Cursor <cursoragent@cursor.com>

jaypoulz · 2026-02-04T20:32:52Z

/payload-job periodic-ci-openshift-release-master-nightly-4.22-e2e-metal-ovn-two-node-fencing-recovery-techpreview periodic-ci-openshift-release-master-nightly-4.21-e2e-metal-ovn-two-node-fencing-recovery-techpreview

openshift-ci · 2026-02-04T20:33:01Z

@jaypoulz: trigger 2 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

periodic-ci-openshift-release-master-nightly-4.22-e2e-metal-ovn-two-node-fencing-recovery-techpreview
periodic-ci-openshift-release-master-nightly-4.21-e2e-metal-ovn-two-node-fencing-recovery-techpreview

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/ae1c9a20-0208-11f1-898d-a20aec5e3119-0

xueqzhan · 2026-02-04T20:55:30Z

/approve
/hold

openshift-ci-robot · 2026-02-04T20:56:28Z

Scheduling required tests:
/test e2e-aws-csi
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi

openshift-ci-robot · 2026-02-04T22:03:32Z

@jaypoulz: This pull request references OCPEDGE-2166 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.22.0" version, but no target version was set.

Details

In response to this:

Individual recovery tests now skip when cluster preconditions aren't met, but an AfterSuite hook ensures the suite fails with diagnostic information about which tests were skipped and why. This makes precondition failures visible to CI analysis services while maintaining test stability.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

jaypoulz · 2026-02-04T23:58:22Z

/retest-required

jaypoulz · 2026-02-05T01:58:34Z

/verified by https://pr-payload-tests.ci.openshift.org/runs/ci/ae1c9a20-0208-11f1-898d-a20aec5e3119-0 and https://pr-payload-tests.ci.openshift.org/runs/ci/34c776c0-01e2-11f1-9f85-431cf25d2b94-0
See https://prow.ci.openshift.org/view/gs/test-platform-results/logs/openshift-origin-30716-nightly-4.22-e2e-metal-ovn-two-node-fencing-recovery-techpreview/2019147191737651200 for a no-skip run.
See https://prow.ci.openshift.org/view/gs/test-platform-results/logs/openshift-origin-30716-nightly-4.21-e2e-metal-ovn-two-node-fencing-recovery-techpreview/2019147192312270848 for a perfect run (except for one bugged test that will be resolved outside of this PR)
See https://prow.ci.openshift.org/view/gs/test-platform-results/logs/openshift-origin-30716-nightly-4.22-e2e-metal-ovn-two-node-fencing-recovery-techpreview/2019077922362494976 for what a pre-skip validation looks like when it skips a test.

openshift-ci-robot · 2026-02-05T01:58:46Z

@jaypoulz: This PR has been marked as verified by https://pr-payload-tests.ci.openshift.org/runs/ci/ae1c9a20-0208-11f1-898d-a20aec5e3119-0 and https://pr-payload-tests.ci.openshift.org/runs/ci/34c776c0-01e2-11f1-9f85-431cf25d2b94-0.

Details

In response to this:

/verified by https://pr-payload-tests.ci.openshift.org/runs/ci/ae1c9a20-0208-11f1-898d-a20aec5e3119-0 and https://pr-payload-tests.ci.openshift.org/runs/ci/34c776c0-01e2-11f1-9f85-431cf25d2b94-0
See https://prow.ci.openshift.org/view/gs/test-platform-results/logs/openshift-origin-30716-nightly-4.22-e2e-metal-ovn-two-node-fencing-recovery-techpreview/2019147191737651200 for a no-skip run.
See https://prow.ci.openshift.org/view/gs/test-platform-results/logs/openshift-origin-30716-nightly-4.21-e2e-metal-ovn-two-node-fencing-recovery-techpreview/2019147192312270848 for a perfect run (except for one bugged test that will be resolved outside of this PR)
See https://prow.ci.openshift.org/view/gs/test-platform-results/logs/openshift-origin-30716-nightly-4.22-e2e-metal-ovn-two-node-fencing-recovery-techpreview/2019077922362494976 for what a pre-skip validation looks like when it skips a test.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

eggfoobar · 2026-02-05T02:08:40Z

/lgtm

openshift-ci · 2026-02-05T02:09:03Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: clobrano, eggfoobar, jaypoulz, xueqzhan

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [xueqzhan]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-trt · 2026-02-05T03:01:10Z

Risk analysis has seen new tests most likely introduced by this PR.
Please ensure that new tests meet guidelines for naming and stability.

New tests seen in this PR at sha: 2964e64

"[openshift-tests] cluster precondition validation" [Total: 7, Pass: 7, Fail: 0, Flake: 0]

openshift-trt · 2026-02-05T03:16:02Z

Risk analysis has seen new tests most likely introduced by this PR.
Please ensure that new tests meet guidelines for naming and stability.

New tests seen in this PR at sha: 2964e64

"[openshift-tests] cluster precondition validation" [Total: 7, Pass: 7, Fail: 0, Flake: 0]

jaypoulz · 2026-02-05T11:51:52Z

/retest-required

jaypoulz · 2026-02-05T12:32:39Z

/unhold

openshift-trt · 2026-02-05T15:23:28Z

Risk analysis has seen new tests most likely introduced by this PR.
Please ensure that new tests meet guidelines for naming and stability.

New Test Risks for sha: 2964e64

Job Name	New Test Risk
pull-ci-openshift-origin-main-e2e-aws-ovn-fips	High - "[openshift-tests] cluster precondition validation" is a new test that was not present in all runs against the current commit.

New tests seen in this PR at sha: 2964e64

"[openshift-tests] cluster precondition validation" [Total: 8, Pass: 8, Fail: 0, Flake: 0]

openshift-ci · 2026-02-05T15:42:50Z

@jaypoulz: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jan 22, 2026

openshift-ci bot requested review from eggfoobar and qJkee January 22, 2026 20:29

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 22, 2026

jaypoulz force-pushed the OCPEDGE-2011-tnf-node-replacement-updates branch from fca0530 to 447b67f Compare January 22, 2026 21:26

clobrano suggested changes Jan 23, 2026

View reviewed changes

test/extended/two_node/tnf_node_replacement.go Outdated Show resolved Hide resolved

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 26, 2026

jaypoulz force-pushed the OCPEDGE-2011-tnf-node-replacement-updates branch from b2eb821 to cd9d035 Compare January 26, 2026 14:20

jaypoulz changed the title ~~OCPEDGE-2011: test(two-node): fail tests on unhealthy cluster preconditions~~ OCPEDGE-2011: test(two-node): track precondition skips and fail suite on cluster health issues Jan 26, 2026

clobrano approved these changes Jan 27, 2026

View reviewed changes

openshift-ci bot assigned clobrano Jan 27, 2026

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Jan 27, 2026

jaypoulz force-pushed the OCPEDGE-2011-tnf-node-replacement-updates branch from cd9d035 to c387d04 Compare January 27, 2026 13:49

openshift-ci bot removed the lgtm Indicates that a PR is ready to be merged. label Jan 27, 2026

openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Feb 4, 2026

jaypoulz force-pushed the OCPEDGE-2011-tnf-node-replacement-updates branch from 5b15323 to 2964e64 Compare February 4, 2026 20:32

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Feb 4, 2026

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 4, 2026

jaypoulz changed the title ~~OCPEDGE-2011: test(two-node): stablize tnf recovery suite~~ OCPEDGE-2166: test(two-node): stablize tnf recovery suite Feb 4, 2026

openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Feb 5, 2026

openshift-ci bot assigned eggfoobar Feb 5, 2026

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Feb 5, 2026

openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Feb 5, 2026

openshift-merge-bot bot merged commit 0095365 into openshift:main Feb 5, 2026
20 checks passed

Conversation

jaypoulz commented Jan 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci-robot commented Jan 22, 2026

Uh oh!

openshift-ci-robot commented Jan 22, 2026 • edited by openshift-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eggfoobar commented Jan 22, 2026

Uh oh!

openshift-ci bot commented Jan 22, 2026

Uh oh!

eggfoobar commented Jan 22, 2026

Uh oh!

openshift-ci-robot commented Jan 22, 2026

Uh oh!

eggfoobar commented Jan 22, 2026

Uh oh!

openshift-ci bot commented Jan 22, 2026

Uh oh!

eggfoobar commented Jan 22, 2026

Uh oh!

openshift-ci bot commented Jan 22, 2026

Uh oh!

clobrano left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jaypoulz commented Jan 26, 2026

Uh oh!

openshift-ci-robot commented Jan 26, 2026 • edited by openshift-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jaypoulz commented Jan 26, 2026

Uh oh!

openshift-ci bot commented Jan 26, 2026

Uh oh!

clobrano left a comment

Choose a reason for hiding this comment

Uh oh!

jaypoulz commented Jan 27, 2026

Uh oh!

openshift-ci bot commented Jan 27, 2026

Uh oh!

jaypoulz commented Feb 4, 2026

Uh oh!

openshift-ci-robot commented Feb 4, 2026

Uh oh!

jaypoulz commented Feb 4, 2026

Uh oh!

jaypoulz commented Feb 4, 2026

Uh oh!

openshift-ci bot commented Feb 4, 2026

Uh oh!

xueqzhan commented Feb 4, 2026

Uh oh!

openshift-ci-robot commented Feb 4, 2026

Uh oh!

openshift-ci-robot commented Feb 4, 2026 • edited by openshift-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jaypoulz commented Feb 4, 2026

Uh oh!

jaypoulz commented Feb 5, 2026

Uh oh!

openshift-ci-robot commented Feb 5, 2026

Uh oh!

eggfoobar commented Feb 5, 2026

Uh oh!

openshift-ci bot commented Feb 5, 2026

Uh oh!

openshift-trt bot commented Feb 5, 2026

Uh oh!

openshift-trt bot commented Feb 5, 2026

Uh oh!

jaypoulz commented Feb 5, 2026

Uh oh!

jaypoulz commented Feb 5, 2026

Uh oh!

jaypoulz commented Jan 22, 2026 •

edited

Loading

openshift-ci-robot commented Jan 22, 2026 •

edited by openshift-ci bot

Loading

openshift-ci-robot commented Jan 26, 2026 •

edited by openshift-ci bot

Loading

openshift-ci-robot commented Feb 4, 2026 •

edited by openshift-ci bot

Loading