USHIFT-6519: Switch to using RHEL 9.8 in bootc presubmits and periodics

[ggiguash]

Also addresses USHIFT-6521

Summary by CodeRabbit

Release Notes

• Tests

  • Updated test infrastructure and scenarios to support RHEL 98 boot images across all presubmission test configurations.

• Chores

  • Adjusted package repository priority settings to ensure beta OpenShift packages are preferred over RHEL mirrors.
  • Added RHEL 9.8 early kernel repository configuration.
  • Adjusted logging timing in web server startup sequence.

[openshift-ci-robot]

@ggiguash: This pull request references USHIFT-6519 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.22.0" version, but no target version was set.

Details

In response to this:

Also addresses USHIFT-6521

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[openshift-ci-robot]

@ggiguash: This pull request references USHIFT-6519 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.22.0" version, but no target version was set.

Details

In response to this:

Also addresses USHIFT-6521

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[ggiguash]

/test ?

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ggiguash

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [ggiguash]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ggiguash]

/test e2e-aws-tests-bootc
/test e2e-aws-tests-bootc-arm
/test e2e-aws-tests-bootc-periodic
/test e2e-aws-tests-bootc-periodic-arm

[ggiguash]

/test e2e-aws-tests-bootc

[ggiguash]

/test e2e-aws-tests-bootc
/test e2e-aws-tests-bootc-arm
/test e2e-aws-tests-bootc-periodic
/test e2e-aws-tests-bootc-periodic-arm

[ggiguash]

/test e2e-aws-tests-bootc
/test e2e-aws-tests-bootc-arm
/test e2e-aws-tests-bootc-periodic
/test e2e-aws-tests-bootc-periodic-arm

[ggiguash]

/test e2e-aws-tests-bootc
/test e2e-aws-tests-bootc-arm
/test e2e-aws-tests-bootc-periodic
/test e2e-aws-tests-bootc-periodic-arm

[coderabbitai]

📝 Walkthrough

Walkthrough

This PR upgrades test infrastructure references from RHEL 96 to RHEL 98 across containerfiles, boot scenario scripts, and package repositories. Additionally, a web server startup log message is relocated to occur after VM shutdown completes.

Changes

Cohort / File(s)	Summary
Web Server Startup test/bin/manage_webserver.sh	Relocated startup log message to emit after action_stop completes and immediately before nginx launch.
Layer 1 Base Containerfiles test/image-blueprints-bootc/layer1-base/group2/rhel98-bootc-crel*.containerfile	Updated base image from rhel96-test-agent:latest to rhel98-test-agent:latest across three containerfiles.
Layer 2 Presubmit Containerfiles test/image-blueprints-bootc/layer2-presubmit/group1/rhel98-bootc-source*.containerfile, test/image-blueprints-bootc/layer2-presubmit/group2/rhel98-bootc-source-*.containerfile	Updated base image references from rhel96-bootc-source to rhel98-bootc-source across six containerfiles.
Layer 3 Periodic Containerfiles test/image-blueprints-bootc/layer3-periodic/group1/rhel98-bootc-source-*.containerfile, test/image-blueprints-bootc/layer3-periodic/group2/rhel98-bootc-source-gitops.containerfile	Updated base image from rhel96-bootc-source to rhel98-bootc-source across three containerfiles.
Layer 3 Image Boot Files test/image-blueprints-bootc/layer3-periodic/group2/rhel96-bootc-source-isolated.image-bootc, test/image-blueprints-bootc/layer3-periodic/group2/rhel98-bootc-source-*.image-bootc	Removed deprecated rhel96 image reference; added rhel98 image reference entry.
Presubmit Scenario Scripts (Version Upgrade) test/scenarios-bootc/presubmits/el96-prel@el98-src@upgrade-ok.sh, test/scenarios-bootc/presubmits/el98-base@el98-src@upgrade-ok.sh, test/scenarios-bootc/presubmits/el98-base@src@opt@delta-upgrade-ok.sh, test/scenarios-bootc/presubmits/el98-src@*.sh (19 files)	Updated VM boot sources, blueprints, and test references from rhel96-bootc-* to rhel98-bootc-* variants across all scenario setup and execution steps.
Package Source Repositories test/package-sources-bootc/rhel98-mirror.repo, test/package-sources-bootc/microshift-rhocp-y*.repo	Added new repository stanza for rhel-9.8 early kernel; added priority=50 directive to beta OpenShift repositories to ensure package precedence over RHEL mirrors.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 32.65% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and accurately describes the main change: updating RHEL 9.6 to 9.8 across bootc presubmit and periodic test configurations.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

• 📝 Generate docstrings (stacked PR)
• 📝 Generate docstrings (commit on current branch)

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[ggiguash]

/test e2e-aws-tests-bootc
/test e2e-aws-tests-bootc-arm
/test e2e-aws-tests-bootc-periodic
/test e2e-aws-tests-bootc-periodic-arm

[ggiguash]

@coderabbitai perform full review

[coderabbitai]

@ggiguash: I'll perform a full review of the PR changes.

✅ Actions performed

Full review triggered.

[openshift-ci-robot]

@ggiguash: This pull request references USHIFT-6519 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.22.0" version, but no target version was set.

Details

In response to this:

Also addresses USHIFT-6521

Summary by CodeRabbit

Release Notes

• Tests

• Added local testing framework configuration commands for composer, hypervisor, and cache setup.

• Introduced build targets for ostree and bootc images across presubmit, periodic, and release workflows.

• Updated testing infrastructure to support RHEL 9.8 images and scenarios.

• Chores

• Updated base container images from RHEL 9.6 to RHEL 9.8.

• Enhanced container build configurations with host networking support.

• Updated repository configurations for local testing environment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[coderabbitai]

Actionable comments posted: 2

🤖 Fix all issues with AI agents

In `@test/Makefile`:
- Around line 76-79: The Makefile declares PHONY: instead of the special .PHONY:
so the target test-build-ostree-presubmit is not marked as phony; change the
target declaration from PHONY to .PHONY and include test-build-ostree-presubmit
in that .PHONY list so Make treats test-build-ostree-presubmit as a phony target
rather than a real file.

In `@test/scenarios-bootc/presubmits/el96-prel`@el98-src@upgrade-ok.sh:
- Line 11: The boot blueprint is incorrect: the launch_vm call currently uses
--boot_blueprint rhel98-bootc but the scenario is el96-prel@el98-src and must
use the rhel96 boot blueprint; update the launch_vm invocation to use
--boot_blueprint rhel96-bootc (matching the start image) so the boot blueprint
aligns with the start image for the el96-prel scenario.

🧹 Nitpick comments (2)

test/Makefile (2)

63-65: find_layer may return multiple or zero matches silently.

If no directory matches (e.g., typo in token), the build command will run with an empty -l argument and fail with a confusing error. Consider adding a guard or using $(or $(call find_layer,...),$(error ...)).

---

52-58: Double-space typo on line 57.

There's an extra leading space/tab before ./bin/manage_build_cache.sh (the ⟶ on line 57 shows \t rather than just \t). In a bash -c heredoc this is harmless, but it looks unintentional.

[ggiguash]

/test all

[openshift-ci-robot]

@ggiguash: This pull request references USHIFT-6519 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.22.0" version, but no target version was set.

Details

In response to this:

Also addresses USHIFT-6521

Summary by CodeRabbit

Release Notes

• Tests
• Added local testing framework with new configuration, cache management, and build commands
• Updated test suite to use RHEL 9.8 base images across scenarios and containerfiles
• Enhanced testing infrastructure to support development workflows

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@test/Makefile`:
- Around line 63-65: The find_layer function silently returns empty or
multi-line values causing bad -l args; change its implementation (symbol:
find_layer) to validate the number of matches and fail loudly: run a shell
snippet that finds matching directories, splits by newline, counts them, and if
count != 1 prints a single canonical path to stdout but otherwise invokes make
error (or exits non‑zero) with a clear message; ensure consuming targets (e.g.
test-build-ostree-* and test-build-bootc-*) receive exactly one safe path or the
Makefile stops with an error.

[openshift-ci-robot]

@ggiguash: This pull request references USHIFT-6519 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.22.0" version, but no target version was set.

Details

In response to this:

Also addresses USHIFT-6521

Summary by CodeRabbit

• Tests
• Added local testing targets for configuring, caching, and building test artifacts to support local development.
• Migrated test scenarios and test images to RHEL 9.8 across the suite.
• Added a new mirror/repo entry to support RHEL 9.8 testing.
• Bug Fixes
• Adjusted webserver start/stop messaging to ensure stop completes before start logs and launch.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[ggiguash]

/test ?

[ggiguash]

/test e2e-aws-tests-cache
/test e2e-aws-tests-cache-arm

[openshift-ci]

@ggiguash: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-aws-tests-bootc-arm	3b55a49	link	true	/test e2e-aws-tests-bootc-arm
ci/prow/e2e-aws-tests-bootc-periodic-arm	3b55a49	link	true	/test e2e-aws-tests-bootc-periodic-arm
ci/prow/e2e-aws-tests-bootc	3b55a49	link	true	/test e2e-aws-tests-bootc
ci/prow/e2e-aws-tests-bootc-periodic	3b55a49	link	true	/test e2e-aws-tests-bootc-periodic

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[ggiguash]

/test e2e-aws-tests-cache
/test e2e-aws-tests-cache-arm

[ggiguash]

/test ?

[ggiguash]

/test e2e-aws-tests-bootc-release
/test e2e-aws-tests-bootc-release-arm

[openshift-ci-robot]

@ggiguash: This pull request references USHIFT-6519 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.22.0" version, but no target version was set.

Details

In response to this:

Also addresses USHIFT-6521

Summary by CodeRabbit

Release Notes

• Tests

• Updated test infrastructure and scenarios to support RHEL 98 boot images across all presubmission test configurations.

• Chores

• Adjusted package repository priority settings to ensure beta OpenShift packages are preferred over RHEL mirrors.

• Added RHEL 9.8 early kernel repository configuration.

• Adjusted logging timing in web server startup sequence.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[coderabbitai]

🧹 Nitpick comments (1)

test/image-blueprints-bootc/layer3-periodic/group1/rhel98-bootc-source-ai-model-serving.containerfile (1)

3-3: LGTM — rhel96 → rhel98 base image swap is correct.

Aligns with the PR objective and matches the companion group2/rhel98-bootc-source-ai-model-serving.image-bootc reference. No other changes needed here.

As a drive-by nit: using :latest means builds silently pick up whatever localhost/rhel98-bootc-source happens to be at build time. Pinning to a digest or a versioned tag would make builds reproducible — but since this is the established pattern across the whole layer, it's out of scope for this PR.

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In
`@test/image-blueprints-bootc/layer3-periodic/group1/rhel98-bootc-source-ai-model-serving.containerfile`
at line 3, Replace the floating image reference
"localhost/rhel98-bootc-source:latest" with a pinned identifier to make builds
reproducible: either use a specific versioned tag (e.g. :vX.Y.Z) or the image
digest (sha256:...) in the FROM line of
rhel98-bootc-source-ai-model-serving.containerfile so the build always uses the
exact same base image; leave the rest of the Dockerfile unchanged.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In
`@test/image-blueprints-bootc/layer3-periodic/group1/rhel98-bootc-source-ai-model-serving.containerfile`:
- Line 3: Replace the floating image reference
"localhost/rhel98-bootc-source:latest" with a pinned identifier to make builds
reproducible: either use a specific versioned tag (e.g. :vX.Y.Z) or the image
digest (sha256:...) in the FROM line of
rhel98-bootc-source-ai-model-serving.containerfile so the build always uses the
exact same base image; leave the rest of the Dockerfile unchanged.

---

ℹ️ Review info

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Cache: Disabled due to data retention organization setting

Knowledge base: Disabled due to data retention organization setting

📥 Commits

Reviewing files that changed from the base of the PR and between 3b55a49 and d511766.

📒 Files selected for processing (39)

• test/bin/manage_webserver.sh
• test/image-blueprints-bootc/layer1-base/group2/rhel98-bootc-crel-isolated.containerfile
• test/image-blueprints-bootc/layer1-base/group2/rhel98-bootc-crel-optionals.containerfile
• test/image-blueprints-bootc/layer1-base/group2/rhel98-bootc-crel.containerfile
• test/image-blueprints-bootc/layer2-presubmit/group1/rhel98-bootc-source-base.containerfile
• test/image-blueprints-bootc/layer2-presubmit/group1/rhel98-bootc-source.containerfile
• test/image-blueprints-bootc/layer2-presubmit/group2/rhel98-bootc-source-aux.containerfile
• test/image-blueprints-bootc/layer2-presubmit/group2/rhel98-bootc-source-fake-next-minor.containerfile
• test/image-blueprints-bootc/layer2-presubmit/group2/rhel98-bootc-source-fips.containerfile
• test/image-blueprints-bootc/layer2-presubmit/group2/rhel98-bootc-source-optionals.containerfile
• test/image-blueprints-bootc/layer3-periodic/group1/rhel98-bootc-source-ai-model-serving.containerfile
• test/image-blueprints-bootc/layer3-periodic/group1/rhel98-bootc-source-isolated.containerfile
• test/image-blueprints-bootc/layer3-periodic/group2/rhel96-bootc-source-isolated.image-bootc
• test/image-blueprints-bootc/layer3-periodic/group2/rhel98-bootc-source-ai-model-serving.image-bootc
• test/image-blueprints-bootc/layer3-periodic/group2/rhel98-bootc-source-gitops.containerfile
• test/image-blueprints-bootc/layer3-periodic/group2/rhel98-bootc-source-isolated.image-bootc
• test/package-sources-bootc/microshift-rhocp-y.repo
• test/package-sources-bootc/microshift-rhocp-y1.repo
• test/package-sources-bootc/rhel98-mirror.repo
• test/scenarios-bootc/presubmits/el96-prel@el98-src@upgrade-ok.sh
• test/scenarios-bootc/presubmits/el98-base@el98-src@upgrade-ok.sh
• test/scenarios-bootc/presubmits/el98-base@src@opt@delta-upgrade-ok.sh
• test/scenarios-bootc/presubmits/el98-src@ai-model-serving-online.sh
• test/scenarios-bootc/presubmits/el98-src@auto-recovery.sh
• test/scenarios-bootc/presubmits/el98-src@backup-and-restore-on-reboot.sh
• test/scenarios-bootc/presubmits/el98-src@backups.sh
• test/scenarios-bootc/presubmits/el98-src@configuration.sh
• test/scenarios-bootc/presubmits/el98-src@downgrade-block.sh
• test/scenarios-bootc/presubmits/el98-src@dual-stack.sh
• test/scenarios-bootc/presubmits/el98-src@el100-src@upgrade-ok.sh
• test/scenarios-bootc/presubmits/el98-src@ipv6.sh
• test/scenarios-bootc/presubmits/el98-src@multi-nic.sh
• test/scenarios-bootc/presubmits/el98-src@optional.sh
• test/scenarios-bootc/presubmits/el98-src@router.sh
• test/scenarios-bootc/presubmits/el98-src@standard-suite1.sh
• test/scenarios-bootc/presubmits/el98-src@standard-suite2.sh
• test/scenarios-bootc/presubmits/el98-src@storage.sh
• test/scenarios-bootc/presubmits/el98-src@upgrade-fails-then-recovers.sh
• test/scenarios-bootc/presubmits/el98-src@upgrade-fails.sh

💤 Files with no reviewable changes (1)

• test/image-blueprints-bootc/layer3-periodic/group2/rhel96-bootc-source-isolated.image-bootc

🚧 Files skipped from review as they are similar to previous changes (24)

• test/scenarios-bootc/presubmits/el98-src@multi-nic.sh
• test/image-blueprints-bootc/layer3-periodic/group1/rhel98-bootc-source-isolated.containerfile
• test/bin/manage_webserver.sh
• test/scenarios-bootc/presubmits/el98-src@backups.sh
• test/image-blueprints-bootc/layer2-presubmit/group2/rhel98-bootc-source-optionals.containerfile
• test/scenarios-bootc/presubmits/el98-base@src@opt@delta-upgrade-ok.sh
• test/image-blueprints-bootc/layer2-presubmit/group1/rhel98-bootc-source-base.containerfile
• test/scenarios-bootc/presubmits/el98-src@dual-stack.sh
• test/scenarios-bootc/presubmits/el98-src@downgrade-block.sh
• test/image-blueprints-bootc/layer2-presubmit/group2/rhel98-bootc-source-fake-next-minor.containerfile
• test/image-blueprints-bootc/layer1-base/group2/rhel98-bootc-crel-optionals.containerfile
• test/scenarios-bootc/presubmits/el98-src@ai-model-serving-online.sh
• test/image-blueprints-bootc/layer3-periodic/group2/rhel98-bootc-source-ai-model-serving.image-bootc
• test/image-blueprints-bootc/layer2-presubmit/group1/rhel98-bootc-source.containerfile
• test/scenarios-bootc/presubmits/el98-src@standard-suite1.sh
• test/scenarios-bootc/presubmits/el98-src@upgrade-fails.sh
• test/scenarios-bootc/presubmits/el98-src@optional.sh
• test/image-blueprints-bootc/layer3-periodic/group2/rhel98-bootc-source-gitops.containerfile
• test/scenarios-bootc/presubmits/el98-src@backup-and-restore-on-reboot.sh
• test/image-blueprints-bootc/layer2-presubmit/group2/rhel98-bootc-source-aux.containerfile
• test/image-blueprints-bootc/layer3-periodic/group2/rhel98-bootc-source-isolated.image-bootc
• test/scenarios-bootc/presubmits/el98-src@standard-suite2.sh
• test/image-blueprints-bootc/layer2-presubmit/group2/rhel98-bootc-source-fips.containerfile
• test/image-blueprints-bootc/layer1-base/group2/rhel98-bootc-crel-isolated.containerfile