Add the steps to reboot the computes after update. #2587
Conversation
|
Thanks for the PR! ❤️ |
sathlan
force-pushed
the
update-reboot
branch
from
09bc6f0 to
0641dd5
Compare
|
Current tested with ping test running in the background and found not loss of connectivity. |
|
Build failed (check pipeline). Post https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/4f12a60d8dc64dad92d4f7b5f5bec990 ✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 40m 14s |
sathlan
force-pushed
the
update-reboot
branch
from
0641dd5 to
a22e794
Compare
|
Build failed (check pipeline). Post https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/bf06493eaeea430c898bf25520dfdd04 ✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 44m 32s |
sathlan
force-pushed
the
update-reboot
branch
from
a22e794 to
4a5a0f2
Compare
|
Build failed (check pipeline). Post https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/caf50b2e762a4aaeb3326c9399dffd15 ✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 21m 02s |
sathlan
force-pushed
the
update-reboot
branch
11 times, most recently
from
c056d3c to
85e4367
Compare
sathlan
added
enhancement
sathlan
force-pushed
the
update-reboot
branch
from
442ab42 to
39a32d4
Compare
sathlan
force-pushed
the
update-reboot
branch
from
39a32d4 to
ef7112a
Compare
sathlan
force-pushed
the
update-reboot
branch
from
ef7112a to
ff86b85
Compare
|
Build failed (check pipeline). Post https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/0030f67185a246beaa6641ac614816e3 ✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 43m 19s |
|
Recheck |
sathlan
force-pushed
the
update-reboot
branch
from
ff86b85 to
c4edb86
Compare
|
Build failed (check pipeline). Post https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/f6409174cbdc4deba2df5666caf90894 ✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 25m 38s |
|
LGTM |
|
recheck |
This sequence implements reboot of the compute nodes after the update. By default it's not run and `cifmw_update_reboot_test` must be set to true to activate it. We have one instance created. If the hypervisor being rebooted has the instance that instance will be live-migrated to another hypervisor before the reboot and migrated back to that original hypervisor after the reboot. Some basic sanity checks are performed after the reboot and before the migration back to ensure that the necessary services are up and running. During the reboot we start two scripts. One monitors and log the reboot of the hypervisors. The other log where the instance is currently running. The log files can be found in `~/ci-framework-data/tests/update/` in `monitor_servers.log` and `monitor_vm_placement.log` respectively. A note about node evacuation. We are still using node evaction from the nova cli. This command has not been ported to the openstack cli. There's a discussion about it [on launchpad](https://bugs.launchpad.net/python-openstackclient/+bug/2055552). Also, we do the evacuation only if there are more than one hypervisor available. When only one compute is available we stop and and after reboot, we just restart the instance. The official documentation mention only the live-migration path, but as we also use the live-migration in the test sequence that part is covered. We still expect customer to use the nova cli as it's way more user friendly and is still currently working. Closes: https://issues.redhat.com/browse/OSPRH-8937
sathlan
force-pushed
the
update-reboot
branch
from
c4edb86 to
0319a88
Compare
|
/lgtm |
|
Build failed (check pipeline). Post https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/70768976357c4ef3aa9e64c478f55901 ✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 47m 51s |
|
recheck cifmw-pod-ansible-test |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: jistr, pablintino The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
/cherry-pick 18.0-fr1 |
|
@sathlan: new pull request created: #2702 DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
5 participants
This sequence implements reboot of the compute nodes after the
update. By default it's not run and
cifmw_update_reboot_testmust beset to true to activate it.
We have one instance created. If the hypervisor being rebooted has
the instance that instance will be live-migrated to another hypervisor
before the reboot and migrated back to that original hypervisor after
the reboot.
Some basic sanity checks are performed after the reboot and before the
migration back to ensure that the necessary services are up and
running.
During the reboot we start two scripts. One monitors and log the
reboot of the hypervisors. The other log where the instance is
currently running. The log files can be found in
~/ci-framework-data/tests/update/inmonitor_servers.logandmonitor_vm_placement.logrespectively.A note about node evacuation. We are still using node evaction from
the nova cli. This command has not been ported to the openstack
cli. There's a discussion about it on launchpad.
Also, we do the evacuation only if there are more than one hypervisor
available. When only one compute is available we stop and and after
reboot, we just restart the instance.
The official documentation mention only the live-migration path, but
as we also use the live-migration in the test sequence that part is
covered. We still expect customer to use the nova cli as it's way
more user friendly and is still currently working.
Closes: https://issues.redhat.com/browse/OSPRH-8937