VM HA does not work when server is stopped gracefully

[weizhouapache]

When we stop a server (hypervisor) gracefully, vms on it will NOT be started on other servers even HA is enabled.

ISSUE TYPE

• Bug Report

COMPONENT NAME

VM HA

CLOUDSTACK VERSION

4.14/4.15

CONFIGURATION

advanced zone

OS / ENVIRONMENT

KVM

SUMMARY

STEPS TO REPRODUCE

1. create a service offering with HA enabled
2. deploy a vm
3. go to the server where vm is running on, and stop it by "poweroff" command

EXPECTED RESULTS

vm will be started on other servers.

ACTUAL RESULTS

vm is NOT started on other servers.

only when the server is Up again, cloudstack detects the vm is stopped out-of-band and starts it.

[weizhouapache]

@rhtyd @DaanHoogland @PaulAngus @svenvogel have you faced this issue ? is this an expected behavior ?

[DaanHoogland]

This doesn't sound familiar @weizhouapache . To recap,

• a hosts is HA enabled? (genuine question , here)
• a vm with a ha enabled offering is created on it.
• the server is powered off. Is by means of an remote (ipmi) action or by logging on to the console and issueing a halt command?

I would not expect this behaviour as a user but it might be a side effect of the host-ha, hence my questions.

[GabrielBrascher]

@weizhouapache adding to @DaanHoogland's questions:

• what is the primary storage used on the tests?

[weizhouapache]

@weizhouapache adding to @DaanHoogland's questions:

• what is the primary storage used on the tests?

@GabrielBrascher we use nfs as zone-wide storage pool.

[weizhouapache]

This doesn't sound familiar @weizhouapache . To recap,

• a hosts is HA enabled? (genuine question , here)
• a vm with a ha enabled offering is created on it.
• the server is powered off. Is by means of an remote (ipmi) action or by logging on to the console and issueing a halt command?

I would not expect this behaviour as a user but it might be a side effect of the host-ha, hence my questions.

@DaanHoogland

1. do you mean OOBM based ha ? no, we do not use it.
2. all vms are ha-enabled.

if kvm cloudstack-agent is stopped, there is no thread in cloudstack to determine its actual host state.

[RodrigoDLopez]

hi @weizhouapache,
My environment is similar to yours, apart from the storage pool that I use cluster-wide, on this environment I could see HA working on cloudstack version 4.13.1.1 with an NFS cluster-wide storage pool.

On 4.13.1.1 version I could see my VMs migrate to a suitable HOST after I force shut down the HOST where those VMs were, but with gracefully shut down I couldn't see the HA working, those VMs never migrate to another suitable HOST, only when this HOST came back the cloudstack detected this VMs as stopped. But at 4.14 and 4.15 version, I couldn't see HA working even when I force shut down, those VMs are never detected as stopped by cloudstack

[shwstppr]

@weizhouapache are planning/working on a fix for this?

[weizhouapache]

@shwstppr yeah. I will create a pr in coming days.

[weizhouapache]

this issue should have been fixed by #5164

for users who do not use NFS, there are two PRs open for review and testing:

KVM High Availability regardless of storage #4978
KVM HA Enhancements for Ceph RBD Primary Storage #5862