Pooled memory heartbeat in shared memory architecture
Examples provide a pooled memory heartbeat for virtual machine hosts. A virtual controller creates a pooled memory heartbeat file system in a shared memory partition of a pooled memory. An agent running on each host in a plurality of virtual machine hosts updates a heartbeat file at an update time interval to lock the heartbeat file. The lock indicates the heartbeat status for a given host is active. A master agent accesses the shared memory partition to check the heartbeat status of each host in the pooled memory file system. The heartbeat status is used to determine whether a host that has lost pooled memory access, is network isolated, or failed. If the pooled memory heartbeat status for a given host indicates the host is a failed host, the set of virtual machines running on the given host are respawned on another healthier host.
1. A method for monitoring pooled memory heartbeats to determine host status, the method comprising:
checking a heartbeat file stored in at least one shared partition on a pooled memory, the heartbeat file associated with a selected host in a plurality of host computing devices;
on determining the checked heartbeat file is locked, identifying a pooled memory heartbeat status of the selected host as an active pooled memory heartbeat;
on determining the checked heartbeat file is unlocked, identifying the pooled memory heartbeat status as an inactive pooled memory heartbeat, the inactive pooled memory heartbeat indicates the selected host has lost connection to the pooled memory;
on determining a network heartbeat status for the selected host is active and the pooled memory heartbeat status is active, identifying the selected host as a normally operating host;
on determining the network heartbeat status of the selected host is inactive and the pooled memory heartbeat status is active, identifying the selected host as a network isolated host; and
on determining the network heartbeat status is inactive and the pooled memory heartbeat is inactive, identifying the selected host as a failed host.
2. The method of claim 1 , further comprising:
on determining a shared data store heartbeat status is active and the pooled memory heartbeat status is active, identifying the selected host as the normally operating host;
on determining the shared data store heartbeat status of the selected host is inactive and the pooled memory heartbeat status is active, identifying the selected host as a host having lost storage connectivity; and
on determining the shared data store heartbeat status is inactive and the pooled memory heartbeat is inactive, identifying the selected host as the failed host.
3. The method of claim 1 , further comprising:
on determining the network heartbeat status is active, a shared data store heartbeat status is active, and the pooled memory heartbeat status is active, identifying the selected host as the normally operating host; and
on determining the network heartbeat is inactive, the shared data store heartbeat is inactive, and the pooled memory heartbeat is inactive, identifying the selected host as the failed host and respawning a set of virtual machines associated with the failed host on a different host.
4. The method of claim 1 , further comprising:
checking a shared data store heartbeat and a network heartbeat of the selected host to determine a shared data store heartbeat status and the network heartbeat status of the selected host;
on determining the network heartbeat status is inactive, the pooled memory heartbeat status is active, and the shared data store heartbeat status is active, identifying the selected host as the network isolated host; and
on determining the network heartbeat status is inactive, the pooled memory heartbeat status is active, and the shared data store heartbeat status is inactive, identifying the selected host as the network isolated host having lost storage connectivity.
5. The method of claim 1 , further comprising:
on determining the network heartbeat status is active, the pooled memory heartbeat status is active, and a shared data store heartbeat status is inactive, identifying the selected host as a host with lost storage connectivity;
on determining the network heartbeat status is active, the pooled memory heartbeat status is inactive, and the shared data store heartbeat status is active, identifying the selected host as a host with lost pooled memory;
on determining the network heartbeat status is active, the pooled memory heartbeat status is inactive, and the shared data store heartbeat status is inactive, identifying the selected host as a host with lost pooled memory and lost storage connectivity; and
on determining the network heartbeat is inactive, the pooled memory heartbeat status is active, and the shared data store heartbeat is inactive, identifying the selected host as the network isolated host with lost storage connectivity.
6. The method of claim 1 further comprising:
iteratively updating a pooled memory heartbeat file at an occurrence of an update time interval to lock the pooled memory heartbeat file, wherein a failure to update the pooled memory heartbeat file within the update time interval unlocks the pooled memory heartbeat file.
7. A computer system comprising:
a memory device providing pooled memory accessible by a plurality of hosts, the pooled memory comprising at least one shared partition;
a processor; and
a non-transitory computer readable medium having stored thereon program code for monitoring pooled memory heartbeats to determine host status, the program code causing the processor to:
check a heartbeat file stored in at least one shared partition on a pooled memory, the heartbeat file associated with a selected host in a plurality of host computing devices;
on determining the checked heartbeat file is locked, identify a pooled memory heartbeat status of the selected host as an active pooled memory heartbeat;
on determining the checked heartbeat file is unlocked, identify the pooled memory heartbeat status as an inactive pooled memory heartbeat, the inactive pooled memory heartbeat indicates the selected host has lost connection to the pooled memory;
identify the selected host as a normally operating host on determining a network heartbeat status for the selected host is active and the pooled memory heartbeat status is active;
identify the selected host as a network isolated host on determining the network heartbeat status of the selected host is inactive and the pooled memory heartbeat status is active; and
identify the selected host as a failed host on determining the network heartbeat status is inactive and the pooled memory heartbeat is inactive.
8. The computer system of claim 7 wherein the program code further causes the processor to:
identify the selected host as the normally operating host on determining a shared data store heartbeat status is active and the pooled memory heartbeat status is active;
identify the selected host as a host having lost storage connectivity on determining the shared data store heartbeat status of the selected host is inactive and the pooled memory heartbeat status is active; and
identify the selected host as the failed host on determining the shared data store heartbeat status is inactive and the pooled memory heartbeat is inactive.
9. The computer system of claim 7 wherein the program code further causes the processor to:
identify the selected host as the normally operating host on determining the network heartbeat status is active, a shared data store heartbeat status is active, and the pooled memory heartbeat status is active; and
identify the selected host as the failed host and respawning a set of virtual machines associated with the failed host on a different host on determining the network heartbeat is inactive, the shared data store heartbeat is inactive, and the pooled memory heartbeat is inactive.
10. The computer system of claim 7 wherein the program code further causes the processor to:
check a shared data store heartbeat and a network heartbeat of the selected host to determine a shared data store heartbeat status and the network heartbeat status of the selected host;
identify the selected host as the network isolated host on determining the network heartbeat status is inactive, the pooled memory heartbeat status is active, and the shared data store heartbeat status is active; and
identify the selected host as the network isolated host having lost storage connectivity on determining the network heartbeat status is inactive, the pooled memory heartbeat status is active, and the shared data store heartbeat status is inactive.
11. The computer system of claim 7 further comprising:
a shared data store heartbeat, wherein the program code further causes the processor to:
identify the selected host as a host with lost storage connectivity on determining the network heartbeat status is active, the pooled memory heartbeat status is active, and a shared data store heartbeat status is inactive;
identify the selected host as a host with lost pooled memory on determining the network heartbeat status is active, the pooled memory heartbeat status is inactive, and the shared data store heartbeat status is active;
identify the selected host as a host with lost pooled memory and lost storage connectivity on determining the network heartbeat status is active, the pooled memory heartbeat status is inactive, and the shared data store heartbeat status is inactive; and
identify the selected host as the network isolated host with lost storage connectivity on determining the network heartbeat is inactive, the pooled memory heartbeat status is active, and the shared data store heartbeat is inactive.
12. The computer system of claim 7 further comprising:
a pooled memory heartbeat file in a shared partition in the memory, wherein the program code further causes the processor to iteratively update the pooled memory heartbeat file at an occurrence of an update time interval to lock the pooled memory heartbeat file, wherein a failure to update the pooled memory heartbeat file within the update time interval unlocks the pooled memory heartbeat file.
13. A non-transitory computer readable storage medium having stored thereon program code executable by a first computer system at a first site, the program code embodying a method comprising:
checking a heartbeat file stored in at least one shared partition on a pooled memory, the heartbeat file associated with a selected host in a plurality of host computing devices;
identifying a pooled memory heartbeat status of the selected host as an active pooled memory heartbeat, on determining the checked heartbeat file is locked;
identifying the pooled memory heartbeat status as an inactive pooled memory heartbeat on determining the pooled memory heartbeat file is unlocked, the inactive pooled memory heartbeat indicates the selected host has lost connection to the pooled memory;
on determining a network heartbeat status for the selected host is active and the pooled memory heartbeat status is active, identifying the selected host as a normally operating host;
on determining the network heartbeat status of the selected host is inactive and the pooled memory heartbeat status is active, identifying the selected host as a network isolated host; and
on determining the network heartbeat status is inactive and the pooled memory heartbeat is inactive, identifying the selected host as a failed host.
14. The non-transitory computer readable storage medium of claim 13 , wherein the program code embodying the method further comprises:
on determining a shared data store heartbeat status is active and the pooled memory heartbeat status is active, identifying the selected host as the normally operating host;
on determining the shared data store heartbeat status of the selected host is inactive and the pooled memory heartbeat status is active, identifying the selected host as a host having lost storage connectivity; and
on determining the shared data store heartbeat status is inactive and the pooled memory heartbeat is inactive, identifying the selected host as the failed host.
15. The non-transitory computer readable storage medium of claim 13 , wherein the program code embodying the method further comprises:
on determining the network heartbeat status is active, a shared data store heartbeat status is active, and the pooled memory heartbeat status is active, identifying the selected host as the normally operating host; and
on determining the network heartbeat is inactive, the shared data store heartbeat is inactive, and the pooled memory heartbeat is inactive, identifying the selected host as the failed host and respawning a set of virtual machines associated with the failed host on a different host.
16. The non-transitory computer readable storage medium of claim 13 , wherein the program code embodying the method further comprises:
checking a shared data store heartbeat and a network heartbeat of the selected host to determine a shared data store heartbeat status and the network heartbeat status of the selected host;
on determining the network heartbeat status is inactive, the pooled memory heartbeat status is active, and the shared data store heartbeat status is active, identifying the selected host as the network isolated host; and
on determining the network heartbeat status is inactive, the pooled memory heartbeat status is active, and the shared data store heartbeat status is inactive, identifying the selected host as the network isolated host having lost storage connectivity.
17. The non-transitory computer readable storage medium of claim 13 , wherein the program code embodying the method further comprises:
iteratively updating a pooled memory heartbeat file at an occurrence of an update time interval to lock the pooled memory heartbeat file, wherein a failure to update the pooled memory heartbeat file within the update time interval unlocks the pooled memory heartbeat file.