Virtualized file server resilience
Particular embodiments provide a plurality of host machines, one or more virtual disks comprising a plurality of storage devices, and a virtualized file server (VFS) comprising a plurality of file server virtual machines (FSVMs), wherein each of the FSVMs is running on one of the host machines and conducts I/O transactions with the one or more virtual disks. A distributed health monitoring service (HMS) may be running on each of the host machines running a FSVM. The HMS may monitor the FSVMs to determine whether any of the FSVMs has failed or is having problems. The HMS may detect that a plurality of the FSVMs have failed, wherein the failed FSVMs form a portion of the FSVMs comprising the VFS. For each of the failed FSVMs, the HMS may reassign an IP address corresponding to the failed FSVM to a live one of the FSVMs.
1. A system for managing a virtualization environment, the system comprising:
a virtualized file server (VFS) comprising a plurality of file server virtual machines (FSVMs), wherein each of the FSVMs is running on one of a plurality of host machines configured to form a cluster, and wherein each of the FSVMs is configured to conduct I/O transactions with one or more virtual disks, wherein a health monitoring service is running on each of the host machines running a FSVM, and wherein the VFS is configured to:
by the distributed health monitoring service, monitor the and detect that a plurality of the FSVMs have failed; and
for each of the failed FSVMs, reassign an IP address corresponding to the failed FSVM to a live one of a plurality of live ones of the FSVMs.
2. The system of claim 1 , selecting, for at least one of the FSVMS, a live FSVM to which its corresponding IP address is assigned.
3. The system of claim 2 , wherein said selecting is based on an affinity of requests served by the failed FSVM with a configuration of the live FSVM, a protocol to which the requests conform, resources assigned to the host machine on which the live FSVM is running security restrictions applying to storage resources previously served by the failed FSVM, or combinations thereof.
4. The system of claim 1 , further comprising, for each of the failed FSVMs:
selecting the live FSVM to which its corresponding IP address is assigned based on a selected one of a plurality of failover paths defined for use when a FSVM fails within the cluster.
5. The system of claim 4 , further comprising, for each of the failed FSVMs:
selecting the live FSVM to which its corresponding IP address is assigned based on a previous primary failover path, a preferred failover path, or a failover path selected based on a round-robin ordering.
6. The system of claim 4 , further comprising, for each of the failed FSVMs:
selecting the live FSVM to which its corresponding IP address is assigned based on which of the live FSVMs has few outstanding or pending requests as compared to FSVMs associated with other failover paths.
7. The system of claim 4 , wherein a host machine associated with the selected failover path has low resource usage statistics as compared with other failover paths.
8. One or more computer-readable non-transitory storage media embodying software for managing a virtualization environment, the media comprising instructions operable when executed to perform operations comprising:
monitor a plurality of file server virtual machines (FSVMs) of a virtualized file server (VFS) hosted by a plurality of host machines configured to form a cluster;
detect that a plurality of the FSVMs have failed, wherein the failed FSVMs form a portion of the FSVMs comprising the VFS; and
for each of the failed FSVMs, reassign an IP address corresponding to the failed FSVM to a live one of a plurality of live ones of the FSVMs.
9. The media of claim 8 wherein the operations further comprise select the live FSVM to which its corresponding IP address is assigned.
10. The media of claim 9 , wherein the selection of the live FSVM is based on:
an affinity of requests served by the failed FSVM with a configuration of the live FSVM, a protocol to which the requests conform, resources assigned to the host machine on which the live FSVM is running; security restrictions applying to storage resources previously served by the failed FSVM, or combinations thereof.
11. The media of claim 8 , wherein the operations further comprise, for each of the failed FSVMs:
select the live FSVM to which its corresponding IP address is assigned based on a selected one of a plurality of failover paths defined for use when a FSVM fails within the cluster.
12. The media of claim 11 , wherein the operations further comprise, for each of the failed FSVMs:
select the live FSVM to which its corresponding IP address is assigned based on a previous primary failover path, a preferred failover path, or a failover path selected based on a round-robin ordering.
13. The media of claim 11 , wherein the operations further comprise, for each of the failed FSVMs:
select the live FSVM to which its corresponding IP address is assigned based on which of the live FSVMs has few outstanding or pending requests as compared to FSVMs associated with other failover paths.
14. The media of claim 11 , wherein a host machine associated with the selected failover path has low resource usage statistics as compared with other failover paths.
15. A method for managing a virtualization environment, the method comprising:
by one or more computing devices, providing a virtualized file server (VFS) comprising a plurality of file server virtual machines (FSVMs), wherein each of the FSVMs is running on one of a plurality of host machines and conducts I/O transactions with one or more virtual disks, wherein a health monitoring service is running on each of the host machines running a FSVM, and wherein the VFS is configured to:
by the distributed health monitoring service, monitor the FSVMs, detect that a plurality of the FSVMs have failed, wherein the failed FSVMs form a portion of the FSVMs comprising the VFS; and
for each of the failed FSVMs, reassign an IP address corresponding to the failed FSVM to a live one of a plurality of live ones of the FSVMs.
16. The method of claim 15 , wherein the VFS is further configured to, for each of the failed FSVMs select the live FSVM to which its corresponding IP address is assigned.
17. The method of claim 16 , wherein the selection of the live FSVM is based on an affinity of requests served by the failed FSVM with a configuration of the live FSVM, a protocol to which the requests conform, resources assigned to the host machine on which the live FSVM is running, security restrictions applying to storage resources previously served by the failed FSVM, or combinations thereof.
18. The method of claim 15 , wherein the VFS is further configured to, for each of the failed FSVMs:
select the live FSVM to which its corresponding IP address is assigned based on a selected one of a plurality of failover paths defined for use when a FSVM fails within the cluster.
19. The method of claim 18 , wherein the VFS is further configured to, for each of the failed FSVMs:
select the live FSVM to which its corresponding IP address is assigned based on a previous primary failover path, a preferred failover path, or a failover path selected based on a round-robin ordering.
20. The method of claim 19 , wherein the VFS is further configured to, for each of the failed FSVMs:
select the live FSVM to which its corresponding IP address is assigned based on which of the live FSVMs has few outstanding or pending requests as compared to FSVMs associated with other failover paths.