Managing high-availability file servers
Methods, systems and computer program products for implementing high-availability file services in a clustered computing environment. Two or more clusters are interconnected to carry out operations for replication of file content between file servers. The file servers and their respective network links are registered with a file server witness. The file servers operate in synchrony, where each file I/O is replicated from one file server to another file server over a first set of network paths. A file server witness communicates with each file server using a second set of two or more network paths interfaced with respective file servers. The file server witness monitors the file servers to determine operational health of the file servers. Upon receipt of a file I/O request, the file I/O request is directed to one of the two file servers based at least in part on the determined operational health.
1 . A method, comprising:
implementing, using a first virtual machine, a first file server in a first failure domain to provide file services;
implementing, using a second virtual machine, a second file server in a second failure domain to provide the file services, wherein a failure affecting the first failure domain does not affect the second failure domain;
generating monitoring results by respectively monitoring the first and the second file servers, wherein said monitoring is performed with at least one system monitor in at least one of the first and second failure domains, the at least one system monitor is external to the first and the second file servers, and the at least one system monitor comprises a first system monitor deployed in a first controller of a first cluster of the first failure domain and a second system monitor deployed in a second controller of a second cluster of the second failure domain; and
implementing a witness external to the first and second failure domains, wherein the witness is in communication with the at least one system monitor and each of the first and second controllers and executes an operation to select either the first file server or the second file server to serve as a primary file server after detecting an operational status of the first file server or the second file server from the monitoring results.
2 . The method of claim 1 , wherein the first file server in the first failure domain synchronously replicates data to the second file server in the second failure domain, or the first file server at the first failure domain and the second file server at the second failure domain store file services content in a distributed storage pool.
3 . The method of claim 1 , wherein the first file server in the first failure domain corresponds to a primary file server and is transitioned from a primary mode into a fail mode, and the second file server at the second failure domain corresponds to a secondary file server and is transitioned from a replication mode into the primary mode.
4 . The method of claim 3 , wherein the first file server at the first failure domain, after a remediation, is transitioned from the fail mode into the replication mode while the second file server at the second failure domain remains in the primary mode, the witness is in a third failure domain, and the witness is implemented as a third virtual machine.
5 . The method of claim 1 , wherein the witness is in a third failure domain external to the first failure domain and the second failure domain, wherein the witness receives the operational status from the at least one system monitor, the first virtual machine or the second virtual machine is of a first type of virtual machines, and the witness is implemented in a separate virtual machine of a second type that is different from the first type.
6 . The method of claim 1 , wherein the at least one system monitor receives a heartbeat corresponding to the first file server in the first failure domain, wherein the heartbeat is used to monitor availability of the first file server at the first failure domain.
7 . The method of claim 1 , wherein a plurality of file servers including the first file server in the first failure domain and the second file server in the second failure domain are registered with the witness, and the witness bridges one or more hosts with the plurality of file servers in multiple clusters to provide the file services to the one or more hosts.
8 . The method of claim 1 , wherein the first file server in the first failure domain is in a first cluster and the second file server in the second failure domain is in a second cluster.
9 . The method of claim 1 , wherein each of the first failure domain and the second failure domain respectively corresponds to a logical collection of hardware components that are commonly affected by a failure within the corresponding logical collection.
10 . The method of claim 1 , the at least one system monitor comprising a first system monitor and a second system monitor, wherein
the first failure domain comprises the first system monitor that is in communication with the witness,
the second failure domain comprises the second system monitor that is in communication with the witness,
the first system monitor of the first failure domain is external to and in communication with the first file server,
the second system monitor of the second failure domain is external to and in communication with the second file server, and
results are generated by respectively monitoring the first and second file servers with respective first and second system monitors.
11 . The method of claim 2 , wherein
the first failure domain comprises a first system monitor and the second failure domain comprises a second system monitor,
the first system monitor of the first failure domain is external to and in communication with the first file server and the witness, the first system monitor receiving a first heartbeat corresponding the first file server,
the second system monitor of the second failure domain is external to and in communication with the second file server and the witness, the second system monitor receiving a second heartbeat corresponding to the second file server, and
the witness executes an operation to select either the first file server or the second file server based at least in part on respective first and second heartbeats and respective availabilities of respective first and second file servers.
12 . The method of claim 1 , wherein the first controller is a virtual machine.
13 . The method of claim 1 , wherein the second controller is a virtual machine.
14 . A system, comprising:
a processor;
a memory for holding programmable code, wherein the programmable code includes instructions which, when executed by the processor, causes the processor to perform a set of acts, the set of acts comprising:
implementing, using a first virtual machine, a first file server in a first failure domain to provide file services;
implementing, using a second virtual machine, a second file server in a second failure domain to provide the file services, wherein a failure affecting the first failure domain does not affect the second failure domain;
generating monitoring results by respectively monitoring the first and the second file servers, wherein said monitoring is performed with at least one system monitor in at least one of the first and second domains, the at least one system monitor is external to the first and second file servers, and the at least one system monitor comprises a first system monitor deployed in a first controller of a first cluster of the first failure domain and a second system monitor deployed in a second controller of a second cluster of the second failure domain; and
implementing a witness external to the first and second failure domains, wherein the witness is in communication with the at least one system monitor and each of the first and second controllers and executes an operation to select either the first or the second file server to serve as a primary file server after detecting an operational status of the first file server or the second file server from the monitoring results.
15 . The system of claim 14 , wherein the first file server in the first failure domain synchronously replicates data to the second file server in the second failure domain, or the first file server at the first failure domain and the second file server at the second failure domain store file services content in a distributed storage pool.
16 . The system of claim 14 , wherein the first file server in the first failure domain corresponds to a primary file server and is transitioned from a primary mode into a fail mode, and the second file server in the second failure domain corresponds to a secondary file server and is transitioned from a replication mode into the primary mode.
17 . The system of claim 16 , wherein the first file server in the first failure domain, after a remediation, is transitioned from the fail mode into the replication mode while the second file server in the second failure domain remains in the primary mode, the witness is in a third failure domain, and the witness is implemented as a third virtual machine.
18 . The system of claim 14 , wherein the witness is in a third failure domain external to the first and second domains, wherein the witness receives the operational status from the at least one system monitor, the first virtual machine or the second virtual machine is of a first type of virtual machines, and the witness is implemented in a separate virtual machine of a second type that is different from the first type.
19 . The system of claim 14 , wherein the at least one system monitor receives a heartbeat corresponding to the first file server in the first failure domain, wherein the heartbeat is used to monitor availability of the first file server in the first failure domain.
20 . The system of claim 14 , wherein a plurality of file servers including the first file server in the first failure domain and the second file server in the second failure domain are registered with the witness, and the witness bridges one or more hosts with the plurality of file servers in multiple clusters to provide the file services to the one or more hosts.
21 . The system of claim 14 , wherein the first file server in the first failure domain is in a first cluster and the second file server in the second failure domain is in a second cluster.
22 . The system of claim 14 , wherein each of the first failure domain and the second failure domain respectively corresponds to a logical collection of hardware components that are commonly affected by a failure within the corresponding logical collection.
23 . The system of claim 14 , wherein the first controller is a virtual machine.
24 . The system of claim 14 , wherein the second controller is a virtual machine.
25 . The system of claim 14 , wherein the witness is in a third failure domain external to the first failure domain and the second failure domain.
26 . The system of claim 25 , wherein the first file server is in a first cluster in the first failure domain, the second file server is in a second cluster in the second failure domain, and the witness is in a third cluster in the third failure domain, wherein the third failure domain is separate from and external to the first failure domain and the second failure domain.
27 . The system of claim 14 , the at least one system monitor comprising a first system monitor and a second system monitor, wherein
the first failure domain comprises the first system monitor that is in communication with the witness,
the second failure domain comprises the second system monitor that is in communication with the witness,
the first system monitor of the first failure domain is external to and in communication with the first file server,
the second system monitor of the second failure domain is external to and in communication with the second file server, and
results are generated by respectively monitoring the first and second file servers with respective first and second system monitors.
28 . The system of claim 14 , wherein
the first failure domain comprises a first system monitor and the second failure domain comprises a second system monitor,
the first system monitor of the first failure domain is external to and in communication with the first file server and the witness, the first system monitor receiving a first heartbeat corresponding the first file server,
the second system monitor of the second failure domain is external to and in communication with the second file server and the witness, the second system monitor receiving a second heartbeat corresponding to the second file server, and
the witness executes an operation to select either the first file server or the second file server based at least in part on respective first and second heartbeats and respective availabilities of respective first and second file servers.
29 . A computer program product embodied on a non-transitory computer readable medium having stored thereon a sequence of instructions which, when executed by a processor, causes the processor to execute a set of acts, the set of acts comprising:
implementing, using a first virtual machine, a first file server in a first failure domain to provide file services;
implementing, using a first virtual machine, a second file server in a second failure domain to provide the file services, wherein a failure affecting the first failure domain does not affect the second failure domain;
generating monitoring results by respectively monitoring the first and the second file servers, wherein said monitoring is performed with at least one system monitor in at least one of the first and second failure domains, the at least one system monitor is external to the first and second file servers, and the at least one system monitor comprises a first system monitor deployed in a first controller of a first cluster of the first failure domain and a second system monitor deployed in a second controller of a second cluster of the second failure domain; and
implementing a witness external to the first and second failure domains, wherein the witness is in communication with the at least one system monitor and each of the first and second controllers and executes an operation to select either the first file server or the second file server to serve as a primary file server after detecting an operational status of the first file server or the second file server from the monitoring results.
30 . The computer program product of claim 29 , wherein the first file server in the first failure domain synchronously replicates data to the second file server in the second failure domain, or the first file server at the first site and the second file server in the second failure domain store file services content in a distributed storage pool.
31 . The computer program product of claim 29 , wherein the first file server in the first failure domain corresponds to a primary file server and is transitioned from a primary mode into a fail mode, and the second file server in the second failure domain corresponds to a secondary file server and is transitioned from a replication mode into the primary mode.
32 . The computer program product of claim 31 , wherein the first file server in the first failure domain, after a remediation, is transitioned from the fail mode into the replication mode while the second file server in the second remains in the primary mode, the witness is in a third failure domain, and the witness is implemented as a third virtual machine.
33 . The computer program product of claim 29 , wherein the witness is in a third failure domain external to the first and second domains, wherein the witness receives the operational status from the at least one system monitor, the first virtual machine or the second virtual machine is of a first type of virtual machines, and the witness is implemented in a third virtual machine of a second type that is different from the first type.
34 . The computer program product of claim 29 , wherein the at least one system monitor receives a heartbeat corresponding to the first file server in the first failure domain, wherein the heartbeat is used to monitor availability of the first file server in the first failure domain.
35 . The computer program product of claim 29 , wherein a plurality of file servers including the first file server in the first failure domain and the second file server in the second failure domain are registered with the witness, and the witness bridges one or more hosts with the plurality of file servers in multiple clusters to provide the file services to the one or more hosts.
36 . The computer program product of claim 29 , wherein the first file server in the first failure domain is in a first cluster and the second file server in the second failure domain is in a second cluster.
37 . The computer program product of claim 29 , wherein each of the first failure domain and the second failure domain respectively correspond to a logical collection of hardware components that are commonly affected by a failure within the corresponding logical collection.
38 . The method of claim 1 , wherein the witness is in a third failure domain external to the first failure domain and the second failure domain.
39 . The method of claim 38 , wherein the first file server is in a first cluster in the first failure domain, the second file server is in a second cluster in the second failure domain, and the witness is in a third cluster in the third failure domain, wherein the third failure domain is separate from and external to the first failure domain and the second failure domain.
40 . The computer program product of claim 29 , wherein the first controller is a virtual machine.
41 . The computer program product of claim 29 , wherein the second controller is a virtual machine.
42 . The computer program product of claim 29 , wherein the witness is in a third failure domain external to the first failure domain and the second failure domain.
43 . The computer program product of claim 42 , wherein the first file server is in a first cluster in the first failure domain, the second file server is in a second cluster in the second failure domain, and the witness is in a third cluster in the third failure domain, wherein the third failure domain is separate from and external to the first failure domain and the second failure domain.
44 . The computer program product of claim 29 , the at least one system monitor comprising a first system monitor and a second system monitor, wherein
the first failure domain comprises the first system monitor that is in communication with the witness,
the second failure domain comprises the second system monitor that is in communication with the witness,
the first system monitor of the first failure domain is external to and in communication with the first file server,
the second system monitor of the second failure domain is external to and in communication with the second file server, and
results are generated by respectively monitoring the first and second file servers with respective first and second system monitors.
45 . The computer program product of claim 29 , wherein
the first failure domain comprises a first system monitor and the second failure domain comprises a second system monitor,
the first system monitor of the first failure domain is external to and in communication with the first file server and the witness, the first system monitor receiving a first heartbeat corresponding the first file server,
the second system monitor of the second failure domain is external to and in communication with the second file server and the witness, the second system monitor receiving a second heartbeat corresponding to the second file server, and
the witness executes an operation to select either the first file server or the second file server based at least in part on respective first and second heartbeats and respective availabilities of respective first and second file servers.