High-availability cluster with proactive maintenance

1. A high-availability cluster apparatus, the apparatus comprising:

a plurality of computing nodes of said cluster;

a cluster master communicatively connected to each of the nodes;

non-volatile memory associated with the cluster master, wherein the non-volatile memory is configured to store data keeping track of quality measures of the nodes; and

computer-executable code of the cluster master configured to periodically execute a preventative maintenance task to test a least-recently tested active node so as to proactively maintain the cluster, wherein the preventative maintenance task is configured to swap out a least-recently-tested active node from the HA cluster and swap in a best-available spare node into the HA cluster, run functional tests on the least-recently-tested active node after that node is swapped out of the HA cluster such that the least-recently-tested node becomes a just-tested node, and restore a freshness measure of the just-tested node to a most fresh value and to change the status variable of the just-tested node to a spare status.

2. The apparatus of claim 1 , wherein the non-volatile memory is further configured to store data keeping track of freshness measures of the nodes.

3. The apparatus of claim 2 , wherein the cluster master is configured to keep track of status variables of the nodes.

4. The apparatus of claim 1 , wherein the cluster master comprises a separate computing system from the nodes of the cluster.

5. The apparatus of claim 1 , wherein the cluster master comprises an application that is distributed over the nodes of the cluster.

6. The apparatus of claim 1 , further comprising:

point-to-point links between input/output interfaces of the cluster master and each of the nodes.

7. The apparatus of claim 1 , further comprising:

a network interconnecting input/output interfaces of the cluster master and each of the nodes.

8. The apparatus of claim 3 , further comprising:

computer-executable code of the cluster master configured to periodically execute a gather task so as to determine the status variable and the quality and freshness measures for each node.

9. The apparatus of claim 8 , further comprising:

computer-executable code of the cluster master configured to periodically execute a quality check task so as to check the quality measure of each active node.

10. The apparatus of claim 9 , wherein the quality check task is configured to replace a poor-quality active node with a best-available spare node.

11. A method of pro-actively maintaining a high-availability (HA) cluster having a plurality of nodes, the method comprising:

keeping track of status variables for the nodes;

keeping track of quality measures for the nodes;

keeping track of freshness measures for the nodes; and

periodically executing a preventative maintenance task to test a least-recently-tested active node,

wherein the preventative maintenance task is configured to swap out the least-recently-tested active node from the HA cluster and swap in a best-available spare node into the HA cluster, run functional tests on the least-recently-tested active node after that node is swapped out of the HA cluster such that the least-recently-tested node becomes a just-tested node, and restore a freshness measure of the just-tested node to a most fresh value and to change the status variable of the just-tested node to a spare status.

12. The method of claim 11 , wherein the status variables, quality measures, and freshness measures are kept track of by periodically executing a gather data task by a cluster master.

13. The method of claim 11 , further comprising:

periodically executing a quality check task to maintain a quality level of active nodes in the HA cluster.

14. The method of claim 13 , wherein the quality check task is configured to replace a poor-quality active node with a best-available spare node.

15. The method of claim 11 , wherein the preventative maintenance task is configured to determine the least-recently-tested active node by utilizing the freshness measures.

16. The method of claim 11 , wherein the preventative maintenance task is further configured to change the status variable of the just-tested node to a repair status if the just-tested node failed the functional tests.

17. A method of preventative maintenance of a high-availability (HA) cluster, the method comprising:

determining a least-recently-tested active node;

swapping out the least-recently-tested active node from the HA cluster;

swapping in a stand-by node into the HA cluster;

executing functional tests on the least-recently-tested active node after that node is swapped out of the HA cluster such that the least-recently-tested node becomes a just-tested node;

restoring a freshness measure of the just-tested node to a most fresh value; and

changing a status variable of the just-tested node to a stand-by status.

18. The method of claim 17 , further comprising:

changing the status variable of the just-tested node to a repair status if the just-tested node failed the functional tests.