Automatic data replica manager in distributed caching and data processing systems

1. A method for enabling fault tolerance of a datacenter cluster caching system having one or more replica nodes in the same tier, the method comprising:

detecting a failure of the datacenter cluster;

performing load balancing among nodes of the datacenter cluster in accordance with the detected failure by moving data among storage tiers of the nodes; and

migrating replica data from a first replica node in the datacenter cluster to a second replica node in the datacenter cluster by:

writing new incoming data on the second replica node; and

maintaining unchanged cached data on the first replica node,

wherein the load balancing is performed without pausing a subsystem corresponding to the detected failure, and without copying an entirety of a dataset of a node of the datacenter cluster corresponding to the detected failure.

2. The method of claim 1 , further comprising:

evicting victim data from a solid-state drive of a primary node;

receiving the new incoming data at the primary node;

writing the new incoming data to a hard disk drive of the primary node; and

removing the victim data from a solid-state drive of the first replica node.

3. The method of claim 1 , wherein the detected failure comprises a virtual machine crash on a primary node of the datacenter cluster, the method further comprising:

writing dirty data from a solid-state drive of the primary node to a hard disk drive of the primary node;

maintaining the dirty data in a solid-state drive of a replica node;

flagging the dirty data in the solid-state drive of the replica node as nondirty;

restarting the crashed virtual machine on the primary node; and

forwarding incoming I/O requests to the solid-state drive of the primary node and to the solid-state drive of the replica node.

4. The method of claim 1 , wherein the detected failure comprises a cache device failure on a primary node of the datacenter cluster such that information cannot be written to the primary node, the method further comprising:

writing dirty data from a solid-state drive of a replica node of the datacenter cluster to a hard disk drive of the primary node;

maintaining the dirty data in the solid-state drive of the replica node;

flagging the dirty data as nondirty;

broadcasting the information to associated nodes of the datacenter cluster; and

writing back dirty data from respective solid-state drives of the associated nodes of the datacenter cluster to respective hard disk drives of the associated nodes.

5. The method of claim 4 , further comprising:

flagging data in the solid-state drives of the associated nodes with a nondirty flag;

replacing the solid-state drive on the primary node;

continuing to write incoming I/O requests to the solid-state drive of the primary node and the solid-state drive of the replica node; and

writing new replica data from the associated nodes to the solid-state drive of the primary node.

6. The method of claim 1 , wherein the detected failure comprises a cache device failure on the first replica node of the datacenter cluster, the method further comprising:

disconnecting a primary node of the datacenter cluster;

writing back dirty data from a solid-state drive of the primary node to a hard disk drive of the primary node;

finding a remote second replica node of the datacenter cluster;

storing the dirty data in a solid-state drive of the second replica node;

flagging the dirty data in the solid-state drive of the second replica node as nondirty; and

continuing to write incoming I/O requests to the solid-state drive of the primary node and the solid-state drive of the second replica node.

7. The method of claim 6 , wherein finding a remote second replica node of the datacenter cluster comprises using a dynamic evaluation process.

8. The method of claim 1 , wherein the detected failure comprises a communication failure between a primary node of the datacenter cluster and a first replica node of the datacenter cluster, the method further comprising:

writing back dirty data from a solid-state drive of the primary node to a hard disk drive of the primary node;

finding a second replica node to replace the first replica node;

writing I/O requests to a solid-state drive of the primary node and a solid-state drive of the second replica node; and

broadcasting to all nodes of the datacenter cluster a request to release all old replica data corresponding to the first replica node.

9. The method of claim 8 , wherein finding the second replica node comprises using a dynamic evaluation process.