Method and system for data recovery in a data system
The present teaching relates to request management and data recovery in a data system. In one example, a failure in connection with first data is detected at a first node in a data system. Information associated with a most recent transaction related to the first node is obtained from the persistent storage. Each of other nodes in the data system is requested to transmit one or more transaction requests previously sent to the first node after the most recent transaction. The one or more transaction requests are received from at least one of the other nodes. A sequence of one or more transactions associated with the one or more transaction requests is determined. The one or more transactions are executed according to the sequence in order to recover the first data at the first node.
1. A method, implemented on a machine having at least one processor, storage, and a communication platform connected to a network for data recovery at a first node in a data system comprising a plurality of nodes and a persistent storage, the method comprising:
detecting, by the first node, a failure in connection with first data;
obtaining, by the first node, information associated with a most recent transaction related to the first node from the persistent storage, wherein the most recent transaction is executed prior to the failure in connection with the first data;
requesting, by the first node and based on the information associated with the most recent transaction, each of the other nodes in the data system to transmit one or more transaction requests previously sent to the first node after the most recent transaction, wherein the one or more transaction requests are associated with the first data;
receiving, by the first node, from at least one of the other nodes, the one or more transaction requests;
determining, by the first node, a sequence of one or more transactions associated with the one or more transaction requests; and
executing, by the first node, the one or more transactions according to the sequence in order to recover the first data at the first node.
2. The method of claim 1 , wherein the obtaining comprises:
restoring a latest snapshot of a memory state of the first node from the persistent storage;
determining a first transaction associated with the latest snapshot;
retrieving, from the persistent storage, one or more second transactions based on the first transaction; and
executing the one or more second transactions, wherein the most recent transaction is a transaction executed last among the one or more second transactions.
3. The method of claim 2 , wherein:
the first transaction is a transaction last executed before the latest snapshot was taken;
the one or more second transactions were executed previously in an order following the first transaction; and
the one or more second transactions are executed according to the order.
4. The method of claim 1 , further comprising:
obtaining a request for a new transaction; and
holding the request until the first data at the first node is recovered.
5. The method of claim 1 , further comprising de-duplicating a transaction request among the one or more transaction requests when multiple copies of the transaction request are received.
6. A system having at least one processor, storage, and a communication platform connected to a network for data recovery at a first node in a data system comprising a plurality of nodes and a persistent storage, the system comprising:
a recovery initiator located in the first node and configured for detecting a failure in connection with first data;
a commit log retrieving unit located in the first node and configured for obtaining information associated with a most recent transaction related to the first node from the persistent storage, wherein the most recent transaction is executed prior to the failure in connection with the first data;
a buffered request demander located in the first node and configured for requesting, based on the information associated with the most recent transaction, each of the other nodes in the data system to transmit one or more transaction requests previously sent to the first node after the most recent transaction, wherein the one or more transaction requests are associated with the first data;
a buffered request receiver located in the first node and configured for receiving, from at least one of the other nodes, the one or more transaction requests;
a transaction sequence determiner located in the first node and configured for determining a sequence of one or more transactions associated with the one or more transaction requests; and
one or more partition execution engines located in the first node and configured for executing the one or more transactions according to the sequence in order to recover the first data at the first node.
7. The system of claim 6 , further comprising a snapshot restoring unit, located in the first node, configured for restoring a latest snapshot of a memory state of the first node from the persistent storage and determining a first transaction associated with the latest snapshot, wherein:
the commit log retrieving unit is further configured for retrieving, from the persistent storage, one or more second transactions based on the first transaction; and
the one or more partition execution engines are further configured for executing the one or more second transactions, wherein the most recent transaction is a transaction executed last among the one or more second transactions.
8. The system of claim 7 , wherein:
the first transaction is a transaction last executed before the latest snapshot was taken;
the one or more second transactions were executed previously in an order following the first transaction; and
the one or more second transactions are executed according to the order.
9. The system of claim 6 , wherein the transaction sequence determiner is further configured for de-duplicating a transaction request among the one or more transaction requests when multiple copies of the transaction request are received.
10. A machine-readable, non-transitory and tangible medium having information recorded thereon for data recovery at a first node in a data system comprising a plurality of nodes and a persistent storage, wherein the information, when read by the machine, causes the machine to perform the following:
detecting, by the first node, a failure in connection with first data;
obtaining, by the first node, information associated with a most recent transaction related to the first node from the persistent storage, wherein the most recent transaction is executed prior to the failure in connection with the first data;
requesting, by the first node and based on the information associated with the most recent transaction, each of the other nodes in the data system to transmit one or more transaction requests previously sent to the first node after the most recent transaction, wherein the one or more transaction requests are associated with the first data;
receiving, by the first node, from at least one of the other nodes, the one or more transaction requests;
determining by the first node, a sequence of one or more transactions associated with the one or more transaction requests; and
executing by the first node, the one or more transactions according to the sequence in order to recover the first data at the first node.