Parallel restore of a large object
A request to restore an instance of an object that is stored in a storage associated with a cluster of nodes to a target destination is received. Data of the object is divided into a plurality of data portions. Corresponding data portions of the plurality of data portions are assigned to each node of a plurality of nodes of the cluster. It is determined that a first node of the cluster of nodes does not currently store in a storage associated with the first node at least a part of a corresponding data portion of the object assigned to the first node. At least the part of the corresponding data portion of the object assigned to the first node is received at the first node from a second node of the cluster of nodes. At least the part of the corresponding data portion of the object received from the second node is provided from the first node to the target destination.
1. A method, comprising:
analyzing, by a first node of a cluster of nodes, a first data portion of an object assigned to the first node;
determining, by the first node, that the first node does not currently store in a storage associated with the first node at least a part of the first data portion of the object assigned to the first node;
providing from the first node to a second node of the cluster a request to provide one or more data chunks included in a range of file offsets that are not are stored in the storage associated with the first node and are stored in a storage associated with the second node, wherein the storage associated with the first node is separate from the storage associated with the second node;
receiving at the first node from the second node the one or more requested data chunks associated with the first data portion of the object assigned to the first node that are not currently stored in the storage associated with the first node and are stored in the storage associated with the second node; and
providing from the first node to a target destination, at least the one or more requested data chunks associated with the first data portion of the object assigned to the first node that is received from the second node.
2. The method of claim 1 , further comprising receiving an acknowledgement from the target destination.
3. The method of claim 2 , wherein the acknowledgement is received from a restore agent associated with the target destination.
4. The method of claim 2 , wherein the acknowledgement is received from a restore agent associated with the target destination after all of the first data portion of the object assigned to the first node has been written to a sparse file associated with the object.
5. The method of claim 2 , wherein the acknowledgement is received from a restore agent associated with the target destination after some of the first data portion of the object assigned to the first node has been written to a sparse file associated with the object.
6. The method of claim 2 , further comprising providing the acknowledgement to an orchestrating node of the cluster.
7. The method of claim 6 , wherein the orchestrating node coordinates a restore of the object to the target destination.
8. The method of claim 1 , further comprising determining whether a size of the object is greater than or equal to a size threshold.
9. The method of claim 8 , wherein the object is divided into a plurality of data portions in response to determining the size of the object is greater than or equal to the size threshold.
10. The method of claim 8 , wherein the first node is assigned the first data portion of the object.
11. A computer program product embodied in a non-transitory computer readable medium and comprising computer instructions for:
analyzing, by a first node of a cluster of nodes, a data portion of an object assigned to the first node;
determining, by the first node, that the first node does not currently store in a storage associated with the first node at least a part of a first data portion of the object assigned to the first node;
providing from the first node to a second node of the cluster a request to provide one or more data chunks included in a range of file offsets that are not are stored in the storage associated with the first node and are stored in a storage associated with the second node, wherein the storage associated with the first node is separate from the storage associated with the second node;
receiving at the first node from the second node the one or more requested data chunks associated with the first data portion of the object assigned to the first node that are not currently stored in the storage associated with the first node and are stored in the storage associated with the second node; and
providing from the first node to a target destination, at least the one or more requested data chunks associated with the first data portion of the object assigned to the first node that is received from the second node.
12. The computer program product of claim 11 , further comprising computer instructions for receiving an acknowledgement from the target destination.
13. The computer program product of claim 12 , wherein the acknowledgement is received from a restore agent associated with the target destination.
14. The computer program product of claim 12 , wherein the acknowledgement is received from a restore agent associated with the target destination after all of the data portion of the object assigned to the first node has been written to a sparse file associated with the object.
15. The computer program product of claim 12 , wherein the acknowledgement is received from a restore agent associated with the target destination after some of the data portion of the object assigned to the first node has been written to a sparse file associated with the object.
16. The computer program product of claim 12 , further comprising computer instructions for providing the acknowledgement to an orchestrating node of the cluster.
17. The computer program product of claim 16 , wherein the orchestrating node coordinates a restore of the object to the target destination.
18. A system, comprising:
a processor configured to:
analyze, by a first node of a cluster of nodes, a data portion of an object assigned to the first node;
determine, by the first node, that the first node does not currently store in a storage associated with the first node at least a part of a first data portion of the object assigned to the first node;
provide from the first node to a second node of the cluster a request to provide one or more data chunks included in a range of file offsets that are not are stored in the storage associated with the first node and are stored in a storage associated with the second node, wherein the storage associated with the first node is separate from the storage associated with the second node;
receive at the first node from the second node the one or more requested data chunks associated with the first data portion of the object assigned to the first node that are not currently stored in the storage associated with the first node and are stored in the storage associated with the second node; and
provide from the first node to a target destination, at least the one or more requested data chunks associated with the first data portion of the object assigned to the first node that is received from the second node; and
a memory coupled to the processor and configured to provide the processor with instructions.
19. The system of claim 18 , wherein the processor is configured to receive an acknowledgement from the target destination.
20. The system of claim 19 , wherein the processor is configured to provide the acknowledgement to an orchestrating node of the cluster.