Dynamic sharding for state-based processing
View Patent ↗Enabling a distributed data processing system to process a data set from local storage devices by dynamically reallocating portions of the data set.
1. A system, comprising:
a first node to store data in a first local storage device;
a second node including a second local storage device;
a distributed cache separate from the first node and the second node, the distributed cache in communication with the first node and the second node via a network, the first node to:
when a size of the data stored in the first local storage device is less than a threshold, perform a plurality of backup operations to duplicate the data from the first local storage device to the distributed cache;
when the size of the data stored in the first local storage device is greater than or equal to the threshold: (a) send at least a portion of the data from the first local storage device to the distributed cache to reduce the size of the data stored in the first local storage device to be less than the threshold, and (b) output an instruction to transfer the at least the portion of the data from the distributed cache to the second local storage device; and
a controller to, in response to receiving the instruction from the first node, transfer the at least the portion of the data from the distributed cache to the second local storage device, the at least the portion of the data based on an attribute that is the same across multiple items of the at least the portion of the data.
2. The system of claim 1 , wherein the threshold is a ratio of the size of the data stored in the first local storage device relative to a capacity of the first local storage device.
3. The system of claim 1 , wherein the first local storage device is in direct communication with the first node.
4. The system of claim 1 , wherein the first local storage device includes volatile memory.
5. The system of claim 1 , wherein the distributed cache includes nonvolatile memory.
6. The system of claim 1 , wherein the controller is further to provide instructions to the first node, the second node, and the distributed cache.
7. The system of claim 1 , wherein the controller is further to:
transfer the at least the portion of the data from the distributed cache to the second local storage device in response to a determination that the first node has experienced a failure and based on the size of the data stored in the first local storage device being greater than or equal to the threshold.
8. A method, comprising:
storing data in a first local storage device of a first node;
performing, by executing a first instruction with a processor, a plurality of backup operations to duplicate the data from the first local storage device to a distributed cache when a size of the data stored in the first local storage device is less than a threshold;
sending, by executing a second instruction with the processor, at least a portion of the data from the first local storage device to the distributed cache via a network when the size of the data stored in the first local storage device is greater than or equal to the threshold, the sending of the at least the portion of the data to reduce the size of the data stored in the first local storage device to be less than the threshold; and
sending, by executing a third instruction with the processor, a fourth instruction from the first node to a controller, the fourth instruction to cause the controller to transfer the at least the portion of the data from the distributed cache to a second local storage device of a second node, the at least the portion of the data based on an attribute that is the same across multiple items of the at least the portion of the data.
9. The method of claim 8 , wherein the threshold is a ratio of the size of the data stored in the first local storage device relative to a capacity of the first local storage device.
10. The method of claim 8 , wherein the first local storage device includes volatile memory.
11. The method of claim 8 , wherein the distributed cache includes nonvolatile memory.
12. The method of claim 8 , further including:
transferring the at least the portion of the data from the distributed cache to a third local storage device of a third node in response to a failure of the first node, the transferring based on the size of the data stored in the first local storage device being greater than or equal to the threshold.
13. A storage device or storage disk comprising instructions that, when executed by a processor at a first node, cause the processor to at least:
store data in a first local storage device of the first node;
perform a plurality of backup operations to duplicate the data from the first local storage device to a distributed cache when a size of the data stored in the first local storage device is less than a threshold;
send at least a portion of the data from the first local storage device to the distributed cache via a network when the size of the data stored in the first local storage device is greater than or equal to the threshold, the processor to send the at least the portion of the data to reduce the size of the data stored in the first local storage device to be less than the threshold; and
instruct a controller to transfer the at least the portion of the data from the distributed cache to a second local storage device of a second node, the at least the portion of the data based on an attribute that is the same across multiple items of the at least the portion of the data.
14. The storage device or storage disk of claim 13 , wherein the threshold is a ratio of the size of the data stored in the first local storage device relative to a capacity of the first local storage device.
15. The storage device or storage disk of claim 13 , wherein the first local storage device includes volatile memory.
16. The storage device or storage disk of claim 13 , wherein the distributed cache includes nonvolatile memory.
17. The storage device or storage disk of claim 13 , wherein the instructions are further to cause the processor to:
instruct the controller to transfer the at least the portion of the data from the distributed cache to a third local storage device of a third node in response to a failure of the first node and based on the size of the data stored in the first local storage device being greater than or equal to the threshold.
18. The system of claim 1 , wherein the first node is to perform the plurality of backup operations periodically when the size of the data stored in the first local storage device is less than the threshold.
19. The method of claim 8 , wherein the performing of the plurality of backup operations is periodic.
20. The storage device or storage disk of claim 13 , wherein the instructions are to cause the processor to perform the plurality of backup operations periodically.