Automatic data replica manager in distributed caching and data processing systems
A method of data storage includes determining a latency distance from a primary node to each of two or more replica nodes, choosing a preferred replica node of the two or more replica nodes based on the determined latency distances, and write-caching data into the preferred replica node.
1 . A method of data storage, the method comprising:
determining a first network parameter between a first node and a second node;
determining a second network parameter between the first node and a third node that is greater than the first network parameter; and
writing first data, associated with the first node, from a fourth node to the second node, based on determining that the second network parameter is greater than the first network parameter, such that the first node is configured to access the second node based on the first network parameter,
wherein the first node is configured to serve as a primary node, and
wherein the second node and the fourth node are configured to serve as replica nodes for the first node.
2 . The method of claim 1 , wherein the first network parameter corresponds to a first latency distance, and
wherein the second network parameter corresponds to a second latency distance.
3 . The method of claim 1 , wherein the first data is copied from the first node.
4 . The method of claim 1 , wherein the first network parameter and the second network parameter are less than a third network parameter between the first node and the fourth node.
5 . The method of claim 1 , further comprising:
assigning, to the second node, a first ranking indicating a degree to which the second node is suitable for receiving data that is copied from the first node; and
assigning, to the third node, a second ranking indicating a degree to which the third node is suitable for receiving data that is copied from the first node, the first ranking being based on the first network parameter, the first network parameter being associated with a first path between the first node and the second node, and the second ranking being based on the second network parameter, the second network parameter being associated with a second path between the first node and the third node.
6 . The method of claim 5 , further comprising updating the first ranking or the second ranking based on changes in the first network parameter or the second network parameter.
7 . The method of claim 5 , wherein at least one of the first node, the second node, or the third node comprises a physical host for running virtual machines.
8 . The method of claim 5 , wherein the first ranking or the second ranking is further based on:
an access frequency of data associated with the second node or the third node; or
service level agreements (SLAs) associated with the second node or the third node.
9 . The method of claim 1 , further comprising maintaining the first data on the fourth node.
10 . The method of claim 1 , wherein at least one of the first node, the second node, or the third node comprises:
a solid-state drive tier as a cache tier comprising a cache partition for storing data of local virtual machines; and
a replica partition for storing replica data from other nodes.
11 . The method of claim 1 , wherein the third node comprises a replica node.
12 . The method of claim 1 , wherein the first data comprises replica data.
13 . A method of reading replicated data, the method comprising:
determining a difference between an access speed corresponding to a first node and an access speed corresponding to a second node; or
determining a utilization ratio of throughput corresponding to the second node; and
performing a parallel prefetching of a dataset by reading a first part of the dataset from the first node, and reading a second part of the data set from the second node, such that a makespan of the parallel prefetching is less than a makespan of reading the first part of the dataset and the second part of the dataset from the first node, based on:
comparing the difference with a first threshold; or
comparing the utilization ratio of throughput with a second threshold.
14 . The method of claim 13 , wherein:
the first node comprises a primary node configured to store the first part of the dataset at a cache partition of the first node; and
the second node comprises a replica node configured to store the second part of the dataset at a replica partition of the second node.
15 . A method of prefetching data, the method comprising:
splitting a dataset into a first part and a second part;
determining a first ratio of a total size of the dataset to a first access speed associated with a first node comprising the dataset, the first node being configured to serve as a primary node;
determining a second ratio of the second part to a second access speed associated with a second node, the second node being configured to serve as a replica node; and
triggering a loading of the first part from the first node and the second part from the second node based on the first ratio being greater than or equal to the second ratio.
16 . The method of claim 15 , wherein the first ratio is greater than or equal to a greater of:
a second ratio of a first size of the first part to the first access speed; and
a third ratio of a second size of the second part to a second access speed associated with the second node.
17 . The method of claim 16 , wherein the first access speed accounts for a first network parameter associated with the first node, and wherein the second access speed accounts for a second network parameter associated with the second node.
18 . The method of claim 16 , further comprising:
determining a ratio of the first access speed to a sum of the first access speed and the second access speed; and
achieving a total input/output (I/O) time that is equal to a ratio of the total size of the dataset to the sum of the first access speed and the second access speed.
19 . The method of claim 16 , further comprising:
determining a difference between the first access speed and the second access speed, or determining a utilization ratio of throughput corresponding to the second node; and
triggering parallel prefetching.