Storage management in a clustered storage environment
View Patent ↗This disclosure describes techniques for enabling efficient and/or expeditious retrieval of data records stored across a clustered storage framework. An example system manages storage of data records in the storage clusters based on predefined placement rules. Frequently accessed data records are stored in a local high-performance store storage medium to reduce retrieval latency. The system determines which storage cluster to retrieve a requested data record from by selecting the cluster with the highest weight, as determined based on performance-related metrics like cluster utilization and geographic proximity.
1. A method comprising:
receiving, by a processor and from a first computing device, a first request to retrieve a first data record from a storage framework, the storage framework comprising:
a first storage cluster, and
a second storage cluster;
querying, by the processor and based on the first request, the storage framework to determine:
a first weight associated with the first storage cluster, and
a second weight associated with the second storage cluster;
predicting, by the processor and based on the first weight, a first likelihood of retrieving the first data record from the first storage cluster;
predicting, by the processor and based on the second weight, a second likelihood of retrieving the first data record from the second storage cluster;
determining, by the processor and based on the first likelihood, the second likelihood, and a range associated with a probability distribution, a likelihood threshold;
determining, by the processor, first data representing that a first value sampled from the probability distribution exceeds the likelihood threshold;
providing, by the processor and based on the first data, the first request to the first storage cluster;
receiving, by the processor and based on providing the first request to the first storage cluster, the first data record from the first storage cluster; and
providing, by the processor and to the first computing device, the first data record.
2. The method of claim 1 , further comprising:
determining, by the processor, a retrieval latency associated with routing the first request to the first storage cluster and receiving the first data record from the first storage cluster; and
updating, by the processor, the first weight based on the retrieval latency.
3. The method of claim 1 , further comprising:
determining, by the processor, a capacity measure associated with the first storage cluster; and
updating, by the processor, the first weight based on the capacity measure.
4. The method of claim 1 , further comprising:
determining, by the processor, a first retrieval statistic associated with the first data record;
determining, by the processor, that the first retrieval statistic exceeds a threshold; and
based on determining that the first retrieval statistic exceeds the threshold, storing, by the processor, the first data record on the storage framework.
5. The method of claim 4 , further comprising:
receiving, by the processor, a notification indicative of an update to the first data record; and
based on receiving the notification, removing, by the processor, the first data record from the storage framework.
6. The method of claim 5 , further comprising:
determining, by the processor, a second retrieval statistic associated with a second data record;
determining, by the processor, whether the second retrieval statistic has a value that exceeds the threshold; and
updating the threshold based on the threshold.
7. The method of claim 6 , further comprising:
determining, by the processor, that the first retrieval statistic fails to exceed the threshold; and
based on determining that the first retrieval statistic fails to exceed the threshold, removing, by the processor, the first data record from the storage framework.
8. The method of claim 1 , further comprising:
receiving, by the processor, a notification about unavailability of the first storage cluster; and
based on receiving the notification, updating, by the processor, the first weight.
9. The method of claim 1 , further comprising:
determining, by the processor, a network utilization of the first storage cluster; and
updating, by the processor, the first weight based on the network utilization.
10. A computing system, comprising:
a processor; and
memory storing computer-executable instructions that, when executed by the processor, cause the computing system to perform operations comprising:
receiving, from a first computing device, a first request to retrieve a first data record from a storage framework, the storage framework comprising:
a first storage cluster, and
a second storage cluster;
querying, based on the first request, a storage medium to determine:
a first weight associated with the first storage cluster, and
a second weight associated with the second storage cluster;
predicting, based on the first weight, a first likelihood of retrieving the first data record from the first storage cluster;
predicting, based on the second weight, a second likelihood of retrieving the first data record from the second storage cluster;
determining, based on the first likelihood, the second likelihood, and a range associated with a probability distribution, a likelihood threshold;
determining first data representing that a first value sampled from the probability distribution exceeds the likelihood threshold;
providing, based on the first data, the first request to the first storage cluster;
receiving, based on providing the first request to the first storage cluster, the first data record from the first storage cluster; and
providing, to the first computing device, the first data record.
11. The computing system of claim 10 , the operations further comprising:
determining a retrieval latency associated with routing the first request to the first storage cluster and receiving the first data record from the first storage cluster; and
updating the first weight based on the retrieval latency.
12. The computing system of claim 10 , the operations further comprising:
determining a first retrieval statistic associated with the first data record;
determining that the first retrieval statistic exceeds a threshold; and
based on determining that the first retrieval statistic exceeds the threshold, storing the first data record on the storage medium.
13. The computing system of claim 12 , the operations further comprising:
receiving a notification indicative of an update to the first data record; and
based on receiving the notification, removing the first data record from the storage medium.
14. The computing system of claim 10 , the operations further comprising:
determining a capacity measure associated with the first storage cluster; and
updating the first weight based on the capacity measure.
15. One or more non-transitory computer-readable media storing computer-executable instructions that, when executed by a processor, cause the processor to perform operations, comprising:
receiving, from a first computing device, a first request to retrieve a first data record from a storage framework, the storage framework comprising:
a first storage cluster, and
a second storage cluster;
querying, based on the first request, a storage medium to determine:
a first weight associated with the first storage cluster, and
a second weight associated with the second storage cluster;
predicting, based on the first weight, a first likelihood of retrieving the first data record from the first storage cluster;
predicting, based on the second weight, a second likelihood of retrieving the first data record from the second storage cluster;
determining, based on the first likelihood, the second likelihood, and a range associated with a probability distribution, a likelihood threshold;
determining first data representing that a first value sampled from the probability distribution exceeds the likelihood threshold;
providing, based on the first data, the first request to the first storage cluster;
receiving, based on providing the first request to the first storage cluster, the first data record from the first storage cluster; and
providing, to the first computing device, the first data record.
16. The one or more non-transitory computer-readable media of claim 15 , the operations further comprising:
determining a retrieval latency associated with routing the first request to the first storage cluster and receiving the first data record from the first storage cluster; and
updating the first weight based on the retrieval latency.
17. The one or more non-transitory computer-readable media of claim 15 , the operations further comprising:
determining a first retrieval statistic associated with the first data record;
determining that the first retrieval statistic exceeds a threshold; and
based on determining that the first retrieval statistic exceeds the threshold, storing the first data record on the storage medium.
18. The one or more non-transitory computer-readable media of claim 17 , the operations further comprising:
receiving a notification indicative of an update to the first data record; and
based on receiving the notification, removing the first data record from the storage medium.
19. The one or more non-transitory computer-readable media of claim 15 , the operations further comprising:
determining a capacity measure associated with the first storage cluster, and
updating the first weight based on the capacity measure.