Query prediction modeling for distributed databases
A computer-implemented method for optimizing a distributed database includes receiving, at an aggregator, one or more query logs comprising past queries received by the distributed database. The method includes generating, by the aggregator, a query prediction model based on the one or more query logs. The method includes predicting, by the aggregator, a future query using the query prediction model. The predicted future query is predicted to be received by an edge device. The method includes causing, by the aggregator, data for responding to the predicted future query to be transmitted to the edge device.
1. A computer-implemented method for optimizing a distributed database, the method comprising:
receiving, at an aggregator, one or more query logs including one or more past queries received by the distributed database;
generating, by the aggregator, a query prediction model based on the one or more query logs;
predicting, by the aggregator, a future query to be received by a first edge device, wherein the aggregator performs the predicting using the query prediction model;
in response to the predicted future query being directed to a data set stored in edge storage of a second edge device, transmitting, by the aggregator, data for responding to the predicted future query to the first edge device via one or more networks,
wherein the data set includes sensor data, and
wherein the transmitting includes causing, by the aggregator, a subset of the data set to be stored as a redundant data set in edge storage of the first edge device;
generating, by the query prediction model, summary data based on the redundant data set;
storing the summary data on a dynamic ledger maintained by the aggregator,
wherein the dynamic ledger includes location information that indicates storage locations in at least one of the edge storage of the first edge device or the edge storage of the second edge device for the sensor data, and
wherein the dynamic ledger includes edge device role data that defines roles for the first edge device and the second edge device; and
responding, by the first edge device, to a future query based at least partially on the summary data.
2. The method of claim 1 wherein the data for responding to the predicted future query includes data stored at the second edge device.
3. The method of claim 1 further comprising:
locating the data for responding to the predicted future query using a shard algorithm,
wherein the shard algorithm is associated with a set of lookup tables that is used to locate the data for responding to the predicted future query in the distributed database.
4. The method of claim 3 wherein the shard algorithm is a neural network algorithm for partitioning of data that is local to an edge cluster of the distributed database.
5. The method of claim 3 wherein the shard algorithm is a genetic algorithm for partitioning of data that is local to an edge cluster of the distributed database.
6. The method of claim 3 wherein the shard algorithm is a logical algorithm for partitioning of data within an edge cluster based on a set of logical rules.
7. The method of claim 1 ,
wherein the second edge device is connected to a set of sensors and is configured to maintain the sensor data generated via the set of sensors in the edge storage of the second edge device, and
wherein the summary data includes at least one of: an average of the sensor data by region, an average of the sensor data by time, a maximum of the sensor data by region, a maximum of the sensor data by time, a minimum of the sensor data by region, or a minimum of the sensor data by time.
8. The method of claim 1 wherein the summary data includes at least one of statistical data or outlier data.
9. The method of claim 1 further comprising instructing, by the aggregator, the second edge device to generate additional summary data.
10. The method of claim 1 wherein the dynamic ledger is a blockchain.
11. The method of claim 1 wherein the data for responding to the predicted future query is a probability distribution model.
12. The method of claim 11 further comprising generating the probability distribution model based on data stored at the second edge device.
13. The method of claim 11 further comprising storing the probability distribution model on the dynamic ledger.
14. The method of claim 1 wherein the future query is an edge query language (EDQL) query.
15. The method of claim 1 wherein the data for responding to the predicted future query is based on the sensor data.
16. The method of claim 1 wherein the distributed database includes a mesh network of edge devices.
17. The method of claim 1 wherein the predicted future query is a distributed join query such that the distributed database is configured to execute any join query.
18. The method of claim 1 wherein the data for responding to the predicted future query is a reference table that is replicated on one or more devices in a cluster.
19. A system for optimizing a distributed database, the system comprising:
a first edge device communicatively coupled to a first edge storage, wherein the first edge device includes processing hardware and storage hardware;
a second edge device communicatively coupled to a second edge storage, wherein the second edge device includes processing hardware and storage hardware; and
an aggregator communicatively coupled to the first edge device and the second edge device,
wherein the aggregator includes processing hardware and storage hardware,
wherein the aggregator is configured to:
receive, from at least one of the first edge device or the second edge device, one or more query logs including one or more past queries received by the distributed database, wherein the one or more query logs are stored in at least one of the first edge storage or the second edge storage;
generate a query prediction model based on the one or more query logs;
predict a future query to be received by the first edge device using the query prediction model;
in response to the predicted future query being directed to a data set stored in the second edge storage, transmit data for responding to the predicted future query to the first edge device via one or more networks, including cause a subset of the data set to be stored as a redundant data set in the first edge storage, wherein the data set includes sensor data;
generate summary data based on the redundant data set using the query prediction model; and
store the summary data on a dynamic ledger,
wherein the dynamic ledger includes location information that indicates storage locations in at least one of the first edge storage or the second edge storage for the sensor data,
wherein the dynamic ledger includes edge device role data that defines roles for the first edge device and the second edge device, and
wherein the first edge device is configured to respond to the future query based at least partially on the summary data.
20. The system of claim 19 ,
wherein the aggregator is configured to locate edge data for responding to the predicted future query using a shard algorithm, and
wherein the shard algorithm is at least one of: a neural network algorithm, a genetic algorithm, or a logical algorithm.
21. The method of claim 1 ,
wherein the redundant data set includes a smaller volume of data than the data set, and
wherein the first edge device generates a response to a future query based on the redundant data set faster than a response to a future data query based on the data set.
22. The method of claim 1 further comprising:
receiving, by the first edge device, a plurality of data queries; and
reducing a failure rate for responding to the plurality of data queries by responding, by the first edge device, to the plurality of data queries based on the redundant data set.
23. The method of claim 1 wherein the causing the subset of the data set to be stored as the redundant data set in the edge storage of the first edge device enables the distributed database to be fault tolerant to a cyberattack to the second edge device.
24. The system of claim 19 wherein the redundant data set includes at least some of the sensor data.