REAL TIME DATA AGGREGATION AND ANALYSIS
Disclosed are methods, systems, and computer-readable medium for real time data aggregation and analytics. For instance, the method may include receiving indexed streaming data from a plurality of sources, receiving historical data form the plurality of sources, aggregating the indexed streaming data and the historical data in real time to generate aggregated data, generating a machine learning model based on the historical data and providing the machine learning model to the real-time cube, providing the aggregated data to a real-time cube based on a query for the aggregated data, extracting updated data from the aggregated data and, providing the extracted updated data for visualization.
1 . A system comprising at least one memory storing instructions and at least one processor executing the instructions to perform operations, the system further comprising:
a data aggregator;
a real-time cube;
the data aggregator configured to:
receive indexed streaming data from a plurality of clients;
receive historical data from the plurality of clients;
aggregate the indexed streaming data and the historical data in real-time to generate aggregated data; and
provide the aggregated data to the real-time cube based on a query for the aggregated data; and
a machine learning component configured to generate a machine learning model based on the historical data and provide the machine learning model to the real-time cube;
the real-time cube configured to:
query aggregated from the data aggregator;
receive the queried aggregated data from the data aggregator;
extract updated data from the queried aggregated data; and
provide the extracted updated data.
2 . The system of claim 1 , wherein the real-time cube is configured to query real-time data from the data aggregator based on at least one of a push request or a poll request.
3 . The system of claim 2 , wherein the updated data is a subset of the real-time data that is modified between a current query and a previous query.
4 . The system of claim 1 , wherein the machine learning model generated by the machine learning component outputs one or more query recommendations.
5 . The system of claim 4 , wherein the one or more query recommendations are based on the aggregated data being input into the machine learning model.
6 . The system of claim 1 , wherein the machine learning model generated by the machine learning component outputs one or more trigger thresholds based on the historical data.
7 . The system of claim 6 , wherein the real-time cube generates a trigger warning based on real-time data meeting the one or more trigger thresholds.
8 . The system of claim 1 , wherein the real-time cube passively receives the real-time data from the data aggregator.
9 . The system of claim 1 , further comprising a data broker configured to index streaming data.
10 . The system of claim 9 , wherein the data broker provides the indexed streaming data to the data aggregator.
11 . The system of claim 10 , wherein the data broker receives streaming data from a plurality of clients.
12 . The system of claim 10 , wherein the data broker comprises a cluster of data brokers.
13 . The system of claim 1 , further comprising a historical database configured to:
receive relational data;
receive streaming data form a data processor; and
provide the historical data to the data aggregator and the machine learning component.
14 . A method for real-time data aggregation and analytics, the method comprising:
receiving indexed streaming data from a plurality of sources;
receiving historical data form the plurality of sources;
aggregating the indexed streaming data and the historical data in real time to generate aggregated data;
generating a machine learning model based on the historical data and providing the machine learning model to a real-time cube;
providing the aggregated data to the real-time cube based on a query for the aggregated data;
extracting updated data from the aggregated data; and
providing the extracted updated data for visualization.
15 . The method of claim 14 , wherein the query for the aggregated data is based on the machine learning model.
16 . The method of claim 14 , wherein the machine learning model provides one or more triggers based on the historical data.
17 . The method of claim 16 , wherein the one or more triggers are applied by a visualization component that displays the extracted updated data.
18 . A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform data aggregation and analytics operations, the operations comprising:
receiving indexed streaming data from a plurality of sources;
receiving historical data form the plurality of sources;
aggregating the indexed streaming data and the historical data in real time to generate aggregated data;
generating a machine learning model based on the historical data and providing the machine learning model to a real-time cube;
providing the aggregated data to the real-time cube based on a query for the aggregated data;
extracting updated data from the aggregated data; and
providing the extracted updated data for visualization.
19 . The non-transitory computer-readable medium of claim 18 , wherein the query for the aggregated data is based on the machine learning model.
20 . The non-transitory computer-readable medium of claim 18 , wherein the machine learning model provides one or more triggers based on the historical data.