Back-calculation of aggregates
Described are techniques for back-calculating one or more values of a new, real-time aggregate before sufficient data to calculate the new, real-time aggregate has been collected, wherein the back-calculating is based on data collected for one or more aggregates that have been executing prior to start of execution of the new, real-time aggregate.
1 . A method implemented by a data processing system for back-calculating one or more values of a near real-time aggregate over a period of time, with the back-calculating being before sufficient new data items are received to determine a value of the near real-time aggregate using the new data items, the method including:
starting execution by the data processing system of one or more instructions for generating a near real-time aggregate over a period of time that begins when the one or more instructions start execution, with the near real-time aggregate being based on one or more events;
retrieving, from a data storage system, historical data collected from one or more previously executing instructions;
storing, in a data store, one or more items of the historical data representing the one or more events occurring over the period of time;
from time to time, receiving new data items by the data processing system, with a new data item being a data item received once execution of the one or more instructions starts;
storing, in the data store, one or more of the new data items identified as representing the one or more events; and
until the data processing system is able to generate a value of the near real-time aggregate over the period of time that begins when the one or more instructions start execution using the new data items and not using the historical data,
based on the one or more of the new data items and on the one or more items of the historical data, storing, in the data store, one or more values of the near real-time aggregate, with the or more values being accessible from the data store for retrieval as the execution of the one or more instructions continues;
wherein the one or more stored values provide for one or more values of the near real-time aggregate over the period of time that begins when the one or more instructions start execution and before the data processing system is able to generate a value of the near real-time aggregate over the period of time using the new data items and not using the historical data,
whereby the one or more values that are based on the one or more of the new data items and on the one or more items of the historical data reduces an amount of time from when execution of the one or more instructions starts to when the one or more values of the near real-time aggregate are stored in the data store,
wherein starting the execution of the one or more instructions for generating the near real-time aggregate over the period of time includes:
receiving, at a controller, a definition of the near real-time aggregate that newly defines the near real-time aggregate;
generating, by the controller, a signal for sending to a query generator to cause the query generator to query the data storage system; and
causing, by the query generator based on the signal, the retrieving, from the data storage system, of the historical data collected from one or more previously executing instructions, the historical data representing the one or more events occurring over the period of time for generation of the near real-time aggregate,
wherein generating, by the controller, the signal for the query generator to cause the query generator to query the data storage system includes:
specifying, in the signal, one or more event types for accessing from the data storage system, wherein causing the retrieving includes retrieving only events having the one or more event types from the data storage system, and wherein the events include the one or more events occurring over the period of time for generation of the near real-time aggregate.
2 . The method of claim 1 , further including:
generating a query to retrieve from an archive the historical data.
3 . The method of claim 2 , further including:
sending the historical data to a batch processing module.
4 . The method of claim 3 , further including:
sending the occurrences of one or more events to a real-time processing module.
5 . The method of claim 4 wherein as the one or more instructions execute for a given day, decrementing one day's worth of historical data, and incrementing one day's worth of the occurrences of one or more events.
6 . The method of claim 2 wherein the query includes a search key.
7 . The method of claim 6 wherein the search key is a customer identifier.
8 . The method of claim 2 wherein a collect module outputs to a batch module of an aggregate that is based on a given number of days of archived events and the batch module.
9 . The method of claim 8 wherein the batch module executes a computation graph that accesses the archived events, sorts the archived events according to a key, filters the sorted archived events according to the given number of days of events, and stores the given number of days of events and updates the near real-time aggregate stored in memory.
10 . The method of claim 1 , wherein the retrieving, from the data storage system, of the historical data is performed by performing, once for generating the near real-time aggregate, batch retrieval of the historic data.
11 . The method of claim 1 , wherein the data storage system is a non-volatile data storage system.
12 . The method of claim 1 , wherein the one or more values of the near real-time aggregate are stored in a memory that is volatile memory.
13 . The method of claim 1 ,
wherein the aggregate is generated by executing the one or more instructions on one or more inputs;
wherein identifying the one or more occurrences in the historical data includes identifying, in the historical data, one or more items of the historical data representing one or more historical occurrences of the one or more events, with the one or more historical occurrences occurring over a historical period of time, with an amount of time specified by the historical period of time corresponding to an amount of time specified by the period of time;
wherein the method further includes:
inputting the identified one or more items of the historical data as the one or more inputs.
14 . A data processing system for back-calculating one or more values of a near real-time aggregate over a period of time, with the back-calculating being before sufficient new data items are received to determine a value of the near real-time aggregate using the new data items, the data processing system including one or more processor devices and memory, with the data processing system including one or more processors configured to perform actions including:
starting execution by the data processing system of one or more instructions for generating a near real-time aggregate over a period of time that begins when the one or more instructions start execution, with the near real-time aggregate being based on one or more events;
retrieving, from a data storage system, historical data collected from one or more previously executing instructions;
storing, in a data store, one or more items of the historical data representing the one or more events occurring over the period of time;
from time to time, receiving new data items by the data processing system, with a new data item being a data item received once execution of the one or more instructions starts;
storing, in the data store, one or more of the new data items identified as representing the one or more events; and
until the data processing system is able to generate a value of the near real-time aggregate over the period of time that begins when the one or more instructions start execution using the new data items and not using the historical data,
based on the one or more of the new data items and on the one or more items of the historical data, storing, in the data store, one or more values of the near real-time aggregate, with the or more values being accessible from the data store for retrieval as the execution of the one or more instructions continues;
wherein the one or more stored values provide for one or more values of the near real-time aggregate over the period of time that begins when the one or more instructions start execution and before the data processing system is able to generate a value of the near real-time aggregate over the period of time using the new data items and not using the historical data,
whereby the one or more values being based on the one or more of the new data items and on the one or more items of the historical data reduces an amount of time from when execution of the one or more instructions starts to when the one or more values of the near real-time aggregate are stored in the data store;
wherein starting the execution of the one or more instructions for generating the near real-time aggregate over the period of time includes:
receiving, at a controller, a definition of the near real-time aggregate that newly defines the near real-time aggregate;
generating, by the controller, a signal for sending to a query generator to cause the query generator to query the data storage system; and
causing, by the query generator based on the signal, the retrieving, from the data storage system, of the historical data collected from one or more previously executing instructions, the historical data representing the one or more events occurring over the period of time for generation of the near real-time aggregate,
wherein generating, by the controller, the signal for the query generator to cause the query generator to query the data storage system includes:
specifying, in the signal, one or more event types for accessing from the data storage system, wherein causing the retrieving includes retrieving only events having the one or more event types from the data storage system, and wherein the events include the one or more events occurring over the period of time for generation of the near real-time aggregate.
15 . The data processing system of claim 14 , further including:
generating a query to retrieve from an archive the historical data.
16 . The data processing system of claim 15 , further including:
sending the historical data to a batch processing module.
17 . The data processing system of claim 16 , further including:
sending the occurrences of one or more events to a real-time processing module.
18 . The data processing system of claim 17 wherein as the one or more instructions execute for a given day, decrementing one day's worth of historical data, and incrementing one day's worth of the occurrences of one or more events.
19 . The data processing system of claim 15 wherein the query includes a search key.
20 . One or more non-transitory computer readable storage devices including instructions for back-calculating one or more values of a near real-time aggregate over a period of time, with the back-calculating being before sufficient new data items are received to determine a value of the near real-time aggregate using the new data items, the instructions causing a data processing system to perform actions including:
starting execution by the data processing system of one or more instructions for generating a near real-time aggregate over a period of time that begins when the one or more instructions start execution, with the near real-time aggregate being based on one or more events;
retrieving, from a data storage system, historical data collected from one or more previously executing instructions;
storing, in a data store, one or more items of the historical data representing the one or more events occurring over the period of time;
from time to time, receiving new data items by the data processing system, with a new data item being a data item received once execution of the one or more instructions starts;
storing, in the data store, one or more of the new data items identified as representing the one or more events; and
until the data processing system is able to generate a value of the near real-time aggregate over the period of time that begins when the one or more instructions start execution using the new data items and not using the historical data,
based on the one or more of the new data items and on the one or more items of the historical data, storing, in the data store, one or more values of the near real-time aggregate, with the or more values being accessible from the data store for retrieval as the execution of the one or more instructions continues;
wherein the one or more stored values provide for one or more values of the near real-time aggregate over the period of time that begins when the one or more instructions start execution and before the data processing system is able to generate a value of the near real-time aggregate over the period of time using the new data items and not using the historical data,
whereby the one or more values being based on the one or more of the new data items and on the one or more items of the historical data reduces an amount of time from when execution of the one or more instructions starts to when the one or more values of the near real-time aggregate are stored in the data store,
wherein starting the execution of the one or more instructions for generating the near real-time aggregate over the period of time includes:
receiving, at a controller, a definition of the near real-time aggregate that newly defines the near real-time aggregate;
generating, by the controller, a signal for sending to a query generator to cause the query generator to query the data storage system; and
causing, by the query generator based on the signal, the retrieving, from the data storage system, of the historical data collected from one or more previously executing instructions, the historical data representing the one or more events occurring over the period of time for generation of the near real-time aggregate,
wherein generating, by the controller, the signal for the query generator to cause the query generator to query the data storage system includes:
specifying, in the signal, one or more event types for accessing from the data storage system, wherein causing the retrieving includes retrieving only events having the one or more event types from the data storage system, and wherein the events include the one or more events occurring over the period of time for generation of the near real-time aggregate.