Computing time-decayed aggregates under smooth decay functions
Aggregates are calculated from a data stream in which data is sent in a sequence of tuples, in which each tuple comprises an item identifier and a timestamp indicating when the tuple was transmitted. The tuples may arrive at a data receiver out-of-order, that is, the sequence in which the tuples arrive are not necessarily in the same sequence as their corresponding timestamps. In calculating aggregates, more recent data may be given more weight by a decay function which is a function of the timestamp associated with the tuple and the current time. The statistical characteristics of the tuples are summarized by a set of linear data summaries. The set of linear data summaries are generated such that only a single linear data summary falls between a set of boundaries calculated from the decay function and a set of timestamps. Aggregates are calculated from the set of linear data summaries.
1. An apparatus comprising:
a processor; and
a memory to store computer program instructions for calculating a time-decayed aggregate from a data stream comprising a sequence of tuples, each tuple comprising an item identifier and an associated timestamp, the computer program instructions when executed on the processor cause the processor to perform operations comprising:
receiving the data stream comprising the sequence of tuples;
generating a smooth decay function wherein the smooth decay function is a function of the difference between a current time and the timestamps;
generating a set of linear data summaries, based at least in part on a first set of timestamps;
generating a set of boundaries, based at least in part on the smooth decay function and on a second set of timestamps;
adding each tuple to a specific linear data summary based at least in part on the timestamp associated with the tuple;
merging two linear data summaries based at least in part on the set of boundaries to generate merged linear data;
calculating the aggregate from the set of linear data summaries;
receiving a query;
answering the query by probing the merged linear data.
2. The apparatus of claim 1 ,
wherein the sequence of tuples is represented by the set {<x i , t i >} wherein x i is the item identifier of the i-th tuple, t i is the timestamp associated with the i-th tuple, and i is a sequential integer index of the i-th tuple;
wherein the decay function is represented by the function g(t−t i ) wherein t is the current time;
wherein the set of linear data summaries is represented by the set {s j } wherein linear data summary s j summarizes the statistical characteristics of tuples with associated timestamps t i in the range t j <t i <t j+1 ;
wherein the set of boundaries b i is generated by the relationship
g ( b i )=(1+θ) −i
where θ is an accuracy parameter; and,
wherein two linear data summaries are merged if the following relationship is true:
( t−b i )< t j+2 <t j <( t−b i+1 ).
3. The apparatus of claim 1 wherein the set of linear data summaries is a quantile-digest data structure.
4. The apparatus of claim 1 , wherein the operations further comprise compressing the set of linear data summaries.
5. The apparatus of claim 1 wherein the decay function is a polynomial decay function.
6. The apparatus of claim 1 wherein the time-decayed aggregate is a time-decayed user-defined aggregate function.
7. The apparatus of claim 1 wherein the time-decayed aggregate is a time-decayed count.
8. The apparatus of claim 1 wherein the time-decayed aggregate is a time-decayed range.
9. The apparatus of claim 1 wherein the time-decayed aggregate is a time-decayed quantile.
10. The apparatus of claim 1 wherein the time-decayed aggregate is a time-decayed heavy hitter.