Computing time-decayed aggregates in data streams
Aggregates are calculated from a data stream in which data is sent in a sequence of tuples, in which each tuple comprises an item identifier and a timestamp indicating when the tuple was transmitted. The tuples may arrive out-of-order, that is, the sequence in which the tuples arrive are not necessarily in the sequence of their corresponding timestamps. In calculating aggregates, more recent data may be given more weight by multiplying each tuple by a decay function which is a function of the timestamp associated with the tuple and the current time. The tuples are recorded in a quantile-digest data structure. Aggregates are calculated from the data stored in the quantile-digest data structure.
1. An apparatus comprising:
a processor; and
a memory to store computer program instructions for calculating a time-decayed aggregate from a data stream comprising a sequence of tuples, each tuple comprising an item identifier and an associated timestamp, the computer program instructions when executed on the processor cause the processor to perform operations comprising:
generating a time-dependent weighted sequence of tuples by multiplying each tuple of the sequence of tuples by a time-dependent weighting factor calculated from a decay function, wherein the decay function is a function of the timestamp associated with the tuple and a current time;
generating a quantile-digest data structure from the time-dependent weighted sequence of tuples;
updating the quantile-digest data structure with the time-dependent weighted sequence of tuples; and,
calculating the time-decayed aggregate from the updated quantile-digest data structure.
2. The apparatus of claim 1 , wherein the quantile-digest data structure comprises a set of quantile-digests.
3. The apparatus of claim 1 , wherein the operations further comprise compressing the quantile-digest data structure.
4. The apparatus of claim 1 , wherein the decay function is a sliding-window function.
5. The apparatus of claim 1 , wherein the decay function is an exponential decay function.
6. The apparatus of claim 1 , wherein the decay function is a polynomial decay function.
7. The apparatus of claim 1 , wherein the time-decayed aggregate is a time-decayed user-defined aggregate function.
8. The apparatus of claim 1 , wherein the time-decayed aggregate is a time-decayed count.
9. The apparatus of claim 1 , wherein the time-decayed aggregate is a time-decayed range.
10. The apparatus of claim 1 , wherein the time-decayed aggregate is a time-decayed quantile.
11. The apparatus of claim 1 , wherein the time-decayed aggregate is a time-decayed heavy hitter.