Computing time-decayed aggregates in data streams
View Patent ↗Aggregates are calculated from a data stream in which data is sent in a sequence of tuples, in which each tuple comprises an item identifier and a timestamp indicating when the tuple was transmitted. The tuples may arrive out-of-order, that is, the sequence in which the tuples arrive are not necessarily in the sequence of their corresponding timestamps. In calculating aggregates, more recent data may be given more weight by multiplying each tuple by a decay function which is a function of the timestamp associated with the tuple and the current time. The tuples are recorded in a quantile-digest data structure. Aggregates are calculated from the data stored in the quantile-digest data structure.
1. A method for calculating a time-decayed aggregate from a data stream comprising a sequence of tuples, each tuple comprising an item identifier and an associated timestamp, comprising the steps of:
generating a time-dependent weighted sequence of tuples by multiplying with a processor each of said sequence of tuples by a time-dependent weighting factor calculated from a decay function, wherein said decay function is a function of the timestamp associated with the tuple and a current time;
generating a quantile-digest data structure from said time-dependent weighted sequence of tuples;
updating said quantile-digest data structure with said time-dependent weighted sequence of tuples; and,
calculating said time-decayed aggregate from the updated quantile-digest data structure.
2. The method of claim 1 wherein said quantile-digest data structure comprises a set of quantile-digests.
3. The method of claim 1 further comprising the step of compressing said quantile-digest data structure.
4. The method of claim 1 wherein said decay function is a sliding-window function.
5. The method of claim 1 wherein said decay function is an exponential decay function.
6. The method of claim 1 wherein said decay function is a polynomial decay function.
7. The method of claim 1 wherein said time-decayed aggregate is a time-decayed user-defined aggregate function.
8. The method of claim 1 wherein said time-decayed aggregate is a time-decayed count.
9. The method of claim 1 wherein said time-decayed aggregate is a time-decayed range.
10. The method of claim 1 wherein said time-decayed aggregate is a time-decayed quantile.
11. The method of claim 1 wherein said time-decayed aggregate is a time-decayed heavy hitter.
12. An apparatus for calculating a time-decayed aggregate from a data stream comprising a sequence of tuples, each tuple comprising an item identifier and an associated timestamp, comprising:
means for generating a time-dependent weighted sequence of tuples by multiplying each of said sequence of tuples by a time-dependent weighting factor calculated from a decay function, wherein said decay function is a function of the timestamp associated with the tuple and a current time;
means for generating a quantile-digest data structure from said time-dependent weighted sequence of tuples;
means for updating said quantile-digest data structure with said time-dependent weighted sequence of tuples; and,
means for calculating said time-decayed aggregate from the updated quantile-digest data structure.
13. The apparatus of claim 12 , further comprising means for generating a set of quantile-digests.
14. The apparatus of claim 12 , further comprising means for compressing said quantile-digest data structure.
15. The apparatus of claim 12 , further comprising means for calculating a time-decayed user-defined aggregate function.
16. The apparatus of claim 12 , further comprising means for calculating a time-decayed count.
17. The apparatus of claim 12 , further comprising means for calculating a time-decayed range.
18. The apparatus of claim 12 , further comprising means for calculating a time-decayed quantile.
19. The apparatus of claim 12 , further comprising means for calculating a heavy hitter.
20. A non-transitory computer readable medium storing computer program instructions for calculating a time-decayed aggregate from a data stream comprising a sequence of tuples, each tuple comprising an item identifier and an associated timestamp, the computer program instructions defining the steps of:
generating a time-dependent weighted sequence of tuples by multiplying each of said sequence of tuples by a time-dependent weighting factor calculated from a decay function, wherein said decay function is a function of the timestamp associated with the tuple and a current time;
generating a quantile-digest data structure from said time-dependent weighted sequence of tuples;
updating said quantile-digest data structure with said time-dependent weighted sequence of tuples; and,
calculating said time-decayed aggregate from the updated quantile-digest data structure.
21. The non-transitory computer readable medium of claim 20 wherein said computer program instructions defining the step of generating a quantile-digest data structure further comprise computer program instructions defining the step of:
generating a set of quantile-digests.
22. The non-transitory computer readable medium of claim 20 wherein said computer program instructions further comprise computer program instructions defining the step of:
compressing said quantile-digest data structure.
23. The non-transitory computer readable medium of claim 20 wherein said computer program instructions defining the step of calculating a time-decayed aggregate further comprise computer instructions defining the step of:
calculating a time-decayed user-defined aggregate function.
24. The non-transitory computer readable medium of claim 20 wherein said computer program instructions defining the step of calculating a time-decayed aggregate further comprise computer instructions defining the step of:
calculating a time-decayed quantile.
25. The non-transitory computer readable medium of claim 20 wherein said computer program instructions defining the step of calculating a time-decayed aggregate further comprise computer instructions defining the step of:
calculating a time-decayed heavy hitter.