Summarized data storage management system for streaming data

1. A method for managing a data storage system, comprising:

storing a first set of explicitly enumerated data values included in a data stream from an electronic device in a first window data structure associated with a first time span;

storing one or more sets of summary statistics in a second set of window data structures, wherein each window data structure of the second set of window data structures is associated with a time span older than the first time span, and the one or more sets of summary statistics are based on a second set of data values included in the data stream; and

storing a third set of explicitly enumerated data values included in the data stream in a third set of window data structures, wherein each data value of the third set of explicitly enumerated data values is annotated to distinguish the annotated data value from the second set of data values that are unannotated;

wherein each window data structure is maintained on at least one storage device of the data storage system; and

wherein the one or more sets of summary statistics reduce operational costs associated with the data storage system.

2. The method of claim 1 , wherein the first window data structure comprises a sliding window, the second set of window data structures comprises a sequence of decayed windows, and the third set of window data structures comprises one or more landmark windows.

3. The method of claim 2 , wherein each decayed window of the sequence of decayed windows is associated with a different time span and has a different window size than another decayed window of the sequence of decayed windows.

4. The method of claim 3 , wherein a decayed window of the sequence of decayed windows is associated with an older time span covering a longer interval of time than another decayed window of the sequence of decayed windows.

5. The method of claim 3 , wherein the first set of explicitly enumerated data values is subsumed into the sequence of decayed windows at a decay rate controlled by a pattern of growth of window sizes of the sequence of decayed windows, and the decay rate is configurable.

6. The method of claim 2 , further comprising:

receiving a new data value;

appending the new data value to a landmark window in response to determining the new data value is annotated;

enqueuing the new data value in the sliding window; and

in response to determining a current size of the sliding window exceeds a pre-determined threshold value:

dequeing an oldest data value maintained by the sliding window;

updating the one or more sets of summary statistics maintained by the sequence of decayed windows to subsume the oldest data value into the sequence of decayed windows; and

merging a pair of consecutive decayed windows included in the sequence of decayed windows in response to determining a decay function is satisfied.

7. The method of claim 2 , further comprising:

receiving a query for a particular summary statistic over a particular temporal range;

generating a first query response based on a particular set of explicitly enumerated elements included in at least one of the sliding window or a landmark window associated with a time span that overlaps the particular temporal range;

determining whether the particular temporal range is completely covered by the particular set of explicitly enumerated elements;

in response to determining the particular temporal range is completely covered by the particular set of explicitly enumerated elements:

returning the first query response; and

in response to determining the particular temporal range is not completely covered by the particular set of explicitly enumerated elements:

generating a second query response based on one or more summary statistics included in at least one decayed window associated with a time span that overlaps the particular temporal range;

generating an error estimate for the second query response, wherein the error estimate is configurable; and

returning a third query response and the error estimate, wherein the third query response is a union of the first query response and the second query response.

8. The method of claim 1 , wherein:

the operational costs associated with the data storage system include at least one of storage consumption, bandwidth consumption, or latency;

each annotated data value of the third set of explicitly enumerated data values is more significant than each unannotated data value of the second set of data values; and

resolution of each annotated data value of the third set of explicitly enumerated data values is higher than each summary statistic of the one or more sets of summary statistics.

9. A data storage system comprising:

at least one processor; and

a non-transitory processor-readable memory device storing instructions that when executed by the at least one processor causes the at least one processor to perform operations including:

storing a first set of explicitly enumerated data values included in a data stream from an electronic device in a first window data structure associated with a first time span;

storing one or more sets of summary statistics in a second set of window data structures, wherein each window data structure of the second set of window data structures is associated with a time span older than the first time span, and the one or more sets of summary statistics are based on a second set of data values included in the data stream; and

storing a third set of explicitly enumerated data values included in the data stream in a third set of window data structures, wherein each data value of the third set of explicitly enumerated data values is annotated to distinguish the annotated data value from the second set of data values that are unannotated;

wherein each window data structure is maintained on at least one storage device of the data storage system; and

wherein the one or more sets of summary statistics reduce operational costs associated with the data storage system.

10. The system of claim 9 , wherein the first window data structure comprises a sliding window, the second set of window data structures comprises a sequence of decayed windows, and the third set of window data structures comprises one or more landmark windows.

11. The system of claim 10 , wherein each decayed window of the sequence of decayed windows is associated with a different time span and has a different window size than another decayed window of the sequence of decayed windows.

12. The system of claim 11 , wherein a decayed window of the sequence of decayed windows is associated with an older time span covering a longer interval of time than another decayed window of the sequence of decayed windows.

13. The system of claim 11 , wherein the first set of explicitly enumerated data values is subsumed into the sequence of decayed windows at a decay rate controlled by a pattern of growth of window sizes of the sequence of decayed windows, and the decay rate is configurable.

14. The system of claim 10 , wherein the operations further comprise:

receiving a new data value;

appending the new data value to a landmark window in response to determining the new data value is annotated;

enqueuing the new data value in the sliding window; and

in response to determining a current size of the sliding window exceeds a pre-determined threshold value:

dequeing an oldest data value maintained by the sliding window;

updating the one or more sets of summary statistics maintained by the sequence of decayed windows to subsume the oldest data value into the sequence of decayed windows; and

merging a pair of consecutive decayed windows included in the sequence of decayed windows in response to determining a decay function is satisfied.

15. The system of claim 10 , wherein the operations further comprise:

receiving a query for a particular summary statistic over a particular temporal range;

generating a first query response based on a particular set of explicitly enumerated elements included in at least one of the sliding window or a landmark window associated with a time span that overlaps the particular temporal range;

determining whether the particular temporal range is completely covered by the particular set of explicitly enumerated elements;

in response to determining the particular temporal range is completely covered by the particular set of explicitly enumerated elements:

returning the first query response; and

in response to determining the particular temporal range is not completely covered by the particular set of explicitly enumerated elements:

generating a second query response based on one or more summary statistics included in at least one decayed window associated with a time span that overlaps the particular temporal range;

generating an error estimate for the second query response, wherein the error estimate is configurable; and

returning a third query response and the error estimate, wherein the third query response is a union of the first query response and the second query response.

16. A non-transitory computer readable storage medium including instructions to perform a method for managing a data storage system, comprising:

storing a first set of explicitly enumerated data values included in a data stream from an electronic device in a first window data structure associated with a first time span;

storing one or more sets of summary statistics in a second set of window data structures, wherein each window data structure of the second set of window data structures is associated with a time span older than the first time span, and the one or more sets of summary statistics are based on a second set of data values included in the data stream; and

storing a third set of explicitly enumerated data values included in the data stream in a third set of window data structures, wherein each data value of the third set of explicitly enumerated data values is annotated to distinguish the annotated data value from the second set of data values that are unannotated;

wherein each window data structure is maintained on at least one storage device of the data storage system; and

wherein the one or more sets of summary statistics reduce operational costs associated with the data storage system.

17. The computer readable storage medium of claim 16 , wherein the first window data structure comprises a sliding window, the second set of window data structures comprises a sequence of decayed windows, and the third set of window data structures comprises one or more landmark windows.

18. The computer readable storage medium of claim 17 , wherein each decayed window of the sequence of decayed windows is associated with a different time span and has a different window size than another decayed window of the sequence of decayed windows.

19. The computer readable storage medium of claim 18 , wherein:

a decayed window of the sequence of decayed windows is associated with an older time span covering a longer interval of time than another decayed window of the sequence of decayed windows; and

the first set of explicitly enumerated data values is subsumed into the sequence of decayed windows at a decay rate controlled by a pattern of growth of window sizes of the sequence of decayed windows, and the decay rate is configurable.

20. The computer readable storage medium of claim 17 , wherein the method further comprises:

receiving a new data value;

appending the new data value to a landmark window in response to determining the new data value is annotated;

enqueuing the new data value in the sliding window; and

in response to determining a current size of the sliding window exceeds a pre-determined threshold value:

dequeing an oldest data value maintained by the sliding window;

updating the one or more sets of summary statistics maintained by the sequence of decayed windows to subsume the oldest data value into the sequence of decayed windows; and

merging a pair of consecutive decayed windows included in the sequence of decayed windows in response to determining a decay function is satisfied.