Dynamically materializing a multi-dimensional data stream cube
A computer readable storage medium comprises executable instructions to generate a stream cube to store data received from multiple data sources. A plurality of multi-dimensional data streams are generated to represent the received data. A materialization strategy is repeatedly determined for the stream cube. The stream cube is materialized according to the materialization strategy to record the multi-dimensional data streams.
1. A computer-implemented method comprising:
generating a stream cube to store data received from multiple data sources;
generating a plurality of multi-dimensional data streams to represent the received data;
repeatedly determining a materialization strategy for the stream cube during a duration of the stream cube, the materialization strategy specifying when to materialize the stream cube;
dynamically materializing the stream cube according to the repeatedly determined materialization strategy to record the multi-dimensional data streams;
refreshing the stream cube according to the selected materialization strategy and
wherein the stream cube comprises a plurality of sub-cubes associated with a plurality of time ranges each sub-cube aggregating multi-dimensional data for a given time range.
2. The method of claim 1 , wherein the multi-dimensional data streams comprise a plurality of multi-dimensional data tuples, each data tuple associated with a timestamp.
3. The method of claim 1 , wherein each sub-cube comprises a plurality of cuboids, each cuboid representing a set of dimensions and measures.
4. The method of claim 1 , wherein the plurality of sub-cubes are grouped by a set of time ranges.
5. The method of claim 1 , wherein repeatedly determining a materialization strategy for the stream cube comprises: repeatedly evaluating system conditions and users' requests and determining how to materialize the stream cube in response to the system conditions and users' requests.
6. The method of claim 5 , wherein the materialization strategy comprises a materialization strategy selected from the group consisting of on-demand materialization, full materialization, and partial materialization.
7. The method of claim 6 , wherein repeatedly determining a materialization strategy for the stream cube comprises: repeatedly determining a materialization path for the stream cube according to a predetermined criterion and a cost function.
8. The method of claim 7 , wherein repeatedly materializing the stream cube comprises: recording the plurality of multi-dimensional data streams along the materialization path.
9. A computer-implemented method comprising:
generate a stream cube for representing multi-dimensional data streams;
repeatedly determine a materialization path for the stream cube during a duration of the stream cube and based on a predetermined criterion and a cost function, the repeatedly determining comprising repeatedly evaluating the predetermined criterion to determine when to compute the materialization path; and
record the multi-dimensional data streams into the stream cube and refresh the stream cube according to a dynamically selected materialization path.
10. The method of claim 9 , wherein the multi-dimensional data streams comprise a plurality of multi-dimensional data tuples, each data tuple associated with a timestamp.
11. The method of claim 10 , wherein the stream cube comprises a plurality of sub-cubes associated with a plurality of time ranges, each sub-cube aggregating multi-dimensional data for a given time range.
12. The method of claim 11 , wherein each sub-cube comprises a plurality of cuboids, each cuboid representing a set of dimensions and measures.
13. The method of claim 12 , wherein recording the multi-dimensional data streams into the stream cube according to the materialization path comprises: updating measures for cuboids along the materialization path within each sub-cube in the plurality of sub-cubes.
14. The method of claim 9 , wherein the predetermined criterion is selected from a group consisting of a query-based criterion, a threshold-based criterion, and a time-based criterion.
15. The method of claim 9 , wherein repeatedly determining a materialization path comprises: computing the cost function for each cuboid.
16. The method of claim 15 , wherein the cost function comprises a function of a set of parameters and a set of weights assigned to the cuboid parameters.
17. The method of claim 16 , wherein the cuboid parameters are selected from a group consisting of cuboid materialization memory cost, cuboid popularity, and cuboid update potential.
18. The method of claim 15 , further comprising: determining a cost for each potential materialization path in the stream cube.
19. The method of claim 18 , further comprising: selecting the materialization path from all potential materialization paths that has the lowest cost.
20. The method of claim 19 , wherein selecting the materialization path comprises: using an optimization technique selected from the group consisting of dynamic programming, integer programming and simulated annealing.
21. A method for dynamically materializing a multi-dimensional stream cube having a plurality of cuboids, comprising:
repeatedly computing a cost for materializing each cuboid in the stream cube;
repeatedly computing a cost for each of a plurality of potential materialization paths in the stream cube during a duration of the stream cube;
dynamically selecting a materialization path from the plurality of potential materialization paths in the stream cube that has the lowest computed cost; and
refreshing the stream cube according to the selected materialization path;
wherein:
cuboids in the stream cube are iterated from a first cuboid to a second cuboid to determine a predecessor cuboid for each cuboid,
the predecessor cuboid is associated with a cost value and both predecessor cuboid and the costs value are stored, and
a materialization path is generated by backtracking from the second cuboid to the first cuboid.
22. A method for materializing a multi-dimensional stream cube having a plurality of cuboids, comprising:
generating a stream cube to store data received from multiple data sources;
generating a plurality of multi-dimensional data streams to represent the received data;
repeatedly determining materialization strategies to implement for the stream cube during a duration of the stream cube, wherein at least two different materialization strategies are determined; and
materializing the stream cube according to determined materialization strategies to record the multi-dimensional data streams.
23. A system comprising:
at least one data processor;
memory coupled to the at least one data processor, the memory storing instructions to cause the at least one data processor to perform operations comprising:
generating a stream cube to store data received from multiple data sources;
generating a plurality of multi-dimensional data streams to represent the received data;
repeatedly determining materialization strategies to implement for the stream cube during a duration of the stream cube, wherein at least two different materialization strategies are determined; and
materializing the stream cube according to determined materialization strategies to record the multi-dimensional data streams.