Apparatus and method for accessing materialized and non-materialized values in a shared nothing system
View Patent ↗A computer readable storage medium includes executable instructions to define an interface to a multi-dimensional cube divided into partitions distributed across nodes supporting a shared nothing multi-dimensional distributed database. The interface includes a metadata module characterizing access paths to data within the multi-dimensional cube and an access path module to convert a data request into an access path protocol to retrieve data from the multi-dimensional cube to satisfy the data request.
1. A method, performed by a computer system having one or more processors, for accessing tuples, each tuple having multiple dimensions, each dimension having one or more dimension levels with some of the dimension levels designated as aggregation levels, each dimension level having values, one of the values of each dimension level being an aggregation value, the tuples include non-aggregated tuples and aggregate tuples, an aggregated tuple having the aggregation value for a dimension level, the method comprising:
generating metadata mapping each combination of values of the aggregation levels to one of a plurality of partitions such that at least two combinations of values of the aggregation levels are mapped to different partitions, each partition being stored on a different node of a cluster of nodes;
materializing aggregated tuples from non-aggregated tuples;
storing each materialized aggregated tuple in the partition to which the combination of values of the aggregation levels of the tuple maps to as indicated by the metadata;
after storing the materialized aggregated tuples, receiving a query specifying a value for each dimension level;
when the query does not include an aggregation value for a dimension level that is not an aggregation level, retrieving as the query result the tuple that matches the query from the partition to which the combination of values of the aggregation levels of the query maps to as indicated by the metadata; and
when the query does include an aggregation value for a dimension level that is not an aggregation level, materializing as the query result a non-aggregated tuple from tuples stored in the partition to which the combination of values of the aggregation levels of the query maps to as indicated by the metadata
wherein materialized aggregated tuples are stored in different partitions so that materialized aggregated tuples that match queries can be retrieved from different partitions stored at different nodes to distribute load of the retrieval among different partitions.
2. The method of claim 1 , further comprising:
expanding hierarchical operators associated with the query.
3. The method of claim 2 wherein the hierarchical operators comprise at least one previous operator.
4. The method of claim 2 wherein the hierarchical operators comprise at least one next operator.
5. The method of claim 2 , wherein each hierarchical operator is fulfilled by looking up metadata and without running large scale scans to resolve the hierarchical operators.
6. The method of claim 1 , further comprising:
expanding children operators associated with the query; and
expanding parent operators associated with the query.
7. The method of claim 1 further comprising caching the query result locally at the computer so that when the same query is later received, retrieving the cached query result without retrieving the query result from the partition.
8. The method of claim 1 further comprising deferring the materializing and storing of an aggregated tuple until a query is received for that aggregated tuple.
9. The method of claim 1 further comprising when the query includes a .CHILDREN operator for an aggregation level, generating an expanded query for each value of the aggregation level.
10. The method of claim 9 including retrieving or materializing the tuple that matches each expanded query and combining the tuples as the query result.
11. The method of claim 1 further comprising when the query includes a .CHILDREN operator for two aggregation levels, generating a query for each possible combination of values of the two aggregation levels.
12. A computer system for accessing tuples, each tuple having multiple dimensions, each dimension having one or more dimension levels with some of the dimension levels designated as aggregation levels, each dimension level having values, one of the values of each dimension level being an aggregation value, the computing system comprising:
a computer-readable medium storing computer-executable instructions of one or more modules that:
generate metadata mapping each combination of values of the aggregation levels to one of a plurality of partitions such that at least two combinations of values of the aggregations levels are mapped to different partitions that are stored at different nodes;
for each combination of values of the aggregation levels,
when the combination of values does not include an aggregation value, store a non-aggregated tuple with that combination of values in the partition to which the combination of values maps to as indicated by the metadata;
when the combination of values does include an aggregation value, materialize a aggregated tuple by:
generating the aggregated tuple with that combination of values from the non-aggregated tuples; and
storing the aggregated tuple with that combination of values in the one partition of the plurality of partitions to which the combination of values maps to as indicated by the metadata;
receive a query specifying a value for each dimension level;
when the query does not include an aggregation value for a dimension level that is not an aggregation level, retrieve as the query result the tuple that matches the query from the one partition of the plurality of partitions to which the combination of values of the aggregation levels of the query maps to as indicated by the metadata; and
when the query does include an aggregation value for a dimension level that is not an aggregation level, materialize as the query result a non-aggregated tuple from tuples stored in the partition to which the combination of values of the aggregation levels of the query maps to as indicated by the metadata; and
a processor for executing the computer-executable instruction stored in the medium.
13. The computer system of claim 12 wherein the computer-executable instructions of the one or more modules further cache the query result locally at the computer so that when the same query is later received, the cached query result is retrieved without retrieving the query result from the partition.
14. The computer system of claim 12 wherein the computer-executable instructions of the one or more modules further deferring the materializing and storing of an aggregated tuple until a query is received for that aggregated tuple.
15. The computer system of claim 12 wherein the computer-executable instructions of the one or more modules further when the query includes a .CHILDREN operator for an aggregation level, generate an expanded query for each value of the aggregation level.
16. The computer system of claim 15 wherein the computer-executable instructions of the one or more modules further retrieve or materialize the tuple that matches each expanded query and combine the tuples as the query result.
17. The computer system of claim 12 wherein the computer-executable instructions of one or more modules further when the query includes a .CHILDREN operator for two aggregation levels, generate a query for each possible combination of values of the two aggregation levels.
18. A hardware memory storing computer-executable instructions for controlling a processor to access tuples, the computer-executable instructions comprising:
instructions that access metadata that maps each combination of values of aggregation levels to one of a plurality of partitions, wherein each tuple has dimensions with a dimension level, a dimension level being designated as an aggregation level, each dimension level having values, one of the values of the dimension level being an aggregation value, the tuples include non-aggregated tuples and aggregated tuples, an aggregated tuple having the aggregation value for the dimension level, each partition being stored at a different node;
instructions that access partitions storing aggregated tuples pre-materialized from non-aggregated tuples, each pre-materialized aggregated tuple stored in a partition to which the combination of values of the aggregation levels of the tuple maps to as indicated by the metadata;
instructions that, when a query does not include an aggregation value for a dimension level that is not an aggregation level, retrieve as the query result the tuple that matches the query from the partition to which the combination of values of the aggregation levels of the query maps to as indicated by the metadata wherein the instructions that retrieve are executed on the node that stores the partition to which the combination of values of the aggregation levels of the query maps to so that the instructions can execute in parallel at multiple nodes; and
instructions that, when a query does include an aggregation value for a dimension level that is not an aggregation level, materializing as the query result a non-aggregated tuple from tuples stored in the partition to which the combination of values of the aggregation levels of the query maps to as indicated by the metadata.
19. The hardware memory of claim 18 wherein the computer-executable instructions further comprise instructions that cache the query result locally at the computer so that when a query is subsequently processed, the cached query result can be retrieved without retrieving the query result from the partition.
20. The hardware memory of claim 18 wherein the computer-executable instructions further comprise instructions that materialize and store of an aggregated tuple.
21. The hardware memory of claim 20 wherein the instructions defer the materializing and storing of an aggregated tuple until a query is processed for that aggregated tuple.
22. The hardware memory of claim 18 wherein the computer-executable instructions further comprise instructions that, when the query includes a .CHILDREN operator for an aggregation level, generate an expanded query for each value of the aggregation level.
23. The hardware memory of claim 22 wherein the computer-executable instructions further comprise instructions that retrieve or materialize the tuple that matches each expanded query and combine the tuples as the query result.
24. The hardware memory of claim 18 wherein the computer-executable instructions further comprise instructions that, when the query includes a .CHILDREN operator for two aggregation levels, generate a query for each possible combination of values of the two aggregation levels.