System with a data aggregation module generating aggregated data for responding to OLAP analysis queries in a user transparent manner
View Patent ↗A system for supporting OLAP analysis over a network. The system comprises an OLAP server for enabling an OLAP user to perform OLAP analysis via interaction with a client machine on the network. The system also includes a data aggregation module comprising a multi-dimensional datastore, an aggregation engine integrated with the multi-dimensional datastore, and a first interface for loading base data from a data source to the aggregation engine. The aggregation engine performs data aggregation operations on loaded base data, generates aggregated data from the base data, and stores the aggregated data in the multi-dimensional datastore. A second interface receives requests for OLAP analysis from the OLAP server, accesses the aggregation engine to retrieve from the multi-dimensional datastore, aggregated data corresponding to requests, and communicates the retrieved aggregated data to the OLAP server for query servicing, in a manner transparent to the OLAP user.
1. A database management system comprising:
a multi-dimensional data store; and
a computer system comprising computer hardware, the computer system communicatively coupled to the multi-dimensional data store and programmed to implement:
a query servicing mechanism configured to service one or more natural language queries, the query servicing mechanism comprising:
a query processing mechanism configured to process a given natural language query using operations to:
receive a first hierarchical database structure defining parent-child relationships of levels within dimensions;
convert the first hierarchical database structure into a functionally equivalent hierarchical database structure enhanced for rapid aggregation relative to the first hierarchical database structure;
aggregate data in the multi-dimensional data store according to the functionally equivalent hierarchical database structure; and
access and forward the aggregated data to the query processing mechanism for use in servicing the given natural language query.
2. The database management system of claim 1 , further comprising a load and indexing module configured to organize the multi-dimensional data store into autonomic segments capable of being rolled up in different sequences according to the functionally equivalent hierarchical database structure, the autonomic segments stored as records indexed such that a given data segment is capable of being independently loaded into a main memory and the given data segment has a size that is comparatively small compared to a maximum data size of available disk memory space.
3. The database management system of claim 2 , wherein the query processing mechanism is further configured to:
aggregate the data on-the-fly by determining a roll-up order based at least in part on previously pre-aggregated data and loading into the main memory a set of memory data segments having data points to perform aggregation on-the-fly to service the given natural language query.
4. The database management system of claim 1 , wherein the functionally equivalent hierarchical database structure is functionally equivalent to multiple hierarchies of the first hierarchical database structure in at least one dimension.
5. The database management system of claim 1 , wherein the functionally equivalent hierarchical database structure reduces data storage requirements relative to the first hierarchical database structure.
6. The database management system of claim 1 , wherein the query processing mechanism is further configured to verify an integrity of the functionally equivalent hierarchical database structure, identify a first error if the first error is present, and report the identified first error.
7. The database management system of claim 1 , wherein the query processing mechanism is further configured to verify an integrity of the functionally equivalent hierarchical database structure, identify a first error if the first error is present, and fix the identified first error.
8. The database management system of claim 1 , wherein the query processing mechanism is further configured to perform a segmented aggregation process based at least in part on the functionally equivalent hierarchical database structure in which data is segmented into segments capable of being rolled up in different rollup orders.
9. The database management system of claim 8 , wherein the query processing mechanism is further configured to perform a partial pre-aggregation according to the functionally equivalent hierarchical database structure to generate an initial set of segments stored as data records in a memory space of the multi-dimensional data store.
10. A database management system comprising:
a computer system comprising computer hardware, the computer system programmed to implement:
a query servicing mechanism configured to service one or more natural language queries, the query servicing mechanism comprising:
a query processing mechanism configured to process a given natural language query using operations to:
receive a hierarchical database structure;
perform a partial pre-aggregation according to an aggregation process to generate an initial set of segments stored as data records in a multi-dimensional data store;
upon determining that servicing a given natural language query needs data stored in the multi-dimensional data store:
determine a rollup order of a subset of segments to service the given natural language query based at least in part on the initial set of segments;
load, into a main memory, records corresponding to the subset of segments;
perform the rollup to aggregate data; and
forward the aggregated data to the query processing mechanism for use in servicing the given natural language query.
11. The database management system of claim 10 , wherein an individual segment corresponds to a group of related data points.
12. The database management system of claim 10 , wherein an individual segment is a slice of multidimensional data.
13. The database management system of claim 10 wherein the records are indexed by a directory file.
14. A method comprising:
by a computer system comprising computer hardware:
receiving a first hierarchical database structure defining parent-child relationships of levels within dimensions;
converting the first hierarchical database structure into a functionally equivalent hierarchical database structure enhanced for rapid aggregation relative to the first hierarchical database structure;
aggregating data in a multi-dimensional data store according to the functionally equivalent hierarchical database structure; and
accessing and forwarding the aggregated data to a query processing mechanism for use in servicing a given natural language query.
15. The method of claim 14 , further comprising organizing the multi-dimensional data store into autonomic segments capable of being rolled up in different sequences according to the functionally equivalent hierarchical database structure, the autonomic segments stored as records indexed such that a given data segment is capable of being independently loaded into a main memory and the given data segment has a size that is comparatively small compared to a maximum data size of available disk memory space.
16. The method of claim 15 , further comprising aggregating the data on-the-fly by determining a roll-up order based at least in part on previously pre-aggregated data and loading into the main memory a set of memory data segments having data points to perform aggregation on-the-fly to service the given natural language query.
17. The method of claim 14 , wherein the functionally equivalent hierarchical database structure is functionally equivalent to multiple hierarchies of the first hierarchical database structure in at least one dimension.
18. The method of claim 14 , wherein the functionally equivalent hierarchical database structure reduces data storage requirements relative to the first hierarchical database structure.
19. The method of claim 14 , further comprising verifying an integrity of the functionally equivalent hierarchical database structure, identifying a first error if the first error is present, and reporting the identified first error.
20. The method of claim 14 , further comprising verifying an integrity of the functionally equivalent hierarchical database structure, identifying a first error if the first error is present, and fixing the identified first error.
21. The method of claim 14 , further comprising performing a segmented aggregation process based at least in part on the functionally equivalent hierarchical database structure in which data is segmented into segments capable of being rolled up in different rollup orders.
22. The method of claim 21 , further comprising performing a partial pre-aggregation according to the functionally equivalent hierarchical database structure to generate an initial set of segments stored as data records in a memory space of the multi-dimensional data store.
23. A method comprising:
by a computer system comprising computer hardware:
receiving a hierarchical database structure;
performing a partial pre-aggregation according to an aggregation process to generate an initial set of segments stored as data records in a multi-dimensional data store;
upon determining that servicing a given natural language query needs data stored in the multi-dimensional data store:
determining a rollup order of a subset of segments to service the given natural language query based at least in part on the initial set of segments;
loading, into a main memory, records corresponding to the subset of segments;
performing the rollup to aggregate data; and
forwarding the aggregated data to a query processing mechanism for use in servicing the given natural language query.
24. The method of claim 23 , wherein an individual segment corresponds to a group of related data points.
25. The method of claim 23 , wherein an individual segment is a slice of multidimensional data.
26. The method of claim 23 wherein the records are indexed by a directory file.