Data aggregation server supporting rapid query response with sparse multi-dimensional data

1. An aggregation server adapted to handle large data volumes and provide rapid query response with sparse data having a tendency to experience data explosion as the number of dimensions increases, said aggregation server comprising:

an aggregation module adapted to aggregate sparse data, said aggregation module operatively coupleable to a relational data store, said aggregation module including a base data loader to load base data from said relational data store, an aggregation engine to aggregate said base data, and a non-relational multi-dimensional data store for storing base data and aggregated data; and

a controller for servicing query statements at any stage of aggregation, said controller in response to determining that data required to service a given query statement has not been pre-aggregated directs said aggregation engine to perform an aggregation on-the-fly to service the given query statement utilizing a query directed rollup in which a multi-dimensional data space is organized as independent segments to permit different rollup orders to service the given query statement, said controller selecting a rollup order based on the given query statement and any partially pre-aggregated data present in said non-relational multi-dimensional data store to service the given query statement.

2. The aggregation server of claim 1 , wherein said aggregation server stores data aggregated on-the-fly to build up a store of aggregated data for servicing queries until at least a partial pre-aggregation is completed.

3. The aggregation server of claim 1 , wherein said aggregation server selects a rollup order to minimize a processing delay associated with aggregating data on-the-fly to service the given query statement.

4. The aggregation server of claim 1 , wherein said aggregation server selects a rollup order to reduce the amount of simultaneously handled data required to aggregate data on-the-fly to service a given query statement.

5. The aggregation server of claim 1 , further comprising hierarchy logic, said hierarchy logic resolving hierarchies to reduce data handling performed on sparse data.

6. The aggregation server of claim 1 , further comprising a data directory adapted to index sparse data associated with said independent segments for rapid retrieval.

7. The aggregation server of claim 1 , wherein after an initial aggregation step said multi-dimensional data space is organized with said independent segments comprising independent multi-dimensional cubes.

8. The aggregation server of claim 1 , wherein said aggregation server has a mode of operation in which data is aggregated on-the-fly after a partial pre-aggregation of data is performed.

9. The aggregation server of claim 1 , wherein said aggregation server is adapted to perform an aggregation on-the-fly utilizing independent segments corresponding to slices of the multidimensional data that are thin compared with the entire multidimensional data such that data handling required to aggregate data on-the-fly for a given query statement is reduced.

10. The aggregation server of claim 1 , wherein said controller is integrated within said aggregation module.

11. The aggregation server of claim 10 , wherein said aggregation module comprises:

a first data communication interface for said aggregation module to interface with said relational data store and receive said base data loaded into said aggregation module from said relational data store; and

a second data communication interface for said aggregation module to interface with one or more clients having a query interface for converting queries from end-users into said query statements.

12. The aggregation server of claim 11 , wherein said first and second data communication interfaces allow said aggregation module to be installed into a software-based computing environment in a cartridge-like manner.

13. The aggregation server of claim 1 , wherein said aggregation module comprises:

a first data communication interface for said aggregation module to interface with said relational data store and receive said base data loaded into said aggregation module from said relational data store; and

a second data communication interface for said aggregation module to interface with one or more clients having a query interface for converting queries from end-users into said query statements.

14. An aggregation server adapted to handle large data volumes and provide rapid query response for sparse data having a tendency to experience data explosion as the number of dimensions increases, said aggregation server comprising:

an aggregation module adapted to aggregate sparse data, said aggregation module operatively coupleable to a relational data store, said aggregation module including a base data loader to load base data from said relational data store, an aggregation engine to aggregate said base data, and a non-relational multi-dimensional data store for storing base data and aggregated data; and

a controller for servicing query statements at any stage of aggregation, said controller in response to determining that data required to service a given query statement has not been pre-aggregated directs said aggregation engine to perform an aggregation on-the-fly to service the given query statement utilizing a query directed rollup in which a multi-dimensional data space is organized as independent segments to permit different rollup orders to service the given query statement, said controller selecting a rollup order based on the given query statement and any partially pre-aggregated data present in said non-relational multi-dimensional data store to service the given query statement;

wherein each of said independent segments represents only a comparatively thin slice of the total multi-dimensional data space to reduce the amount of data that must be handled to perform an aggregation in selected dimensions with respect to an individual segment.

15. The aggregation server of claim 14 , wherein said aggregation server selects a rollup order to minimize a processing delay associated with aggregating data on-the-fly to service the given query statement.

16. The aggregation server of claim 14 , wherein said aggregation server selects a rollup order to reduce the amount of simultaneously handled data required to aggregate data on-the-fly to service a given query statement.

17. The aggregation server of claim 16 , wherein after an initial aggregation step said independent segments comprise independent multi-dimensional cubes.

18. The aggregation server of claim 16 , wherein said aggregation server has a mode of operation in which data is aggregated on-the-fly after a partial pre-aggregation of data is performed.

19. The aggregation server of claim 14 , wherein said aggregation server stores data aggregated on-the-fly to build up a store of aggregated data in said non-relational multi-dimensional data store to service subsequent query statements.

20. The aggregation server of claim 14 , wherein said aggregation server builds up said store for a system requiring only partial pre-aggregation.

21. The aggregation server of claim 14 , wherein after an initial aggregation step said independent segments comprise independent multi-dimensional cubes.

22. The aggregation server of claim 14 , wherein said aggregation server has a mode of operation in which data is aggregated on-the-fly after a partial pre-aggregation of data is performed.

23. The aggregation server of claim 22 , wherein said aggregation server selects a rollup order to minimize a processing delay associated with aggregating data on-the-fly to service the given query statement.

24. The aggregation server of claim 22 , wherein said aggregation server selects a rollup order to reduce the amount of simultaneously handled data required to aggregate data on-the-fly to service a given query statement.

25. The aggregation server of claim 14 , wherein said aggregation server stores data aggregated on-the-fly to build up a store of aggregated data in said non-relational multi-dimensional data store to service subsequent query statements.

26. The aggregation server of claim 25 , wherein said aggregation server builds up said store for a system requiring only partial pre-aggregation.

27. The aggregation server of claim 14 , wherein said controller is integrated within said aggregation module.

28. The aggregation server of claim 27 , wherein said aggregation module comprises:

a first data communication interface for said aggregation module to interface with said relational data store and receive said base data loaded into said aggregation module from said relational data store; and

a second data communication interface for said aggregation module to interface with one or more clients having a query interface for converting queries from end-users into said query statements.

29. The aggregation server of claim 28 wherein said first and second data communication interfaces allow said aggregation module to be installed into a software-based computing environment in a cartridge-like manner.

30. The aggregation server of claim 14 , wherein said aggregation module comprises:

a first data communication interface for said aggregation module to interface with said relational data store and receive said base data loaded into said aggregation module from said relational data store; and

a second data communication interface for said aggregation module to interface with one or more clients having a query interface for converting queries from end-users into said query statements.

31. An aggregation server adapted to handle large data volumes and provide rapid query response for sparse data having a tendency to experience data explosion as the number of dimensions increases, said aggregation server comprising:

an aggregation module adapted to aggregate sparse data, said aggregation module operatively coupled to a relational data store, said aggregation module including a base data loader to load base data, an aggregation engine to aggregate said base data, and a non-relational multi-dimensional data store for storing base data and aggregated data;

hierarchy logic, said hierarchy logic resolving hierarchies to reduce data handling performed on sparse data;

a controller for servicing query statements at any stage of aggregation, said controller in response to determining that data required to service a given query statement has not been pre-aggregated directs said aggregation engine to perform an aggregation on-the-fly to service the given query statement utilizing a query directed rollup in which a multi-dimensional data space is organized as independent segments to permit different rollup orders to be utilized to service the given query statement, said controller selecting a rollup order based on the given query statement and any partially pre-aggregated data present in said non-relational multi-dimensional data store to service the given query statement; and

a data directory adapted to index sparse data associated with said independent segments for rapid retrieval;

wherein each of said independent segments represents only a comparatively thin slice of the total multi-dimensional data space to reduce the amount of data that must be handled to perform an aggregation in selected dimensions with respect to an individual segment.

32. The aggregation server of claim 31 , wherein said controller is integrated within said aggregation module.

33. The aggregation server of claim 32 , wherein said aggregation module comprises:

a first data communication interface for said aggregation module to interface with said relational data store and receive said base data loaded into said aggregation module from said relational data store; and

a second data communication interface for said aggregation module to interface with one or more clients having a query interface for converting queries from end-users into said query statements.

34. The aggregation server of claim 33 , wherein said first and second data communication interfaces allow said aggregation module to be installed into a software-based computing environment in a cartridge-like manner.

35. The aggregation server of claim 31 , wherein said aggregation module comprises:

a first data communication interface for said aggregation module to interface with said relational data store and receive said base data loaded into said aggregation module from said relational data store; and

a second data communication interface for said aggregation module to interface with one or more clients having a query interface for converting queries from end-users into said query statements.