System and method for analyzing and reporting extensible data from multiple sources in multiple formats
A system and method for analyzing and reporting data from multiple sources is provided. The system is a foundation for an analytical platform that covers not only traditional relational data, but also a new generation of extensible data formats designed for the web, such as those based on XML (FIXML, FpML, ebXML, XBRL, ACORD, etc.), as well as HTML, E-mail, Excel, PDF, and others. In a preferred embodiment, the eXtensible on-line analytical processing (XOLAP), is a scalable client/server platform that allows the multi-dimensional analysis of modern data types, as well as traditional relational data, by bringing them all into an internal common XML-based model, without the time and expense of creating a data warehouse.
1. An extended online analytical apparatus for one or more data sources wherein each data source has a data format, comprising:
one or more adapters, executable on a processor, coupled to memory, and configured to convert one or more pieces of data from a plurality of data sources, wherein the plurality of data sources includes at least relational data sources and non-relational data sources;
an aggregator, executable on the processor and configured to aggregate the one or more pieces of data from the one or more data sources to generate aggregated pieces of data;
a virtual schema generator executable on the processor and configured to generate a XOLAP virtual schema from the aggregated pieces of data without using a data warehouse and a data warehouse model, wherein the XOLAP virtual schema is an Extensible Markup Language (XML) based multidimensional index of the plurality of data sources and is updated incrementally by virtually transforming the aggregated pieces of data using one or more virtual schema operators;
an analytical cube generator executable on the processor and configured to generate a multidimensional analytical cube populated with the aggregated pieces of data from the plurality of data sources indexed by the XOLAP virtual schema and by executing commands from a declarative language against the XOLAP virtual schema;
a navigator executable on the processor and configured to generate one or more reports based on a subset of data in the multidimensional analytical cube.
2. The apparatus of claim 1 wherein the one or more virtual schema operators comprises one or more of the following: selection, linking, renaming and type conversion.
3. The apparatus of claim 1 wherein the XOLAP virtual schema is eXtensible Markup Language (XML) based and obtains data from the plurality of data sources utilizing existing data structures of the plurality of data sources.
4. The apparatus of claim 1 , wherein each adaptor of the one or more adapters is configured to convert one or more pieces of data from a particular data source having a particular data format into a common data format, and wherein the aggregator is configured to use the converted one or more pieces of data to generate the aggregated pieces of data.
5. The apparatus of claim 1 further comprising an indexing engine and a scheduler configured to update the XOLAP virtual schema when new pieces of data are received by the aggregator.
6. The apparatus of claim 1 further comprising an application programming interface to an external tool, wherein the application programming interface is configured to permit the external tool to access the piece of data in the analytical cube.
7. The apparatus of claim 1 , wherein the XOLAP virtual schema obtains data from the plurality of data sources while maintaining data physically at the plurality of data sources.
8. A computer-implemented method for generating dimensional analytical models from extensible data, the method comprising:
acquiring, at a server, via one or more adapters one or more pieces of data from a plurality of data sources, wherein the plurality of data sources includes at least relational data sources and non-relational data sources;
generating, using the server, a XOLAP virtual schema from the aggregated pieces of data without using a data warehouse and without a data warehouse model, wherein the XOLAP virtual schema is an Extensible Markup Language (XML) based multidimensional index of the plurality of data sources and is updated incrementally by virtually transforming the aggregated pieces of data using one or more virtual schema operators;
generating, using the server, a multidimensional analytical model populated with the aggregated pieces of data from the plurality of data sources indexed by the XOLAP virtual schema; and
wherein forming the multidimensional analytical model further comprises using a declarative language to describe the multidimensional analytical model by executing commands from the declarative language against the XOLAP virtual schema; and
generating, using the server, one or more reports based on a subset of data in the multidimensional analytical model.
9. The method of claim 8 , wherein using the declarative language further comprises using xPath to describe the multidimensional analytical model by executing commands from the declarative language against the XOLAP schema.
10. The method of claim 8 further comprising performing path analysis using the multidimensional analytical model, wherein the path analysis is based on the declarative language.
11. The method of claim 10 , wherein the one or more virtual schema operators comprises one or more of the following: selection, linking, renaming and type conversion.
12. The method of claim 8 , wherein the XOLAP virtual schema is eXtensible Markup Language (XML) based and obtains data from the plurality of data sources utilizing existing data structures of the plurality of data sources.
13. The method of claim 8 , further comprising determining an aggregated value for a particular piece of data from the one or more pieces of data wherein a first set of pieces of data has a first hierarchical structure that contains the particular piece of data and a second set of pieces of data has a second hierarchical structure that contains the particular piece of data at a different location than the first hierarchical structure, and wherein determining the aggregated value comprises extracting the particular piece of data from each hierarchical structure using the multidimensional analytical model.
14. The method of claim 8 , further comprising converting, using the server, the one or more pieces of data from the plurality of data sources into a common data format.
15. The method of claim 8 , wherein the XOLAP virtual schema obtains data from the plurality of data sources while maintaining data physically at the plurality of data sources.
16. A non-transitory computer readable medium that stores a set of instructions that are executable by at least one or more processors of a server to cause the server to perform a method for generating dimensional analytical models from extensible data, the method comprising:
acquiring via one or more adapters one or more pieces of data from a plurality of data sources, wherein the plurality of data sources includes at least relational data sources and non-relational data sources;
generating a XOLAP virtual schema from the aggregated pieces of data without using a data warehouse and without a data warehouse model, wherein the XOLAP virtual schema is an Extensible Markup Language (XML) based multidimensional index of the plurality of data sources and is updated incrementally by virtually transforming the aggregated pieces of data using one or more virtual schema operators;
generating a multidimensional analytical model populated with the aggregated pieces of data from the plurality of data sources indexed by the XOLAP virtual schema for the one or more pieces of data from the one or more data sources, wherein forming the multidimensional analytical model further comprises using a declarative language to describe the multidimensional analytical model by executing commands from the declarative language against the XOLAP virtual schema; and
generating one or more reports based on a subset of data in the multidimensional analytical model.
17. The non-transitory computer readable medium of claim 16 , wherein using the declarative language further comprises using xPath to describe the multidimensional analytical model by executing commands from the declarative language against the XOLAP virtual schema.
18. The method of claim 16 , wherein the set of instructions that is executable by the at least one or more processors of the server to cause the server to further perform:
performing path analysis using the multidimensional analytical model, wherein the path analysis is based on the declarative language.
19. The non-transitory computer readable medium of claim 16 , wherein the one or more virtual schema operators comprises one or more of the following: selection, linking, renaming and type conversion.
20. The non-transitory computer readable medium of claim 16 , wherein the set of instructions that is executable by the at least one or more processors of the server to cause the server to further perform:
identifying, using the multidimensional analytical model, co-occurrences between the one or more pieces of data from the plurality of data sources.
21. The non-transitory computer readable medium of claim 16 , wherein the set of instructions that is executable by the at least one or more processors of the server to cause the server to further perform:
determining an aggregated value for a particular piece of data from the one or more pieces of data wherein a first set of pieces of data has a first hierarchical structure that contains the particular piece of data and a second set of pieces of data has a second hierarchical structure that contains the particular piece of data at a different location than the first hierarchical structure, and wherein determining the aggregated value comprises extracting the particular piece of data from each hierarchical structure using the multidimensional analytical model.
22. The non-transitory computer readable medium of claim 16 , wherein the XOLAP virtual schema is eXtensible Markup Language (XML) based and obtains data from the plurality of data sources utilizing existing data structures of the plurality of data sources.
23. The non-transitory computer readable medium of claim 16 , the XOLAP virtual schema obtains data from the plurality of data sources while maintaining data physically at the plurality of data sources.