Methods and apparatus for statistical data analysis and reduction for an enterprise application
The invention provides methods of time-wise data reduction that include the steps of inputting data from a source; summarizing that data according to one or more selected epochs in which it belongs; and generating for each such selected epoch one or more RDF triples characterizing the summarized data. The data source may be, for example, a database, a data stream or otherwise. The selected epoch may be a second, minute, hour, week, month, year, or so forth. The triples may be output in the form of RDF document objects. These can be stored, for example, in a hierarchical data store such as, for example, a WebDAV server. Triples parsed from the document objects may be maintained in a relational store that is organized, for example, according to a hashed with origin approach.
1. A method of time-wise data reduction and storage, comprising
A querying one or more data sources,
B selecting an epoch,
C generating a time-wise statistic based on data received from the data sources in response to querying, where the time-wise statistic is generated over said selected epoch and comprises a running statistic for the data,
D generating for said epoch, one or more RDF triples characterizing the time-wise statistic, and
E storing the RDF triples to one or more data stores, along with further RDF triples characterizing the data from which the summaries were generated, where the one or more data stores include any of a hierarchical data store and a relational data store.
2. The method of claim 1 , comprising generating a time-wise statistic based on data received from the data sources with respect to multiple epochs of differing length.
3. The method of claim 1 , comprising querying one or more data sources in an SQL format.
4. The method of claim 1 , comprising parsing an XML file that identifies one or more of the data sources, one or more fields of data for which time-wise statistics are to be generated, and/or one or more epochs for which those time-wise statistics are to be generated.
5. The method of claim 1 , comprising responding to data received from a data source by updating a store associated with an epoch of shorter duration.
6. The method of claim 5 , comprising updating a store associated with an epoch of longer duration based on information maintained in an epoch of shorter duration.
7. The method of claim 1 , comprising outputing the one or more RDF triples in one or more RDF document objects.
8. The method of claim 1 , comprising storing the RDF triples in a relational data store organized according to a hashed with origin approach.
9. A method of time-wise data reduction and storage, comprising
A at least one of querying and filtering data from one or more data sources,
B selecting one or more epoch,
C generating a time-wise statistic for data received in said selected epochs, the time-wise statistic comprising a running statistic for the data, and
D generating RDF document objects comprising one or more RDF triples characterizing the time-wise statistic,
E storing the RDF documents to a first, hierarchical data store, and
F storing the triples therein to a second, relational data store.
10. The method of claim 9 , comprising querying one or more data sources in an SQL format.
11. The method of claim 9 , comprising parsing an XML file that identifies one or more of the data sources, one or more fields of data for which time-wise statistics are to be generated, and/or one or more epochs for which those time-wise statistics are to be generated.
12. The method of claim 9 , comprising responding to data received from a data source by updating a store associated with an epoch of shorter duration.
13. The method of claim 12 , comprising updating a store associated with an epoch of longer duration based on information maintained in an epoch of shorter duration.
14. The method of claim 9 , comprising generating a display or other presentation based on the RDF triples characterizing the time-wise statistic.
15. The method of claim 9 , wherein the selection of one or more epochs comprises selecting two or more epochs of differing lengths.
16. The method of claim 9 , comprising storing the RDf document objects in accord with a WebDAV protocl.