Software and metadata structures for distributed and interactive database architecture for parallel and asynchronous data processing of complex data and for real-time query processing
The various embodiments of the invention provide software and metadata structures for a data processing system and method, for applications such as marketing campaign management, speech recognition and signal processing. An exemplary system embodiment includes a first data repository adapted to store a plurality of entity and attribute data; a second data repository adapted to store a plurality of entity linkage data; a metadata data repository adapted to store a plurality of metadata modules, with a first metadata module having a plurality of selectable parameters, received through a control interface, and having a plurality of metadata linkages to a first subset of metadata modules; and a multidimensional data structure. The control interface may modify the plurality of selectable parameters in response to received control information. A plurality of processing nodes are adapted to use the plurality of selectable parameters to assemble a first plurality of data from the first and second data repositories and from input data, to reduce the first plurality of data to form a second plurality of data, and to aggregate and dimension the second plurality of data for storage in the multidimensional data structure.
1. A tangible medium storing computer readable software for programming a parallel and asynchronous database architecture and data processing system for marketing campaign management and analysis, the tangible medium storing computer readable software comprising:
a first program construct of a plurality of program constructs, the first program construct comprising a plurality of modifiable parameters adapted to be modifiable during data processing, the first program construct further defining a plurality of selectable input data sources and a plurality of selectable attributes;
a second program construct of the plurality of program constructs, the second program construct having a corresponding metadata linkage to the first program construct, the second program construct defining a plurality of data structures for a plurality of data repositories of the computer system, the plurality of data structures comprising at least one relational data structure and at least one multidimensional data structure;
a third program construct of the plurality of program constructs, the second program construct having corresponding metadata linkages to the first and second program constructs, the third program construct adapted to provide for execution of a plurality of data processes using the plurality of modifiable parameters, the plurality of input data sources and the plurality of selectable attributes, the third program construct further adapted to store a plurality of results of the plurality of data processes in the plurality of data structures; and
a fourth program construct of the plurality of program constructs, the fourth program construct having corresponding metadata linkages to the first, second and third program constructs, the fourth program construct adapted to use the plurality of selectable attributes for a dimensioning and aggregating data for storage in the multidimensional data structure.
2. The tangible medium of claim 1 , wherein:
a first subset of the plurality of data processes comprises at least two data assembly processes of the following group of data assembly processes: a data input process, a data layout process, a data conversion process, a data correction process, a data audit process, a data matching process, a linkage assignment process, a linkage selection process, a data coding process, a data enhancement process, a data change detection process, or a data update process; and
a second subset of the plurality of data processes comprises at least one data reduction process of the following group of data reduction processes: a data netdown process, a data suppression process, or a data prioritization process.
3. The tangible medium of claim 2 , wherein the data assembly processes and data reduction processes correspondingly provide for data assembly and data reduction on the basis of the plurality of selectable attributes.
4. The tangible medium of claim 1 , wherein the plurality of data processes are adapted to execute in parallel and asynchronously using a plurality of data processing nodes of the data processing system.
5. The tangible medium of claim 4 , wherein the third program construct is further adapted to provide for storage in a memory or in a database of a corresponding plurality of interim results from the asynchronous and parallel execution of the plurality of data processes.
6. The tangible medium of claim 5 , wherein the third program construct is further adapted to provide for the storage of the corresponding plurality of interim results as a corresponding set of unique identifiers.
7. The tangible medium of claim 1 , further comprising:
a fifth program construct of the plurality of program constructs, the fifth program construct having a corresponding metadata linkage to the third program construct, the fifth program construct adapted to track each data process invocation of a plurality of data process invocations.
8. The tangible medium of claim 7 , further comprising:
a sixth program construct of the plurality of program constructs, the sixth program construct having a corresponding metadata linkage to the fifth program construct, the sixth program construct adapted to associate a cost factor and a usage factor with each data process invocation of a plurality of data process invocations.
9. The tangible medium of claim 1 , further comprising:
a seventh program construct of the plurality of program constructs, the seventh program construct adapted to determine a plurality of sets resulting from the dimensioned and aggregated data in the multidimensional data structure and to perform a plurality of set operations on the plurality of sets, the plurality of set operations comprising union operations, difference operations, and intersection operations.
10. The tangible medium of claim 1 , wherein the plurality of selectable input data sources comprises a plurality of databases and marketing lists containing a plurality of attributes, the plurality of attributes comprising at least two attribute types of the following group of attribute types: a name attribute, an address attribute, a demographic attribute, a behavioral attribute, or an attitudinal attribute.
11. The tangible medium of claim 1 , wherein the plurality of modifiable parameters further define a plurality of selectable data enhancements.
12. The tangible medium of claim 1 , wherein the fourth program construct is further adapted to use the multidimensional data structure to create a response to an input query.
13. The tangible medium of claim 1 , further comprising:
an eighth program construct adapted to determine a plurality of available variations of at least one of the following: selectable data processes, selectable data attributes, or selectable data segmentations.
14. The tangible medium of claim 1 , further comprising:
a ninth program construct adapted to provide data modeling and data analysis using the plurality of modifiable parameters, the plurality of selectable input data sources, and the plurality of selectable attributes.