Data management and processing system for large enterprise model and method therefor
A computer system performs data processing using a data organizer which parses a primary set of data into predefined sets of data components according to one or more enterprise models. The sets of data components from the data organizer are respectively stored in multiple data sources. The data components may be stored as a binary file or hierarchical file. A plurality of data processing systems execute the enterprise models. A task manager controls which enterprise model executes on each data processing system. The data processing systems access the data components from the data sources which correspond the executing enterprise model. The data components are sent over dedicated communication channels to the data processing systems. Each data processing system may have a data interface, which is configured to receive data from different sources and convert the data into a standard format.
1. A computer system for performing statistical modeling using transactional data from retail sales, comprising:
a data organizer coupled for receiving a primary set of transactional data from retail sales, the primary set of transactional data including a plurality of line items for each item of a plurality of retail transactions, each line item including a store number, product number, time of transaction, transaction number, quantity, current price, and customer number, wherein the data organizer parses the primary set of transactional data into predefined sets of data components according to processing requirements of a plurality of statistical models, each of the predefined sets of data components containing data for executing one of the plurality of statistical models;
a plurality of data sources coupled to an output of the data organizer, wherein each of the plurality of data sources stores at least one of the predefined sets of data components from the data organizer in a binary file structure, the predefined sets of data components including a data component having a hierarchical data structure;
a plurality of data processing systems each executing at least one of the plurality of statistical models, each of the plurality of statistical models using a single one of the predefined sets of data components to perform statistical modeling; and
a plurality of communication channels respectively coupled between the plurality of data sources and the plurality of data processing systems, each of the communication channels for routing one of the predefined sets of data components to the data processing system which executes the corresponding statistical model, wherein each of the predefined sets of data components includes only data which is used by the corresponding one of the plurality of statistical models.
2. The computer system of claim 1 , wherein a first one of the plurality of statistical models uses a first predefined set of data components.
3. The computer system of claim 2 , wherein the data organizer extracts the first predefined set of data components from the primary set of transactional data.
4. The computer system of claim 3 , wherein the first predefined set of data components is stored on a first one of the plurality of data sources.
5. The computer system of claim 4 , wherein the first predefined set of data components is organized on the first one of the plurality of data sources as a data structure in computer memory.