Rule-based, event-driven, scalable data collection

1. A method for collecting data from one or more data sources, into which data is continuously stored, and for loading the collected data into one or more data targets, the method including:

defining one or more events;

defining one or more rules associated with each event;

when one of the one or more events occurs,

causing unmarked data to be collected from the one or more data sources in accordance with the rules associated with the one event;

marking the collected, unmarked data in the one or more data sources to indicate that it has been collected; and

causing the collected data to be stored in one or more of the data targets.

2. The method of claim 1 where defining one or more events includes:

predefining as events specific changes in the state of a given data source.

3. The method of claim 1 where defining one or more rules includes:

identifying one or more ranges of one or more parameters associated with the events generated for one or more data sources, such that if the one or more parameters associated with the data in the one or more data sources fall within the one or more ranges, a condition of a rule is satisfied; and

predefining specific courses of action to follow when the condition of the rule is satisfied.

4. The method of claim 3 where identifying parameters includes:

identifying information at one or more of the following levels: the application level, the program level, and the performance level.

5. The method of claim 1 where causing unmarked data to be collected includes:

identifying, according to the rules associated with the one event, the data to be collected; and

collecting the unmarked data from the identified data to be collected.

6. The method of claim 1 where marking the collected, unmarked data includes:

setting a flag associated with the collected, unmarked data.

7. The method of claim 1 where causing the collected data to be stored includes:

providing the collected data to a consumer operator which stores the collected data in the one or more data targets.

8. The method of claim 1 further including:

providing a plurality of producer operators, each producer operator being responsible for collecting data in response to a subset of the events.

9. The method of claim 1 further including:

providing a plurality of producer operators, each producer operator being responsible for collecting data from a subset of the data sources.

10. The method of claim 1 where defining one or more rules includes:

retrieving metadata associated with the one or more data sources.

11. The method of claim 1 further including:

detecting an event by retrieving metadata associated with the one or more data sources and monitoring the state of the one or more data sources based on the parameters specified in the metadata.

12. The method of claim 1 further including:

revising the method based on retrieved metadata associated with the one or more data sources.

13. A computer system, for use in collecting data from one or more data sources, into which data is continuously stored, and for use in loading the collected data into one or more data targets, the computer system including:

one or more nodes;

a plurality of CPUs, each of the one or more nodes providing access to the one or more CPUs;

a plurality of data storage devices, each of the one or more CPUs providing access to a subset of the plurality of data storage devices;

each of the one or more data sources being distributed across the plurality of data storage devices;

a plurality of processes, each of the processes being associated with a subset of the nodes, a subset of the CPUs and a subset of the data storage devices, each of the plurality of processes:

defining one or more events;

defining one or more rules associated with each event;

when one of the one or more events occurs,

causing unmarked data to be collected from the portion of the one or more data sources distributed to the nodes, CPUs, and data storage devices associated with the process, the collection being performed in accordance with the rules associated with the one event; and

marking the collected, unmarked data in the portion of the one or more data sources distributed to the CPUs and data storage devices associated with the process to indicate that it has been collected;

causing the collected data to be stored in one or more of the data targets.

14. The computer system of claim 13 further including

each of the one or more data targets being distributed across the plurality of data storage devices; and where

causing the collected data to be stored in one or more of the data targets includes:

causing the collected data to be stored in the portion of the one or more data targets associated with the process.

15. The computer system of claim 13 where defining one or more events includes:

predefining as events specific changes in the state of a given data source.

16. The computer system of claim 13 where defining one or more rules includes:

identifying one or more ranges of one or more parameters associated with the events generated for one or more data sources, such that if the one or more parameters associated with the data in the one or more data sources fall within the one or more ranges, a condition of a rule is satisfied; and

predefining specific courses of action to follow when the condition of the rule is satisfied.

17. The computer system of claim 16 where identifying parameters includes:

identifying information at one or more of the following levels: the application level, the program level, and the performance level.

18. The computer system of claim 13 where causing unmarked data to be collected includes:

identifying, according to the rules associated with the one event, the data to be collected; and

collecting the unmarked data from the identified data to be collected.

19. The computer system of claim 13 where marking the collected, unmarked data includes:

setting a flag associated with the collected, unmarked data.

20. The computer system of claim 13 where causing the collected data to be stored includes:

providing the collected data to a consumer operator which stores the collected data in the one or more data targets.

21. The computer system of claim 13 further including:

providing a plurality of producer operators, each producer operator being responsible for collecting data in response to a subset of the events.

22. The computer system of claim 13 further including:

providing a plurality of producer operators, each producer operator being responsible for collecting data from a subset of the data sources.

23. The computer system of claim 13 where defining one or more rules associated with each event includes:

retrieving metadata associated with the one or more data sources.

24. The computer system of claim 13 where each of the plurality of processes further includes:

detecting an event by retrieving metadata associated with the one or more data sources and monitoring the state of the one or more data sources based on the parameters specified in the metadata.

25. The computer system of claim 13 where each of the plurality of processes further includes:

revising the computer system based on retrieved metadata associated with the one or more data sources.

26. A computer program, stored on a tangible storage medium, for use in collecting data from one or more data sources, into which data is continuously stored, and loading the collected data into one or more data targets, the program including executable instructions that cause a computer to:

define one or more events;

define one or more rules associated with each event;

when one of the one or more events occurs,

cause unmarked data to be collected from the one or more data sources in accordance with the rules associated with the one event;

mark the collected, unmarked data in the one or more data sources to indicate that it has been collected; and

cause the collected data to be stored in one or more of the data targets.

27. The computer program of claim 26 where, when defining one or more events, the computer:

predefines as events specific changes in the state of a given data source.

28. The computer program of claim 26 where, when defining one or more events, the computer:

identifies one or more ranges of one or more parameters associated with the events generated for one or more data sources, such that if the one or more parameters associated with the data in the one or more data sources fall within the one or more ranges, a condition of a rule is satisfied; and

predefines specific courses of action to follow when the condition of the rule is satisfied.

29. The computer program of claim 28 where, when identifying parameters, the computer:

identifies information at one or more of the following levels: the application level, the program level, and the performance level.

30. The computer program of claim 26 where, when causing unmarked data to be collected, the computer:

identifies, according to the rules associated with the one event, the data to be collected; and

collects the unmarked data from the identified data to be collected.

31. The computer program of claim 26 where, when marking the collected, unmarked data, the computer:

sets a flag associated with the collected, unmarked data.

32. The computer program of claim 26 where, when causing the collected data to be stored, the computer:

provides the collected data to a consumer operator which stores the collected data in the one or more data targets.

33. The computer program of claim 26 further including executable instructions that cause the computer to:

provide a plurality of producer operators, each producer operator being responsible for collecting data in response to a subset of the events.

34. The computer program of claim 26 further including executable instructions that cause the computer to:

provide a plurality of producer operators, each producer operator being responsible for collecting data from a subset of the data sources.

35. The computer program of claim 26 where, when defining one or more rules, the computer:

retrieves metadata associated with the one or more data sources.

36. The computer program of claim 26 further including executable instructions that cause the computer to:

detect an event by retrieving metadata associated with the one or more data sources.

37. The computer program of claim 26 further including executable instructions that cause the computer to:

revise the computer program based on retrieved metadata associated with the one or more data sources.