Apparatus and method for pipelined event processing in a distributed environment

1. A method, comprising:

receiving first data from a first data source that continuously generates a first set of records with first common fields, wherein the first set of records characterize a first real-lime event,

filtering the first data by selecting a first sub-set of fields of the first common fields to form first filtered data, wherein filtering the first data includes accessing in dynamic memory the first data stored as in-flight data;

receiving second data from a second data source that continuously generates a second set of records with second common fields, wherein the second set of records characterize a second real-time event different than the first real-time event,

filtering the second data by selecting a second sub-set of fields of the second common fields to form second filtered data, wherein filtering the second data includes accessing in dynamic memory the second data stored as in-flight data, wherein the first filtered data and the second filtered data are generated simultaneously by separate nodes in a network and reside in dynamic memory as in-flight data;

applying rules to the first filtered data and the second filtered data at the separate nodes in the network in real-time to identify selected real-time events stored in dynamic memory as in-flight data; and

reporting the selected real-time events prior to persistently storing the first filtered data, the second filtered data and the selected real-time events.

2. The method of claim 1 wherein the first filtered data and the second filtered data are each represented as an object with a unique global identifier field, a creation time field and a payload field.

3. The method of claim 2 wherein the first filtered data and the second filtered data are each represented as an object with a location field.

4. The method of claim 2 wherein the first filtered data and the second filtered data are each represented as an object with a context field defining one or more key value pairs.

5. The method of claim 2 wherein the first filtered data and the second filtered data are each represented as an object with an event field defining one or more events.

6. The method of claim 1 further comprising persistently storing the first data, the second data, the first filtered data, the second filtered data and the selected real-time events to form aggregated data.

7. The method of claim 6 further comprising querying the aggregated data.

8. The method of claim 6 further comprising continuously loading the aggregated data in a Not Only SQL database.

9. The method of claim 1 further comprising forming a new dimension of data based upon selected fields of the first filtered data and the second filtered data.

10. The method of claim 1 wherein the first data source and the second data source are selected from a database log, database redo log, operating system log, application log, web server log, application server log, machine generated log and a sensor.

11. A system, comprising:

a first data source that continuously generates a first set of records with first common fields; wherein the first set of records characterize a first real-time event;

a second data source that continuously generates a second set of records with second common fields, wherein the second set of records characterize a second real-time event different than the first real-time event;

a first data processor to filter the first data by accessing the first data stored in dynamic memory as in-flight data and selecting a first sub-set of fields of the first common fields to form first filtered data stored in dynamic memory as in-flight data; and

a second data processor to filter the second data by accessing the second data in dynamic memory as in-flight data and selecting a second sub-set of fields of the second common fields to form second filtered data stored in dynamic memory as in-flight data, wherein the first data processor and the second data processor are on separate nodes in a network, and apply rules to the first filtered data and the second filtered data in real-time to identify and report selected real-time events stored in dynamic memory as in-flight data prior to persistently storing the first filtered data, the second filtered data and the selected real-time events.

12. The system of claim 11 wherein the first filtered data and the second filtered data are each represented as an object with a unique global identifier field, a creation time field and a payload field.

13. The system of claim 12 wherein the first filtered data and the second f data are each represented as an object with a location field.

14. The system of claim 12 wherein the first filtered data and the second filtered data are each represented as an object with a context field defining one or more key value pairs.

15. The system of claim 12 wherein the first filtered data and the second filtered data are each represented as an object with an event field defining one or more events.

16. The system of claim 11 further comprising:

an in-memory cache for volatilely storing the first data, the second data, the first filtered data, the second filtered data and the selected real-time events; and

a persistent data store for persistently storing the first data, the second data, the first filtered data, the second filtered data and the selected real-time events.

17. The system of claim 16 further comprising a query engine to support search, analytics, and visualization of data in the in-memory cache and persistent data store.

18. The system of claim 16 wherein the persistent data store is a Not Only SQL database.

19. The system of claim 11 further comprising a data aggregator to form a new dimension of data based upon selected fields of the first filtered data and the second filtered data.

20. The system of claim 11 wherein the first data source and the second data source are selected from a database log, database redo log, operating system log, application log, web server log, application server log, machine generated log and a sensor.