Apparatus and method for pipelined event processing in a distributed environment
A method includes receiving first data from a first data source that continuously generates a first set of records with first common fields. The first data is filtered by selecting a first sub-set of fields of the first common fields to form first filtered data. Second data is received from a second data source that continuously generates a second set of records with second common fields. The second data is filtered by selecting a second sub-set of fields of the second common fields to form second filtered data, where the first filtered data and the second filtered data are generated simultaneously. Rules are applied to the first filtered data and the second filtered data in real-time to identify selected real-time events. The selected real-time events are reported prior to persistently storing the first filtered data, the second filtered data and the selected real-time events.
1. A method, comprising:
receiving a first set of records from a first data source that regularly generates the first set of records with first common fields, wherein the first set of records characterize a first real-time event;
receiving a second set of records from a second data source that regularly generates the second set of records with second common fields, wherein the second set of records characterize a second real-time event different than the first real-time event;
filtering with a first data processor the first data by accessing the first data stored in dynamic memory as in-flight data and selecting a first sub-set of fields of the first common fields to form first filtered data stored in dynamic memory as in-flight data; and
filtering with a second data processor the second data by accessing the second data in dynamic memory as in-flight data and selecting a second sub-set of fields of the second common fields to form second filtered data stored in dynamic memory as in-flight data, wherein the first data processor and the second data processor are on separate nodes in a network and apply rules to the first filtered data and the second filtered data in real-time to identify and report selected real-time events stored in dynamic memory as in-flight data prior to persistently storing the first filtered data, the second filtered data and the selected real-time events.
2. The method of claim 1 wherein the first filtered data and the second filtered data are each represented as an object with a unique global identifier field, a creation time field and a payload field.
3. The method of claim 2 wherein the first filtered data and the second filtered data are each represented as an object with a location field.
4. The method of claim 2 wherein the first filtered data and the second filtered data are each represented as an object with a context field defining one or more key value pairs.
5. The method of claim 2 wherein the first filtered data and the second filtered data are each represented as an object with an event field defining one or more events.
6. The method of claim 1 further comprising:
volatilely storing in an in-memory cache the first data, the second data, the first filtered data, the second filtered data and the selected real-time events; and
persistently storing in a persistent data store the first data, the second data, the first filtered data, the second filtered data and the selected real-time events.
7. The method of claim 6 further comprising utilizing a query engine to support search, analytics, and visualization of data in the in-memory cache and persistent data store.
8. The method of claim 6 wherein the persistent data store is a Not Only SQL database.
9. The method of claim 1 further comprising utilizing a data aggregator to form a new dimension of data based upon selected fields of the first filtered data and the second filtered data.
10. The method of claim 1 wherein the first data source and the second data source are selected from a database log, database redo log, operating system log, application log, web server log, application server log, machine generated log and a sensor.