Method and system for accelerated stream processing
Disclosed herein are methods and systems for hardware-accelerating various data processing operations in a rule-based decision-making system such as a business rules engine, an event stream processor, and a complex event stream processor. Preferably, incoming data streams are checked against a plurality of rule conditions. Among the data processing operations that are hardware-accelerated include rule condition check operations, filtering operations, and path merging operations. The rule condition check operations generate rule condition check results for the processed data streams, wherein the rule condition check results are indicative of any rule conditions which have been satisfied by the data streams. The generation of such results with a low degree of latency provides enterprises with the ability to perform timely decision-making based on the data present in received data streams.
1. A system for making a field programmable gate array (FPGA) available in a network for loading a processing pipeline thereon for the FPGA to apply parallelism when processing streaming data, the streaming data comprising data arranged in a plurality of fields, the system comprising:
an FPGA that is addressable within a network, the FPGA arranged for configuration in response to a command over the network so that a processing pipeline for receiving and processing streaming data is loadable thereon;
the loadable processing pipeline including a plurality of parallel paths for augmenting the streaming data with a plurality of flags indicative of a plurality of data quality conditions, each of a plurality of the parallel paths including pipelined logic for performing different processing operations on the streaming data, and wherein each of a plurality of the parallel paths includes field selection logic that filters which fields of the streaming data that downstream pipelined logic in that parallel path will process, wherein a plurality of the parallel paths include field selection logic that filter for different fields of the streaming data so that the FPGA is thereby configurable via the loadable processing pipeline to parallel process different fields of the streaming data in different parallel paths with different processing operations.
2. The system of claim 1 wherein the loadable processing pipeline further comprises field parsing logic upstream from the parallel paths, wherein the field parsing logic identifies where boundaries between the fields of the streaming data are located.
3. The system of claim 2 wherein the streaming data further comprises a plurality of records, wherein a plurality of the records include the data arranged in the fields, and wherein the field parsing logic further comprises record parsing logic that identifies where boundaries between the records in the streaming data are located.
4. The system of claim 1 wherein the loadable processing pipeline further comprises join logic downstream from the parallel paths, wherein the join logic merges data from the parallel paths into a consolidated data stream.
5. The system of claim 4 wherein the parallel paths include a bypass path that delivers the received streaming data to the join logic.
6. The system of claim 1 wherein the different processing operations comprise a plurality of different processing operations selected from the group consisting of a range check operation, a character check operation, a threshold check operation, and a matching operation.
7. The system of claim 1 wherein at least one of the parallel paths includes range check logic for performing one of the different processing operations.
8. The system of claim 1 wherein at least one of the parallel paths includes character check logic for performing one of the different processing operations.
9. The system of claim 1 wherein at least one of the parallel paths includes threshold check logic for performing one of the different processing operations.
10. The system of claim 1 wherein at least one of the parallel paths includes matching logic for performing one of the different processing operations.
11. The system of claim 10 wherein the matching logic comprises regular expression pattern matching logic.
12. The system of claim 1 further comprising:
a database;
a processor that (1) manages a flow of commands and streaming data into the FPGA, including a command to load the loadable processing pipeline onto the FPGA, (2) manages a flow of processed streaming data out of the FPGA, and (3) selectively processes and loads at least a portion of the processed streaming data into the database based on the data quality condition flags.
13. The system of claim 12 further comprising:
a network interface through which the processor receives the streaming data.
14. The system of claim 13 wherein the network interface receives the streaming data from a plurality of different data sources.
15. The system of claim 1 further comprising:
a processor that manages a flow of commands and streaming data into the FPGA, including a command to load the loadable processing pipeline onto the FPGA.
16. A method for making a compute resource network-connectable for configuring the compute resource to apply parallelism when processing streaming data, the streaming data comprising data arranged in a plurality of fields, the method comprising:
providing a field programmable gate array (FPGA) within a network; and
providing a communication path in the network for commanding the FPGA to load a processing pipeline onto the FPGA to configure the FPGA for receiving and processing streaming data, the processing pipeline including a plurality of parallel paths that augment the streaming data with a plurality of flags indicative of a plurality of data quality conditions, each of a plurality of the parallel paths including pipelined logic for performing different processing operations on the streaming data, and wherein each of a plurality of the parallel paths includes field selection logic that filters which fields of the streaming data that downstream pipelined logic in that parallel path will process, wherein a plurality of the parallel paths include field selection logic that filter for different fields of the streaming data so that the FPGA is thereby configurable via the processing pipeline to parallel process different fields of the streaming data in different parallel paths with different processing operations.
17. The method of claim 16 further comprising:
a processor (1) managing a flow of commands and streaming data into the FPGA, including a command to load the loadable processing pipeline onto the FPGA, (2) managing a flow of processed streaming data out of the FPGA, and (3) selectively processing and loading at least a portion of the processed streaming data into a database based on the data quality condition flags.
18. The method of claim 17 further comprising:
the processor receiving the streaming data via a network interface.
19. The method of claim 18 further comprising:
the network interface receiving the streaming data from a plurality of different data sources.
20. The method of claim 16 wherein the different processing operations comprise a plurality of different processing operations selected from the group consisting of a range check operation, a character check operation, a threshold check operation, and a matching operation.
21. The method of claim 16 wherein at least one of the parallel paths includes range check logic for performing one of the different processing operations.
22. The method of claim 16 wherein at least one of the parallel paths includes character check logic for performing one of the different processing operations.
23. The method of claim 16 wherein at least one of the parallel paths includes threshold check logic for performing one of the different processing operations.
24. The method of claim 16 wherein at least one of the parallel paths includes matching logic for performing one of the different processing operations.
25. The method of claim 16 wherein the processing pipeline further comprises field parsing logic upstream from the parallel paths, wherein the field parsing logic identifies where boundaries between the fields of the streaming data are located.
26. The method of claim 25 wherein the streaming data further comprises a plurality of records, wherein a plurality of the records include the data arranged in the fields, and wherein the field parsing logic further comprises record parsing logic that identifies where boundaries between the records in the streaming data are located.
27. The method of claim 16 wherein the processing pipeline further comprises join logic downstream from the parallel paths, wherein the join logic merges data from the parallel paths into a consolidated data stream.
28. The method of claim 27 wherein the parallel paths include a bypass path that delivers the received streaming data to the join logic.
29. The method of claim 16 wherein the streaming data comprises a plurality of records, wherein the flags correspond to specific records and are indicative of data quality conditions for the corresponding specific records, and wherein the parallel paths augment the streaming data by adding the flags to their corresponding specific records.
30. The system of claim 1 wherein the streaming data comprises a plurality of records, wherein the flags correspond to specific records and are indicative of data quality conditions for the corresponding specific records, and wherein the parallel paths augment the streaming data by adding the flags to their corresponding specific records.