High speed network data extractor
The present invention describes a system and method of extracting and storing data elements from network packets, thus performing the task of data mining. In one embodiment of the present invention incoming packets are decomposed one protocol layer at a time to extract data elements contained in the protocol headers. Layer-specific parsers perform deep packet inspection in order to extract data elements from upper-level protocols. Extracted data is arranged in rows, which are subsequently stored into a memory-based accumulator. After some length of time the accumulator is flushed to disk files. Another process reads the flushed disk files row-by-row, inserting each row into a relational database. Standard SQL operations are performed on the relational database in order to generate and display reports of the collected data.
1. A system for network data extraction comprising:
a packet decoding engine including at least one protocol decoder operable to decompose and parse received packets and identify extractable data entities from the received packets;
a data extraction engine operable to extract identified data entities from the packets and arrange the data entities into entity sets along with statistical elements reflective of redundant entity sets; and
an accumulator for storage of extracted entity sets and elimination of redundant entity sets.
2. The system of claim 1 , and further comprising a filter processing engine operable to select only packets which contain entities for extraction.
3. The system of claim 2 , further operable to copy entity sets from the accumulator to long-term storage.
4. The system of claim 3 , further comprising a database insertion engine to copy entity sets from long-term storage into a relational database.
5. The system of claim 4 , further operable to allow a user to add an entity set definition, remove an entity set definition, and modify an entity set definition.
6. The system of claim 5 , further operable to allow addition of protocol decoders to extend the set of entities available for inclusion in entity sets.
7. The system of claim 6 , further comprising a reassembly and state storage engine for reassembling packets having IP, TCP and UDP segments of like streams and allocating state storage for said reassembled streams.
8. The system of claim 7 , further comprising a packet recorder engine for recording each packet to long-term storage.
9. A method for network data extraction comprising:
receiving packets of data, each packet having at least one protocol layer and at least one payload section;
decomposing the received packets to delineate a location of the at least one protocol layer within each packet;
parsing the received packets to access data entities in the at least one payload section within the packet;
extracting data entities from the at least one protocol layer and payload section within the packet;
constructing entity sets from the extracted data entities;
inserting at least some of the entity sets into an accumulator; and
inserting at least some of the entity sets from the accumulator into a relational database.
10. The method of claim 9 , wherein the step of inserting at least some of the entity sets into the accumulator comprises temporarily storing rows of extracted data entities in the accumulator before inserting them into the relational database.
11. The method of claim 10 , and further comprising converting redundant entity sets into statistical data for temporary storage in the accumulator to thereby eliminate storing redundant entity sets.
12. The method of claim 11 , and further comprising copying entity sets from the accumulator to long-term storage prior to inserting them into the relational database.
13. The method of claim 9 , and further comprising comparing at least one filter to each received packet in order to further qualify the packets for entity extraction.
14. The method of claim 9 , and further comprising allowing a user to add an entity set definition, remove an entity set definition, and modify an entity set definition.
15. The method of claim 14 , and further comprising allowing the addition of protocol decoders to expand the amount of data entities available for inclusion in entity sets.
16. The method of claim 9 , wherein the packets include IP, TCP and UDP segments, and further comprising reassembling the IP, TCP and UDP segments of like streams and allocating state storage for said reassembled streams subsequent to parsing the received packets.
17. The method of claim 16 , and further comprising recording each packet to long-term storage.
18. The method of claim 9 , wherein a plurality of the received packets are processed in parallel on multiple processors or processor cores.
19. The method of claim 18 , wherein the steps of decomposing and parsing the received packets are executable within the plurality of protocol decoders in parallel on further multiple processors or processor cores.
20. A system for network data extraction comprising:
at least one processor operable to execute computer program instructions;
at least one memory operable to store computer program instructions executable by the processor; and
computer program instructions stored in the at least one memory and executable to perform the steps of:
receiving packets of data, each packet having at least one protocol layer and at least one payload section;
decomposing the received packets to delineate a location of the at least one protocol layer within each packet;
parsing the received packets to access data entities in the at least one payload section within the packet;
extracting data entities from the at least one protocol layer and payload section within the packet;
constructing entity sets from the extracted data entities;
inserting at least some of the entity sets into an accumulator; and
inserting at least some of the entity sets from the accumulator into a relational database.
21. The system of claim 20 , wherein the step of inserting at least some of the entity sets into the accumulator comprises temporarily storing rows of extracted data entities in the accumulator before inserting them into the relational database.
22. The system of claim 21 , and further comprising converting redundant entity sets into statistical data for temporary storage in the accumulator to thereby eliminate storing redundant entity sets.
23. The system of claim 22 , and further comprising copying entity sets from the accumulator to long-term storage prior to inserting them into the relational database.
24. The system of claim 20 , and further comprising comparing at least one filter to each received packet in order to further qualify the packets for entity extraction.
25. The system of claim 20 , and further comprising allowing a user to add an entity set definition, remove an entity set definition, and modify an entity set definition.
26. The system of claim 25 , and further comprising allowing the addition of protocol decoders to expand the amount of data entities available for inclusion in entity sets.
27. The system of claim 20 , wherein the packets include IP, TCP and UDP segments, and further comprising reassembling the IP, TCP and UDP segments of like streams and allocating state storage for said reassembled streams subsequent to parsing the received packets.
28. The system of claim 27 , and further comprising recording each packet to long-term storage.
29. The system of claim 20 , wherein the at least one processor comprises a plurality of processors or processor cores, each processor or processor core being operably associated with a protocol decoder for executing the decomposing and parsing steps to thereby allow the execution of the plurality of protocol decoders in parallel on a plurality of received packets.
30. The method of claim 29 , wherein the steps of decomposing and parsing the received packets are executable within the plurality of protocol decoders in parallel on further multiple processors or processor cores.
31. A computer program product of network data extraction comprising:
a computer readable medium; and
computer program instructions, recorded on the computer readable medium, executable by a processor, for performing the steps of:
receiving packets of data, each packet having at least one protocol layer and at least one payload section;
decomposing the received packets to delineate a location of the at least one protocol layer within each packet;
parsing the received packets to access data entities in the at least one payload section within the packet;
extracting data entities from the at least one protocol layer and payload section within the packet;
constructing entity sets from the extracted data entities;
inserting at least some of the entity sets into an accumulator; and
inserting at least some of the entity sets from the accumulator into a relational database.
32. The system of claim 31 , wherein the step of inserting at least some of the entity sets into the accumulator comprises temporarily storing rows of extracted data entities in the accumulator before inserting them into the relational database.
33. The system of claim 32 , and further comprising converting redundant entity sets into statistical data for temporary storage in the accumulator to thereby eliminate storing redundant entity sets.
34. The system of claim 33 , and further comprising copying entity sets from the accumulator to long-term storage prior to inserting them into the relational database.
35. The system of claim 31 , and further comprising comparing at least one filter to each received packet in order to further qualify the packets for entity extraction.
36. The system of claim 31 , and further comprising allowing a user to add an entity set definition, remove an entity set definition, and modify an entity set definition.
37. The system of claim 31 , and further comprising allowing the addition of protocol decoders to expand the amount of data entities available for inclusion in entity sets.
38. The system of claim 31 , wherein the packets include IP, TCP and UDP segments, and further comprising reassembling the IP, TCP and UDP segments of like streams and allocating state storage for said reassembled streams subsequent to parsing the received packets.
39. The system of claim 38 , and further comprising recording each packet to long-term storage.
40. The system of claim 31 , wherein the at least one processor comprises a plurality of processors or processor cores, each processor or processor core being operably associated with a protocol decoder for executing the decomposing and parsing steps to thereby allow the execution of the plurality of protocol decoders in parallel on a plurality of received packets.
41. The method of claim 40 , wherein the steps of decomposing and parsing the received packets are executable within the plurality of protocol decoders in parallel on further multiple processors or processor cores.