Full flow retrieval optimized packet capture
View Patent ↗A packet capture system may copy packets from an interface to a bucket. When the bucket is full of packets, a new bucket for incoming packets may be started, and the full bucket may be indexed. During the indexing, each packet may be sorted in the bucket by flow, and each flow may be indexed. Once indexing is complete, the packets are written to a flow ordered FCAP file and the indexes are written to disk. The flow ordered nature of the FCAP file combined with the indices and their associated search algorithms allow for rapid retrieval of captured flows.
1. A computer-implemented process for organizing packets by network flows, comprising:
copying packets from a network interface and into a bucket;
when the bucket is full, indexing the packets within the full bucket by network flow and creating a new bucket to retrieve additional packets from the network interface into the new bucket,
wherein the indexing of the packets within the full bucket comprises searching for a corresponding flow within a flow index for each packet within the full bucket, and
adding each packet to the corresponding flow such that the packets are sorted into flows;
creating a flow capture (FCAP) file in memory prior to writing the packets to the FCAP file, wherein
the creating of the FCAP file comprises transmitting a request to a computing system for creating space in the memory for the FCAP file; and
writing the packets to the FCAP file reducing amount of disk input/output (I/O) required to retrieve the packets,
wherein the writing of the packets to the FCAP file comprises organizing the packets within the FCAP by the network flow and in original time order in which the packets are received for the full bucket that is indexed; and
writing indexes of the FCAP file to the memory.
2. The computer-implemented process of claim 1 , wherein the indexing of the bucket comprises sub-indexing each network flow by individual components of 5-tuple.
3. The computer-implemented process of claim 2 , wherein the individual components of 5-tuple comprise source Internet Protocol (IP), destination IP, source port, destination port, and a transport protocol.
4. The computer-implemented process of claim 1 , wherein the indexing of the bucket comprises searching for the corresponding network flow in the flow index for each packet in the bucket.
5. The computer implemented process of claim 4 , wherein the indexing of the bucket comprises adding a packet to the corresponding network flow.
6. The computer implemented process of claim 4 , wherein the indexing of the bucket comprises creating a new network flow when the corresponding network flow is nonexistent and adding the new network flow to a sub-index flow.
7. The computer-implemented process of claim 1 , further comprising: creating the FCAP file in the memory prior to writing the packets to the FCAP file.
8. The computer-implemented process of claim 7 , wherein the creating of the FCAP file comprises transmitting a request to a computing system for creating space in the memory for the FCAP file.
9. The computer-implemented process of claim 8 , wherein the creating of the FCAP file comprises recording the flow index when the space for the FCAP file is created.
10. The computer-implemented process of claim 9 , wherein the creating of the FCAP file comprises creating an index file for each sub-index within the flow index.
11. The computer-implemented process of claim 1 , wherein the writing the packets to the FCAP file further comprises writing the packets to the FCAP file based on the sort order determined during the indexing of the packets.
12. A computer-implemented process, comprising:
retrieving packets from a network interface and storing the packets within a bucket;
when the bucket is full, starting a new bucket to retrieve additional packets from the network interface; and
indexing the packets within the full bucket, wherein the indexing of the full bucket comprises sorting the packets within the full bucket by a network flow, the sorting of the packets within the full bucket comprises
searching for a corresponding flow within a flow index for each packet within the full bucket, and-adding
each packet to the corresponding flow such that the packets are sorted into flows;
creating a flow capture (FCAP) file in memory prior to writing the packets to the FCAP file, wherein
the creating of the FCAP file comprises transmitting a request to a computing system for creating space in the memory for the FCAP file; and
writing the packets to the FCAP file reducing amount of disk input/output (I/O) required to retrieve the packets, wherein the writing of the packets to the FCAP file comprises organizing the packets within the FCAP by the network flow and in original time order in which the packets are received for the full bucket that is indexed; and
writing indexes of the FCAP file to memory.
13. The computer-implemented process of claim 12 , wherein the indexing of the full bucket comprises searching for the corresponding network flow in the flow index for each packet in the full bucket.
14. The computer implemented process of claim 13 , wherein the indexing of the full bucket comprises adding a packet to the corresponding network flow.
15. The computer implemented process of claim 14 wherein the indexing of the full bucket comprises creating a new network flow when the corresponding network flow does not exist and adding the new network flow to a sub-index flow.
16. The computer-implemented process of claim 12 , further comprising:
creating the FCAP file in the memory prior to writing the packets to the FCAP file.
17. The computer-implemented process of claim 12 , wherein the writing of the packets to the FCAP file further comprises writing the packets to the FCAP file based on the sort order determined during the indexing of the full bucket.
18. A computer-implemented process for organizing packets by network flows, comprising:
copying packets from a network interface and into a bucket;
when the bucket is full, indexing the packets within the full bucket by network flow and creating a new bucket to retrieve additional packets from the network interface into the new bucket, wherein the indexing of the packets within the full bucket comprises
searching for a corresponding flow within a flow index for each packet within the full bucket, and
adding each packet to the corresponding flow such that the packets are sorted into flows;
creating a flow capture (FCAP) file in memory prior to writing the packets to the FCAP file, wherein
the creating of the FCAP file comprises transmitting a request to a computing system for creating space in the memory for the FCAP file; and
writing of the packets to the FCAP file reducing amount of disk input/output (I/O) required to retrieve the packets, wherein the writing of the packets to the FCAP file comprises organizing the packets within the FCAP by the network flow and in original time order in which the packets are received for the full bucket that is indexed; and
writing indexes of the FCAP file to the memory.