Data matching using flow based packet data storage
A system for matching data using flow based packet data storage includes a communications interface and a processor. A communications interface receives a packet between a source and a destination. The processor identifies a flow between the source and the destination based on the packet. The processor determines whether some of packet data of the packet indicates a potential match to data in storage using hashes. The processor then stores the data from the most likely data match and second most likely data match without a packet header in a block of memory in the storage based on the flow.
1. A system for matching data using flow based packet data storage, the system comprising:
a communications interface that receives at least one data packet at a network device between a source and a destination, the at least one data packet including data and flow information; and
a processor that:
identifies a flow between the source and the destination based on the flow information in the at least one data packet;
determines whether at least a portion of the data from the received at least one data packet indicates one or more potential matches to data in storage;
retrieves a list of possible data matches;
determines match sizes of at least two likely data matches by directly comparing packet bytes and matched data bytes; and
stores the data from the at least one data packet without a packet header in a block of memory allocated for the flow, or
generates a retrieve instruction for the data match depending on the match sizes.
2. The system of claim 1 , wherein the processor moves the storage data between a fast memory and a slow memory.
3. The system of claim 1 , wherein the flow comprises a session between the source and the destination.
4. The system of claim 1 , wherein the processor allocates the block of the memory for the identified flow.
5. The system of claim 1 , wherein the processor transmits the packet data.
6. The system of claim 1 , wherein the block of memory allocated for the flow is allocated from a list of partially filled blocks of memory.
7. A method for matching data using flow based packet data storage, the method comprising:
receiving at least one data packet at a network device between a source and a destination, the at least one data packet including data and flow information;
identifying a flow between the source and the destination based on the flow information in the at least one data packet;
determining whether at least a portion of the data from the received at least one data packet indicates one or more potential matches to data in storage;
retrieving a list of possible data matches;
determining match sizes at least two likely data matches by directly comparing packet bytes and matched data bytes; and
storing the data from the at least one data packet without a packet header in a block of memory allocated for the flow, or generating a retrieve instruction for the data match depending on the match sizes.
8. The method of claim 7 , further comprising moving the storage data between a fast memory and a slow memory.
9. The method of claim 7 , wherein the flow comprises a session between the source and the destination.
10. The method of claim 7 , further comprising allocating the block of the memory for the identified flow.
11. The method of claim 7 , further comprising transmitting the packet data.
12. The method of claim 7 , further comprising allocating the block of memory for the identified flow from a list of partially filled blocks of memory.
13. A non-transitory computer-readable medium having embodied thereon a program, the program executable by a processor to perform a method for matching data using flow based packet data storage, the method comprising:
receiving at least one data packet at a network device between a source and a destination, the at least one data packet including data and flow information;
identifying a flow between the source and the destination based on the flow information in the at least one data packet;
determining whether at least a portion of the data from the received at least one data packet indicates one or more potential matches to data in storage;
retrieving a list of possible data matches;
determining match sizes of at least two likely data matches by directly comparing packet bytes and matched data bytes; and
storing the data from the at least one data packet without a packet header in a block of memory allocated for the flow, or generating a retrieve instruction for the data match depending on the match sizes.
14. The non-transitory computer-readable medium of claim 13 , wherein the program is further executable to direct the processor to move the storage data between a fast memory and a slow memory.
15. The non-transitory computer-readable medium of claim 13 , wherein the flow comprises a session between the source and the destination.
16. The non-transitory computer-readable medium of claim 13 , wherein the program is further executable to direct the processor to allocate the block of the memory for the identified flow.
17. The non-transitory computer-readable medium of claim 13 , wherein the program is further executable to direct the processor to transmit the packet data.
18. The non-transitory computer-readable medium of claim 13 , wherein the block of memory allocated for the flow is allocated from a list of partially filled blocks of memory.