Hashing of network packet flows for efficient searching
The invention relates to a method and apparatus for efficient storing and retrieval of captured data packets. The packets are parsed to extract flow defining parameters such as source and destination addresses, the extracted addresses are hashed and the address hash numbers are reversibly combined, for example concatenated in a pre-defined order of their values to obtain a single hash index for a flow. The packets are then saved in a packet store in accordance and/or association with their hash index. The packets may be efficiently retrieved based on the two addresses or on a single network address.
1. A method comprising:
receiving, by a network device, a data packet from a network;
parsing, by the network device, the data packet to extract a source address and a destination address;
applying, by the network device, a hash function to the source address to obtain a first hash;
applying, by the network device, a hash function to the destination address to obtain a second hash;
concatenating, by the network device, the first and second hashes to generate a hash index for the data packet,
wherein the hash index defines a packet flow to which the data packet belongs;
storing, by the network device, the first and second hashes in a hash index;
identifying, by the network device, packets in a packet store that have a network address corresponding to the source address or the destination address by:
applying a hash function to the network address to obtain an address hash; and
searching a memory bucket where:
a particular quantity of most significant digits of a hash index of the memory bucket form the address hash, or
a particular quantity of least significant digits of the hash index of the memory bucket form the address hash.
2. The method of claim 1 , wherein
the packet store includes a plurality of memory buckets, and each memory bucket, of the plurality of memory buckets, is associated with one or more hash indices, and
the method further includes:
storing the data packet in a memory bucket, of the plurality of memory buckets, that is associated with the hash index.
3. The method of claim 1 , wherein the first and second hashes are concatenated in an ascending order or in a descending order.
4. The method of claim 1 , further comprising;
identifying packets stored in the packet store that belong to a particular packet flow between a first network address and a second network address by:
computing a hash index for the particular packet flow using one of the first network address or the second network address as the source address and the other of the first network address or the second network address as the destination address; and
searching for one or more memory buckets that are associated with the computed hash index.
5. The method of claim 1 , further comprising:
extracting one or more data fields from the data packet; and
applying one or more hash functions to the one or more extracted data fields to obtain one or more additional hashes,
wherein concatenating the first and second hashes includes:
concatenating the first hash, the second hash, and the one or more additional hashes.
6. The method of claim 5 , wherein concatenating the first hash, the second hash, and the one or more additional hashes includes:
concatenating the first hash, the second hash, and the one or more additional hashes in a same order for each of a plurality of received data packets.
7. The method of claim 5 , wherein the one or more extracted data fields include at least one of:
a VLAN tag,
an MPLS tag,
a source port number,
a destination port number,
a protocol identifier, or
a quality of service parameter.
8. A non-transitory computer-readable medium storing instructions, the comprising:
one or more instructions which, when executed by a processor of a device, cause the processor to:
receive a data packet from a network;
parse the data packet to extract a source address and a destination address;
apply a hash function to the source address to obtain a first hash;
apply a hash function to the destination address to obtain a second hash;
separably combine the first and second hashes to generate a hash index for the data packet,
wherein the hash index defines a packet flow to which the data packet belongs;
store the data packet in a memory bucket, of a plurality of memory buckets included in a packet store, that is associated with the hash index,
wherein the first and second hashes are concatenated in an ascending order of their respective values so that a lesser of the first and second hashes is stored in a most significant bits digits of the hash index and a greater of the first and second hashes is stored in a least significant digits of the hash index; and
identify packets in the packet store that have a particular network address corresponding to the source address or the destination address by:
applying a hash function to the particular network address to obtain an address hash; and
identifying at least one memory bucket, of the plurality of memory buckets, where:
n most significant digits of a hash index of the memory bucket form an address hash and m least significant digits of the hash index correspond to a quantity of digits that is equal to or greater than the address hash, or
n least significant digits of the hash index of the memory bucket form the address hash and m most significant digits of the hash index correspond to a quantity of digits that is equal or less than the address hash.
9. The non-transitory computer-readable medium of claim 8 , wherein the first and second hashes are concatenated in an ascending order or in a descending order.
10. The non-transitory computer-readable medium of claim 8 , wherein the instructions further include:
one or more instructions to identify packets stored in the packet store that belong to a particular packet flow between a first network address and a second network address by:
computing a hash index for the particular packet flow using one of the first network address or the second network address as the source address and the other of the first network address or the second network address as the destination address; and
searching for one or more memory buckets that are associated with the computed hash index.
11. The non-transitory computer-readable medium of claim 8 , wherein the instructions further include:
one or more instructions to extract one or more data fields from the data packet; and
one or more instructions to apply one or more hash functions to the one or more extracted data fields to obtain one or more additional hashes,
wherein the one or more instructions to concatenate the first and second hashes include:
one or more instructions to concatenate the first hash, the second hash, and the one or more additional hashes.
12. A method comprising:
receiving, by a network device, a data packet from a network;
parsing, by the network device, the data packet to extract a source address and a destination address;
applying, by the network device, a hash function to the source address to obtain a first hash;
applying, by the network device, a hash function to the destination address to obtain a second hash;
separably combining, by the network device, the first and second hashes to generate a hash index for the data packet,
wherein the hash index defines a packet flow to which the data packet belongs;
storing, by the network device, the data packet in a memory bucket, of a plurality of memory buckets included in a packet store, that is associated with the hash index,
wherein the first and second hashes are concatenated in a descending order of their respective values; and
identifying, by the network device, packets in the packet store that have a particular network address corresponding to the source address or the destination address by:
applying a hash function to the particular network address to obtain an address hash; and
identifying at least one memory bucket, of the plurality of memory buckets, where:
n most significant digits of a hash index of the memory bucket form an address hash and m least significant digits of the hash index correspond to a quantity of digits that is equal to or less than the address hash, or
n least significant digits of the hash index of the memory bucket form the address hash and m most significant digits of the hash index correspond to a quantity of digits that is equal or greater than the address hash.
13. The method of claim 12 , wherein the first and second hashes are concatenated in an ascending order or in a descending order.
14. The method of claim 12 , further comprising:
identifying packets stored in the packet store that belong to a particular packet flow between a first network address and a second network address by:
computing a hash index for the particular packet flow using one of the first network address or the second network address as the source address and the other of the first network address or the second network address as the destination address; and
searching for one or more memory buckets that are associated with the computed hash index.
15. The method of claim 12 , further comprising:
extracting one or more data fields from the data packet; and
applying one or more hash functions to the one or more extracted data fields to obtain one or more additional hashes,
wherein concatenating the first and second hashes includes:
concatenating the first hash, the second hash, and the one or more additional hashes.
16. A network device comprising:
a network interface for receiving a packet from a network; and
a packet processor, coupled to the network interface, comprising:
a packet parser configured to parse the packet and extract a source address and a destination address;
a hash generator configured to generate a first hash from the source address and a second hash from the destination address;
a hash combiner configured to:
concatenate the first and second hashes to generate a hash index for the packet,
wherein the hash index defines a packet flow to which the packet belongs; and
store the first and second hashes in a hash index; and
a search engine configured to identify packets in a packet store that have a network address corresponding to the source address or the destination address by:
applying a hash function to the network address to obtain an address hash; and
searching a memory bucket where:
a particular quantity of most significant digits of a hash index of the memory bucket form the address hash, or
a particular quantity of least significant digits of the hash index of the memory bucket form the address hash.
17. The network device of claim 16 , wherein the first and second hashes are concatenated in an ascending order.
18. The network device of claim 16 , wherein the first and second hashes are concatenated in a descending order.
19. The network device of claim 16 , wherein the search engine is further configured to:
identify packets stored in the packet store that belong to a particular packet flow between a first network address and a second network address by:
computing a hash index for the particular packet flow using one of the first network address or the second network address as the source address and the other of the first network address or the second network address as the destination address; and
searching for one or more memory buckets that are associated with the computed hash index.
20. The network device of claim 16 , wherein the search engine is further configured to:
apply one or more hash functions to one or more extracted data fields, from the packet, to obtain one or more additional hashes,
wherein the one or more additional hashes are concatenated with the first and second hashes.