De-duplicating of packets in flows at layer 3
Embodiments are directed towards receiving packets communicated over at least one network, determining layer 3 header information for the received packets, normalizing the determined layer 3 header information for each received packet, employing a determined value based on the normalized layer 3 header information to detect each received packet that is a duplicate, disregarding duplicate packets, and enabling monitoring and analysis of at least selected flows that include packets that are determined to be non-duplicated. Also, if the determined layer 3 header information indicates that the received packet is fragmented, that packet is de-fragmented at least in accordance with a fragment offset. Additionally, normalization may include at least one of masking at least one value in the layer 3 header information, or rolling back changes in the layer 3 header information.
1. A method for monitoring flows of packets over a network, wherein a network device enables actions, comprising:
passively receiving packets communicated over at least one network;
determining Open Systems Interconnection (OSI) model layer 2 and layer 3 (layer 2-3) information included in each header of each received packet;
normalizing layer 2-3 information in each header of each received packet;
if at least a value based on normalized layer 2-3 information for a header of a packet is determined to be equivalent to another value based on normalized layer 2-3 information for another header of another subsequently received packet, determining the other subsequently received packet as duplicative of the packet; and
monitoring at least one flow of received packets, wherein the at least one monitored flow's packets are identified as non-duplicative, and wherein each packet that is determined as duplicative is subsequently processed separate from monitoring the at least one flow of received packets.
2. The method of claim 1 , wherein receiving packets, further comprises receiving packets that are aggregated from more at least one of a plurality of intermediate network devices, a plurality of networks, a plurality of network tap connections, or a plurality of port mirrors.
3. The method of claim 1 , wherein normalizing the layer 2-3 information includes masking header information that includes at least one of a destination Media Access Control (MAC) address, a source MAC address, a time-to-live value or a checksum value.
4. The method of claim 1 , wherein normalizing the layer 2-3 information includes rolling back changes to layer 2-3 information in at least one packet header that is made by an intermediate device.
5. The method of claim 1 , wherein identifying the other subsequently received packet as the duplicate, further comprises:
generating a hash value based on at least one of a portion of the normalized layer 2-3 information for the header of the received packet, a portion of a layer 3 payload for the received packet, or a portion of a layer 4 header of the received packet;
generating another hash value based on at least one of a portion of the normalized layer 2-3 information for the other header of the subsequently received packet, a portion of a layer 3 payload of the subsequently received packet, or a portion of a layer 4 header of the subsequently received packet; and
comparing the hash value and the other hash value to determine equivalency between the received packet and the other subsequently received packet.
6. The method of claim 1 , wherein monitoring at least one flow, further comprises at least one of:
determining information in real time regarding at least one of the packets in the at least one monitored flow; or
determining information in real time regarding the at least one monitored flow.
7. The method of claim 1 , further comprising if the layer 2-3 information of at least one packet's header includes at least one of a fragment offset or a packet fragment, reassembling that packet in accordance with at least the fragment offset.
8. The method of claim 1 , further comprising employing layer 4 information of the OSI model instead of layer 2-3 information to perform the actions of claim 1 .
9. A network device for monitoring flows of packets over a network, comprising:
a transceiver device for monitoring communication over the network;
a memory device for storing at least instructions; and
a processor device that is operative to execute instructions that enable actions, including:
passively receiving packets communicated over at least one network;
determining Open Systems Interconnection (OSI) model layer 2 and layer 3 (layer 2-3) information included in each header of each received packet;
normalizing layer 2-3 information in each header of each received packet;
if at least a value based on normalized layer 2-3 information for a header of a packet is determined to be equivalent to another value based on normalized layer 2-3 information for another header of another subsequently received packet, determining the other subsequently received packet as duplicative of the packet; and
monitoring at least one flow of received packets, wherein the at least one monitored flow's packets are identified as non-duplicative, and wherein each packet that is determined as duplicative is subsequently processed separate from monitoring the at least one flow of received packets.
10. The network device of claim 9 , wherein receiving packets, further comprises receiving packets that are aggregated from more at least one of a plurality of intermediate network devices, a plurality of networks, a plurality of network tap connections, or a plurality of port mirrors.
11. The network device of claim 9 , wherein normalizing the layer 2-3 information includes masking header information that includes at least one of a destination Media Access Control (MAC) address, a source MAC address, a time-to-live value or a checksum value.
12. The network device of claim 9 , wherein normalizing the layer 2-3 information includes rolling back changes to layer 2-3 information in at least one packet header that is made by an intermediate device.
13. The network device of claim 9 , wherein identifying the other subsequently received packet as the duplicate, further comprises:
generating a hash value based on at least one of a portion of the normalized layer 2-3 information for the header of the received packet, a portion of a layer 3 payload of the received packet, or a portion of a layer 4 header of the received packet;
generating another hash value based on at least one of a portion of the normalized layer 3 information for the other header of the subsequently received packet, a portion of a layer 3 payload of the subsequently received packet, or a portion of a layer 4 header of the subsequently received packet; and
comparing the hash value and the other hash value to determine equivalency between the received packet and the other subsequently received packet.
14. The network device of claim 9 , wherein monitoring at least one flow, further comprises at least one of:
determining information in real time regarding at least one of the packets in the at least one monitored flow; or
determining information in real time regarding the at least one monitored flow.
15. The network device of claim 9 , further comprising if the layer 2-3 information of at least one packet's header includes at least one of a fragment offset or a packet fragment, reassembling that packet in accordance with at least the fragment offset.
16. The network device of claim 9 , further comprising employing layer 4 information of the OSI model instead of layer 2-3 information to perform the actions of claim 9 .
17. A system for monitoring flows of packets over a network, comprising:
a first network device that includes:
a transceiver device for monitoring communication over the network;
a memory device for storing at least instructions; and
a processor device that is operative to execute instructions that enable actions, comprising communicating at least one flow of packets with at least a second network device; and
a third network device that includes:
a transceiver device for monitoring communication between the first network device and the second network device;
a memory device for storing at least instructions; and
a processor device that is operative to execute instructions that enable actions, comprising:
passively receiving packets communicated over at least one network;
determining Open Systems Interconnection (OSI) model layer 2 and layer 3 (layer 2-3) information included in each header of each received packet;
normalizing layer 2-3 information in each header of each received packet;
if at least a value based on normalized layer 2-3 information for a header of a packet is determined to be equivalent to another value based on normalized layer 2-3 information for another header of another subsequently received packet, determining the other subsequently received packet as duplicative of the packet; and
monitoring at least one flow of received packets, wherein the at least one monitored flow's packets are identified as non-duplicative, and wherein each packet that is determined as duplicative is subsequently processed separate from monitoring the at least one flow of received packets.
18. The system of claim 17 , wherein receiving packets, further comprises receiving packets that are aggregated from more at least one of a plurality of intermediate network devices, a plurality of networks, a plurality of network tap connections, or a plurality of port mirrors.
19. The system of claim 17 , wherein normalizing the layer 2-3 information includes masking header information that includes at least one of a destination Media Access Control (MAC) address, a source MAC address, a time-to-live value or a checksum value.
20. The system of claim 17 , wherein normalizing the layer 2-3 information includes rolling back changes to layer 2-3 information in at least one packet header that is made by an intermediate device.
21. The system of claim 17 , wherein identifying the other subsequently received packet as the duplicate, further comprises:
generating a hash value based on at least one of a portion of the normalized layer 2-3 information for the header of the received packet, a portion of a layer 3 payload of the received packet, or a portion of a layer 4 header of the received packet;
generating another hash value based on at least one of a portion of the normalized layer 3 information for the other header of the subsequently received packet, a portion of a layer 3 payload of the subsequently received packet, or a portion of a layer 4 header of the subsequently received packet; and
comparing the hash value and the other hash value to determine equivalency between the received packet and the other subsequently received packet.
22. The system of claim 17 , wherein monitoring at least one flow, further comprises at least one of:
determining information in real time regarding at least one of the packets in the at least one monitored flow; or
determining information in real time regarding the at least one monitored flow.
23. The system of claim 17 , further comprising if the layer 2-3 information of at least one packet's header includes at least one of a fragment offset or a packet fragment, reassembling that packet in accordance with at least the fragment offset.
24. The system of claim 17 , further comprising employing layer 4 information of the OSI model instead of layer 2-3 information to perform the actions of claim 17 .
25. A processor readable non-transitive storage media that includes instructions for monitoring flows of packets over a network, wherein a network device that executes at least a portion of the instructions enables actions, comprising:
passively receiving packets communicated over at least one network;
determining Open Systems Interconnection (OSI) model layer 2 and layer 3 (layer 2-3) information included in each header of each received packet;
normalizing layer 2-3 information in each header of each received packet;
if at least a value based on normalized layer 2-3 information for a header of a packet is determined to be equivalent to another value based on normalized layer 3 information for another header of another subsequently received packet, determining the other subsequently received packet as duplicative of the packet; and
monitoring at least one flow of received packets, wherein the at least one monitored flow's packets are identified as non-duplicative, and wherein each packet that is determined as duplicative is subsequently processed separate from monitoring the at least one flow of received packets.
26. The media of claim 25 , further comprising employing layer 4 information of the OSI model instead of layer 2-3 information to perform the actions of claim 25 .
27. The media of claim 25 , wherein normalizing the layer 2-3 information includes masking header information that includes at least one of a destination Media Access Control (MAC) address, a source MAC address, a time-to-live value or a checksum value.
28. The media of claim 25 , wherein normalizing the layer 2-3 information includes rolling back changes to layer 2-3 information in at least one packet header that is made by an intermediate device.
29. The media of claim 25 , wherein identifying the other subsequently received packet as the duplicate, further comprises:
generating a hash value based on at least one of a portion of the normalized layer 2-3 information for the header of the received packet, a portion of a layer 3 payload of the received packet, or a portion of a layer 4 header of the received packet;
generating another hash value based on at least one of a portion of the normalized layer 3 information for the other header of the subsequently received packet, a portion of a layer 3 payload of the received packet, or a portion of a layer 4 header of the received packet; and
comparing the hash value and the other hash value to determine equivalency between the packet and the other subsequently received packet.
30. The media of claim 25 , wherein monitoring at least one flow, further comprises at least one of:
determining information in real time regarding at least one of the packets in the at least one monitored flow; or
determining information in real time regarding the at least one monitored flow.