NETWORK FLOW DE-DUPLICATION
Systems, methods, and computer-readable media are provided for de-duplicating sensed data packets in a network. As data packets of a particular network flow move through the network, the data packets can be sensed and reported by various sensors across the network. An optimal sensor of the network can be determined based upon data packets reported by the various sensors. Data packets sensed and reported by the optimal sensor can be preserved for network analysis. Duplicative data packets of the particular network flow sensed and reported by other sensors of the network can be discarded to save storage capacity and processing power of network-flow analysis tools. Analysis of the particular network flow can be performed based upon the data packets sensed by the optimal sensor and non-duplicative data packets of the particular network-flow sensed by other sensors of the network.
1 . A method comprising:
receiving, from a plurality of sensors in a network, data packets of a particular network flow of the network;
analyzing the data packets to determine a specific sensor of the plurality of sensors;
preserving data packets sensed and reported from the specific sensor;
determining duplicative data packets sensed and reported from other sensor(s) of the plurality of sensors based upon the data packets reported from the specific sensor;
discarding the duplicative data packets sensed and reported from the other sensor(s); and
analyzing the particular network flow based upon the data packets reported from the specific sensor and non-duplicative data packets reported from the other sensor(s).
2 . The method of claim 1 , further comprising:
analyzing the data packets from the plurality of sensors to determine a number of data packets sensed by each sensor of the plurality of sensors,
wherein the specific sensor has sensed the most number of data packets of the particular network flow among the plurality of sensors.
3 . The method of claim 1 , wherein the receiving, from the plurality of sensors, the data packets of the particular network flow comprises:
receiving, from the plurality of sensors, data packets of the particular network flow for a predetermined time period,
wherein the specific sensor is determined based upon the data packets of the particular network flow sensed during the predetermined time period.
4 . The method of claim 1 , wherein the receiving, from the plurality of sensors, the data packets of the particular network flow further comprises:
sampling the data packets of the particular network flow received from the plurality of sensors,
wherein the specific sensor is determined based upon sampled data packets of the particular network flow.
5 . The method of claim 1 , further comprising:
reconciling the data packets of the particular network flow received from the plurality of sensors; and
determining non-duplicative data packets of the particular network flow,
wherein the specific sensor has sensed the most number of the non-duplicative data packets of the particular network flow among the plurality of sensors.
6 . The method of claim 1 , wherein the data packets of the particular network flow comprise a set of information to uniquely identify the particular network flow, the set of information including a source address, a destination address, a source port, destination port, a protocol, a user identification (ID), and a starting timestamp.
7 . The method of claim 6 , further comprising:
analyzing the data packets from the plurality of sensors to determine a starting timestamp for each of the data packets,
wherein the specific sensor sensed the earliest data packet of the particular network flow.
8 . The method of claim 1 , wherein the particular network flow is a first user datagram protocol (UDP) network flow, further comprising:
determining that the first UDP network flow being inactive for a predetermined time period; and
using a new flow start-time to distinguish a second UDP network flow from the first UDP network flow.
9 . The method of claim 1 , wherein the particular network flow is a transmission control protocol (TCP) network flow, further comprising:
determining a start of the TCP network flow based upon a three-way hand-shake.
10 . The method of claim 8 , further comprising:
determining an end of the TCP network flow based upon a four-way hand-shake.
11 . A system comprising:
a processor; and
a computer-readable storage medium storing instructions which, when executed by the processor, cause the system to perform operations comprising:
receiving, from a plurality of sensors in a network, data packets of a particular network flow of the network;
analyzing the data packets to determine a specific sensor of the plurality of sensors;
preserving data packets sensed and reported from the specific sensor;
determining duplicative data packets sensed and reported from other sensor(s) of the plurality of sensors based upon the data packets reported from the specific sensor;
discarding the duplicative data packets sensed and reported from the other sensor(s); and
analyzing the particular network flow based upon the data packets reported from the specific sensor and non-duplicative data packets reported from the other sensor(s).
12 . The system of claim 11 , wherein the instructions, when executed by the processor, cause the system to perform operations further comprising:
analyzing the data packets from the plurality of sensors to determine a number of data packets sensed by each sensor of the plurality of sensors,
wherein the specific sensor has sensed the most number of data packets of the particular network flow among the plurality of sensors.
13 . The system of claim 11 , wherein the instructions, when executed by the processor, cause the system to perform operations further comprising:
receiving, from the plurality of sensors, data packets of the particular network flow for a predetermined time period,
wherein the specific sensor is determined based upon the data packets of the particular network flow sensed during the predetermined time period.
14 . The system of claim 11 , wherein the instructions, when executed by the processor, cause the system to perform operations further comprising:
sampling the data packets of the particular network flow received from the plurality of sensors,
wherein the specific sensor is determined based upon sampled data packets of the particular network flow.
15 . The system of claim 11 , wherein the instructions, when executed by the processor, cause the system to perform operations further comprising:
reconciling the data packets of the particular network flow received from the plurality of sensors; and
determining non-duplicative data packets of the particular network flow,
wherein the specific sensor has sensed the most number of the non-duplicative data packets of the particular network flow among the plurality of sensors
16 . The system of claim 11 , wherein the data packets of the particular network flow comprise a set of information to uniquely identify the particular network flow, the set of information including a source address, a destination address, a source port, destination port, a protocol, a user identification (ID), and a starting timestamp, and wherein the instructions, when executed by the processor, cause the system to perform operations further comprising:
analyzing the data packets from the plurality of sensors to determine a starting timestamp for each of the data packets,
wherein the specific sensor sensed the earliest data packet of the particular network flow.
17 . The system of claim 11 , wherein the particular network flow is a transmission control protocol (TCP) network flow, and wherein the instructions, when executed by the processor, cause the system to perform operations further comprising:
determining a start of the TCP network flow based upon a three-way hand-shake; and
determining an end of the TCP network flow based upon a four-way hand-shake.
18 . A non-transitory computer-readable storage medium storing instructions for de-duplicating sensed data packets in a network, that, when executed by at least one processor of a computing system, cause the computing system to perform operations comprising:
receiving, from a plurality of sensors in the network, data packets of a particular network flow of the network;
analyzing the data packets to determine a specific sensor of the plurality of sensors;
preserving data packets sensed and reported from the specific sensor;
determining duplicative data packets sensed and reported from other sensor(s) of the plurality of sensors based upon the data packets reported from the specific sensor;
discarding the duplicative data packets sensed and reported from the other sensor(s); and
analyzing the particular network flow based upon the data packets reported from the specific sensor and non-duplicative data packets reported from the other sensor(s).
19 . The non-transitory computer-readable storage medium of claim 18 , wherein the instructions, when executed by the at least one processor, cause the computing system to perform operations further comprising:
analyzing the data packets from the plurality of sensors to determine a number of data packets sensed by each sensor of the plurality of sensors,
wherein the specific sensor has sensed the most number of data packets of the particular network flow among the plurality of sensors.
20 . The non-transitory computer-readable storage medium of claim 18 , wherein the data packets of the particular network flow comprise a set of information to uniquely identify the particular network flow, the set of information including a source address, a destination address, a source port, destination port, a protocol, a user identification (ID), and a starting timestamp, and wherein the instructions, when executed by the at least one processor, cause the computing system to perform operations further comprising:
analyzing the data packets from the plurality of sensors to determine a starting timestamp for each of the data packets,
wherein the specific sensor sensed the earliest data packet of the particular network flow.