Traffic deduplication in a visibility network

1. A method for implementing traffic deduplication in a visibility network, the method comprising:

receiving, by a packet broker of the visibility network, a control or data packet replicated from a core network;

applying, by the packet broker, a first stage process in which the packet broker attempts to deduplicate the control or data packet based on one or more interfaces of the core network from which the control or data packet originated; and

applying, by the packet broker, a second stage process in which the packet broker attempts to deduplicate the control or data packet based on content of the control or data packet by:

calculating a cyclical redundancy check (CRC) value for the control or data packet based on the packet's payload;

determining whether the CRC value matches a stored CRC value of a previously received packet from a plurality of stored CRC values in a CRC table; and

in response to determining that the CRC value matches the stored CRC value of the previously received packet:

comparing one or more parameters in a header of the control or data packet with corresponding parameters in a header of the previously received packet,

in response to determining that the one or more parameters of the control or data packet match the corresponding parameters of the previously received packet, dropping the control or data packet, and

in response to determining that the one or more parameters of the control or data packet do not match the corresponding parameters of the previously received packet, adding the CRC value to the CRC table, and allowing the control or data packet to be forwarded onward to a probe or tool in the visibility network for analysis.

2. The method of claim 1 wherein applying the first stage process comprises:

determining the one or more interfaces based on header information in the control or data packet.

3. The method of claim 1 wherein applying the first stage process comprises:

applying one or more filtering rules for allowing or dropping the control or data packet in view of the one or more interfaces.

4. The method of claim 3 wherein the core network is a mobile LTE network, and

wherein the one or more filtering rules include a rule indicating that all traffic originating from an S5/S8 interface of the mobile LTE network should be dropped.

5. The method of claim 1 , wherein applying the second stage process further comprises:

if the CRC value matches a stored CRC value, dropping the control or data packet; and

if the CRC value does not match any of the stored CRC values:

adding the CRC value to the CRC table; and

allowing the control or data packet to be forwarded onward to a probe or tool in the visibility network for analysis.

6. A non-transitory computer readable storage medium having stored thereon program code executable by a packet broker in a visibility network, the program code causing the packet broker to:

receive a control or data packet replicated from a core network;

apply a first stage process in which the packet broker attempts to deduplicate the control or data packet based on one or more interfaces of the core network from which the control or data packet originated; and

apply a second stage process in which the packet broker attempts to deduplicate the control or data packet based on content of the control or data packet by:

calculating a cyclical redundancy check (CRC) value for the control or data packet based on the packet's payload;

determining whether the CRC value matches a stored CRC value of a previously received packet from a plurality of stored CRC values in a CRC table; and

in response to determining that the CRC value matches the stored CRC value of the previously received packet:

comparing one or more parameters in a header of the control or data packet with corresponding parameters in a header of the previously received packet,

in response to determining that the one or more parameters of the control or data packet match the corresponding parameters of the previously received packet, dropping the control or data packet, and

in response to determining that the one or more parameters of the control or data packet do not match the corresponding parameters of the previously received packet, adding the CRC value to the CRC table, and allowing the control or data packet to be forwarded onward to a probe or tool in the visibility network for analysis.

7. The non-transitory computer readable storage medium of claim 6 wherein the program code that causes the packet broker to apply the first stage process comprises program code that causes the packet broker to:

determine the one or more interfaces based on header information in the control or data packet.

8. The non-transitory computer readable storage medium of claim 6 wherein the program code that causes the packet broker to apply the first stage process comprises program code that causes the packet broker to:

apply one or more filtering rules for allowing or dropping the control or data packet in view of the one or more interfaces.

9. The non-transitory computer readable storage medium of claim 8 wherein the core network is a mobile LTE network, and

wherein the one or more filtering rules include a rule indicating that all traffic originating from an S5/S8 interface of the mobile LTE network should be dropped.

10. The non-transitory computer readable storage medium of claim 6 , wherein the program code that causes the packet broker to apply the second stage process further comprises program code that causes the packet broker to:

if the CRC value matches a stored CRC value, drop the control or data packet; and

if the CRC value does not match any of the stored CRC values:

add the CRC value to the CRC table; and

allow the control or data packet to be forwarded onward to a probe or tool in the visibility network for analysis.

11. A packet broker for use in a visibility network, the packet broker comprising:

a processor; and

a memory stored thereon program code that, when executed, causes the processor to:

receive a control or data packet replicated from a core network;

apply a first stage process in which the processor attempts to deduplicate the control or data packet based on one or more interfaces of the core network from which the control or data packet originated; and

apply a second stage process in which the processor attempts to deduplicate the control or data packet based on content of the control or data packet by:

calculating a cyclical redundancy check (CRC) value for the control or data packet based on the packet's payload;

determining whether the CRC value matches a stored CRC value of a previously received packet from a plurality of stored CRC values in a CRC table; and

in response to determining that the CRC value matches the stored CRC value of the previously received packet:

comparing one or more parameters in a header of the control or data packet with corresponding parameters in a header of the previously received packet,

in response to determining that the one or more parameters of the control or data packet match the corresponding parameters of the previously received packet, dropping the control or data packet, and

in response to determining that the one or more parameters of the control or data packet do not match the corresponding parameters of the previously received packet, adding the CRC value to the CRC table, and allowing the control or data packet to be forwarded onward to a probe or tool in the visibility network for analysis.

12. The packet broker of claim 11 wherein the program code that causes the processor to apply the first stage process comprises program code that causes the processor to:

determine the one or more interfaces based on header information in the control or data packet.

13. The packet broker of claim 11 wherein the program code that causes the processor to apply the first stage process comprises program code that causes the processor to:

apply one or more filtering rules for allowing or dropping the control or data packet in view of the one or more interfaces.

14. The packet broker of claim 13 wherein the core network is a mobile LTE network, and

wherein the one or more filtering rules include a rule indicating that all traffic originating from an S5/S8 interface of the mobile LTE network should be dropped.

15. The packet broker of claim 11 wherein the program code that causes the processor to apply the second stage process further comprises program code that causes the processor to:

if the CRC value matches a stored CRC value, drop the control or data packet; and

if the CRC value does not match any of the stored CRC values:

add the CRC value to the CRC table; and

allow the control or data packet to be forwarded onward to a probe or tool in the visibility network for analysis.