Methods, systems, and computer readable media for rapid filtering of opaque data traffic
Methods, systems, and computer readable media for rapid filtering of opaque data traffic are disclosed. According to one method, the method includes receiving a packet containing a payload. The method also includes analyzing a portion of the payload for determining whether the packet contains compressed or encrypted data. The method further includes performing, if the packet contains compressed or encrypted data, at least one of sending the packet to an opaque traffic analysis engine for analysis, discarding the packet, logging the packet, or marking the packet.
1. A method for rapid filtering of opaque data traffic, the method comprising:
at an opaque traffic filter (OTF) module implemented using software executed by a processor, wherein the processor includes a data acquisition and generation card (DAG), a graphics processing unit (GPU), or a general-purpose processor:
receiving a packet containing a payload;
analyzing a portion of the payload to determine whether the packet contains compressed or encrypted data, wherein analyzing the portion of the payload to determine whether the packet contains compressed or encrypted data includes using a null hypothesis indicating the portion of the payload is opaque data and an alternate hypothesis indicating that the portion of the payload is transparent data, wherein the alternate hypothesis uses a probability density indicating that a majority of byte values in the portion of the payload are ASCII values, wherein determining that the packet contains compressed or encrypted data includes determining that a byte value distribution of the portion of the packet supports the null hypothesis and fails to support the alternate hypothesis, wherein determining whether the packet contains compressed or encrypted data includes using at least one of a fixed sample-size hypothesis test or an autocorrelation function; and
performing in response to determining that the packet contains compressed or encrypted data, at least one of sending the packet to an opaque traffic analysis engine for analysis, discarding the packet, logging the packet, or marking the packet, wherein the receiving, the analyzing, and the performing are performed by the data acquisition and generation card (DAG), the graphics processing unit (GPU), or the general-purpose processor.
2. The method of claim 1 wherein receiving the packet includes observing and copying the packet from a plurality of packets traversing a link or node.
3. The method of claim 1 wherein the portion of the payload analyzed is less than or equal to the entire payload.
4. The method of claim 1 wherein the portion of the payload analyzed includes about 16 bytes or less than 16 bytes.
5. The method of claim 1 wherein determining that the portion of the payload includes uniformly distributed byte values indicates that the packet contains compressed or encrypted data.
6. The method of claim 1 wherein at least one of a sequential hypothesis test, the fixed sample-size hypothesis test, or the autocorrelation function is used in differentiating an encrypted packet and a compressed packet.
7. The method of claim 1 wherein the DAG, the GPU, or the general-purpose processor performs statistical hypothesis tests or statistical fingerprinting.
8. A system for rapid filtering of opaque data traffic, the system comprising:
a processor, wherein the processor includes a data acquisition and generation card (DAG), a graphics processing unit (GPU), or a general-purpose processor; and
an opaque traffic filter (OTF) module implemented using software executed by the processor, the OTF module configured to receive a packet containing a payload, to analyze a portion of the payload to determine whether the packet contains compressed or encrypted data, wherein analyzing the portion of the payload to determine whether the packet contains compressed or encrypted data includes using a null hypothesis indicating the portion of the payload is opaque data and an alternate hypothesis indicating that the portion of the payload is transparent data, wherein the alternate hypothesis uses a probability density indicating that a majority of byte values in the portion of the payload are ASCII values, wherein determining that the packet contains compressed or encrypted data includes determining that a byte value distribution of the portion of the packet supports the null hypothesis and fails to support the alternate hypothesis, wherein determining whether the packet contains compressed or encrypted data includes using at least one of a fixed sample-size hypothesis test or an autocorrelation function, and to perform, in response to determining that the packet contains compressed or encrypted data, at least one of sending the packet to an opaque traffic analysis engine for analysis, discarding the packet, logging the packet, or marking the packet.
9. The system of claim 8 wherein the system comprises a communications interface configured to observe and copy the packet from a plurality of packets traversing a link or node.
10. The system of claim 8 wherein the portion of the payload analyzed is less than or equal to the entire payload.
11. The system of claim 8 wherein the portion of the payload analyzed includes about 16 bytes or less than 16 bytes.
12. The system of claim 8 wherein determining that the portion of the payload includes uniformly distributed byte values indicates that the packet contains compressed or encrypted data.
13. The system of claim 8 wherein the OTF module is configured to use at least one of a sequential hypothesis test, the fixed sample-size hypothesis test, or the autocorrelation function in differentiating an encrypted packet and a compressed packet.
14. The system of claim 8 wherein the OTF module is configured to inform a packet inspection analysis engine when a session becomes opaque or transparent and/or when a packet of the session is examined that does not match an expected type for the session.
15. The system of claim 8 wherein the processor performs statistical hypothesis tests or statistical fingerprinting.
16. A non-transitory computer readable medium having stored thereon executable instructions that when executed by a processor of a computer control the computer to perform steps comprising:
at an opaque traffic filter (OTF) module implemented using software executed by a processor, wherein the processor includes a data acquisition and generation card (DAG), a graphics processing unit (GPU), or a general-purpose processor:
receiving a packet containing a payload;
analyzing a portion of the payload to determine whether the packet contains compressed or encrypted data, wherein analyzing the portion of the payload to determine whether the packet contains compressed or encrypted data includes using a null hypothesis indicating the portion of the payload is opaque data and an alternate hypothesis indicating that the portion of the payload is transparent data, wherein the alternate hypothesis uses a probability density indicating that a majority of byte values in the portion of the payload are ASCII values, wherein determining that the packet contains compressed or encrypted data includes determining that a byte value distribution of the portion of the packet supports the null hypothesis and fails to support the alternate hypothesis, wherein determining whether the packet contains compressed or encrypted data includes using at least one of a fixed sample-size hypothesis test or an autocorrelation function; and
performing, in response to determining that the packet contains compressed or encrypted data, at least one of sending the packet to an opaque traffic analysis engine for analysis, discarding the packet, logging the packet, or marking the packet, wherein the receiving, the analyzing, and the performing are performed by a data acquisition and generation card (DAG), a graphics processing unit (GPU), or a general-purpose processor.