Data compression with signature-based intrusion detection
A system and method for data compression with intrusion detection, that measures in real-time the probability distribution of an encoded data stream, compares the probability distribution to a reference probability distribution, and uses one or more statistical algorithms to determine the divergence between the two sets of probability distributions to determine if an unusual distribution is the result of a data intrusion. The system further comprises a signature generating component which correlates anomalous event data with known vulnerabilities and exploits to create a signature based on statistical information of the anomalous event. Computed statistics may be compared against a signature database to determine if a data intrusion has occurred.
1. A system for data compression with intrusion detection, comprising:
a plurality of computing devices each comprising at least a processor, a memory, and a network interface;
wherein a plurality of programming instructions stored in one or more of the memories and operating on one or more of the processors of the plurality of computing devices causes the plurality of computing devices to:
receive anomalous event data, the anomalous event data comprising a computed divergence of a data stream, a computed first probability distribution of the data stream, and one or more codewords;
compare the anomalous event data to a database, the database comprising a plurality of signatures, the plurality of signatures comprising statistical information associated with a plurality of known vulnerabilities;
when the comparison yields a match:
generate an intrusion alert, the intrusion alert comprising the anomalous event data; and
send the intrusion alert to a security monitoring system.
2. The system of claim 1 , wherein the plurality of programming instructions further causes the plurality of computing devices to:
receive a first codeword data stream;
compute the first probability distribution of a plurality of codewords within the first codeword data stream; and
compute the divergence between the first computed probability distribution and a reference probability distribution.
3. The system of claim 1 , wherein the plurality of programming instructions further causes the plurality of computing devices to:
obtain a plurality of historical anomalous event data;
obtain threat intelligence data, the threat intelligence data comprising at least the plurality of known vulnerabilities and known malicious actor information;
generate a signature for a known malicious actor based on the plurality of historical anomalous event data and the plurality of known vulnerabilities; and
store the signature in the database.
4. The system of claim 3 , wherein the signature comprises statistical information associated with a known vulnerability, the statistical information being derived from the historical anomalous event data.
5. The system of claim 4 , wherein the statistical information comprises at least a historical divergence and a historical probability distribution.
6. The system of claim 3 wherein the plurality of programming instructions further causes the plurality of computing devices to:
when the comparison does not yield a match:
generate a signature associated with the received anomalous event data; and
store the signature in the database.
7. A method for data compression with intrusion detection, comprising the steps of:
receiving anomalous event data, the anomalous event data comprising a computed divergence of a data stream, a computed first probability distribution of the data stream, and one or more codewords;
comparing the anomalous event data to a database, the database comprising a plurality of signatures, the plurality of signatures comprising statistical information associated with a plurality of known vulnerabilities;
when the comparison yields a match:
generating an intrusion alert, the intrusion alert comprising the anomalous event data; and
sending the intrusion alert to a security monitoring system.
8. The method of claim 7 , further comprising the steps of:
receiving a first codeword data stream;
computing the first probability distribution of a plurality of codewords within the first codeword data stream; and
computing the divergence between the first computed probability distribution and a reference probability distribution.
9. The method of claim 7 , further comprising the steps of:
obtaining a plurality of historical anomalous event data;
obtaining threat intelligence data, the threat intelligence data comprising at least the plurality of known vulnerabilities and known malicious actor information;
generating a signature for a known malicious actor based on the plurality of historical anomalous event data and the plurality of known vulnerabilities; and
storing the signature in the database.
10. The method of claim 9 , wherein the signature comprises statistical information associated with a known vulnerability, the statistical information being derived from the historical anomalous event data.
11. The method of claim 10 , wherein the statistical information comprises at least a historical divergence and a historical probability distribution.
12. The method of claim 9 further comprising the steps of:
when the comparison does not yield a match:
generating a signature associated with the received anomalous event data; and
storing the signature in the database.
13. A non-transitory computer-readable medium comprising a plurality of programming instructions that, when operating on a plurality of computing devices each comprising at least a processor, a memory, and a network interface, cause the plurality of computing devices to:
receive anomalous event data, the anomalous event data comprising a computed divergence of a data stream, a computed first probability distribution of the data stream, and one or more codewords;
compare the anomalous event data to a database, the database comprising a plurality of signatures, the plurality of signatures comprising statistical information associated with a plurality of known vulnerabilities; and
when the comparison yields a match:
generate an intrusion alert, the intrusion alert comprising the anomalous event data; and send the intrusion alert to a security monitoring system.