Data compression with intrusion detection
A system and method for data compression with intrusion detection, that measures in real-time the probability distribution of an encoded data stream, compares the probability distribution to a reference probability distribution, and uses one or more statistical algorithms to determine the divergence between the two sets of probability distributions to determine if an unusual distribution is the result of a data intrusion. The system comprises both encoding and decoding machines, an intrusion detection module, a codebook training module, and various databases which perform various analyses on encoded data streams.
1. A system for data compression with intrusion detection, comprising:
one or more computing devices, each comprising a processor, a memory, and a network interface;
wherein a plurality of programming instructions stored in one or more of the memories and operating on one or more of the processors of the one or more computing devices causes one or more of the computing devices individually or some combination of the computing devices operating together to:
receive a first codeword data stream;
compute a first probability distribution of a plurality of codewords within the first codeword data stream;
receive a second codeword data stream comprising a test dataset;
use one or more algorithms to compute a second probability distribution of the test dataset;
compare the first probability distribution and the second probability distribution to compute a difference in distribution statistics between the test dataset and the first codeword stream;
check whether the difference in distributions exceeds a pre-determined difference threshold, and when the difference in distributions exceeds the difference threshold:
generate an intrusion alert, the intrusion alert comprising an indicia of anomalous behavior based on the difference in distributions;
send the intrusion alert to a security monitoring system;
use the test dataset to retrain codeword encoding and decoding algorithms;
utilize the retrained algorithms to create one or more new data sourceblocks; and
create a new codeword for one or more of the plurality of codewords of the first codeword data stream using the one or more new data sourceblocks.
2. The system of claim 1 , wherein the plurality of programming instructions further causes one or more computing devices to:
train a machine learning algorithm using a training dataset to create reference probability distributions;
generate the second probability distribution for the test dataset of the received second codeword data stream using the trained machine learning algorithm;
store each new data sourceblock and its associated new codeword in an updated codebook; and
send the updated codebook to one or more of the one or more computing devices.
3. The system of claim 1 , wherein the plurality of programming instructions further causes one or more computing devices to:
receive a plurality of codewords from a codeword storage; and
send the plurality of codewords as a codeword data stream to the one or more computing devices for intrusion detection.
4. A method for data compression with intrusion detection, comprising the steps of:
receiving a first codeword data stream at one or more computing devices, each computing device comprising a processor, a memory, and a network interface;
using the one or more computing devices to perform the steps of:
computing a first probability distribution of a plurality of codewords within the first codeword data stream;
receiving a second codeword data stream comprising a test dataset;
using one or more algorithms to compute a second probability distribution of the test dataset;
comparing the first computed probability distribution and the second probability distribution;
checking whether the difference in distributions exceeds a pre-determined difference threshold, and when the difference in distributions exceeds the difference threshold:
generating an intrusion alert, the intrusion alert comprising an indicia of anomalous behavior based on the difference in distributions; and
sending the intrusion alert to a security monitoring system;
using the test dataset to retrain codeword encoding and decoding algorithms;
utilizing the retrained algorithms to create one or more new data sourceblocks; and
creating a new codeword for one or more of the plurality of codewords of the first codeword data stream using the one or more new data sourceblocks.
5. The method of claim 4 , further comprising the steps of using the one or more computing devices to perform the steps of:
training a machine learning algorithm using a training dataset to create reference probability distributions;
generating the second probability distribution for the test dataset of the received second codeword data stream using the trained machine learning algorithm;
storing each new data sourceblock and its associated new codeword in an updated codebook; and
sending the updated codebook to one or more of the one or more computing devices.
6. The method of claim 4 , further comprising the steps of:
receiving a plurality of codewords from a codeword storage; and
sending the plurality of codewords as a codeword data stream to the one or more computing devices for intrusion detection.
7. A non-transitory, computer-readable medium comprising a plurality of programming instructions that, when operating on one or more computing devices each comprising at least a processor, a memory, and a network interface, causes the one or more computing devices to perform the steps of:
receiving a first codeword data stream at one or more computing devices, each computing device comprising a processor, a memory, and a network interface;
using the one or more computing devices to perform the steps of:
computing a first probability distribution of a plurality of codewords within the first codeword data stream;
receiving a second codeword data stream comprising a test dataset; using one or more algorithms to compute a second probability distribution of the test dataset;
comparing the first probability distribution and the second probability distribution;
checking whether the difference in distributions exceeds a pre-determined difference threshold, and when the difference in distributions exceeds the difference threshold:
generating an intrusion alert, the intrusion alert comprising an indicia of anomalous behavior based on the difference in distributions; and
sending the intrusion alert to a security monitoring system;
using the test dataset to retrain codeword encoding and decoding algorithms;
utilizing the retrained algorithms to create one or more new data sourceblocks; and
creating a new codeword for one or more of the plurality of codewords of the first codeword data stream using the one or more new data sourceblocks.
8. The method of claim 7 , further comprising the step of using the one or more computing devices to perform the steps of:
training a machine learning algorithm using a training dataset to create reference probability distributions;
generating the second probability distribution for the test dataset of the received second codeword data stream using the trained machine learning algorithm;
storing each new data sourceblock and its associated new codeword in an updated codebook; and
sending the updated codebook to one or more of the one or more computing devices.
9. The method of claim 7 , further comprising the step of using the one or more computing devices to perform the steps of:
receiving a plurality of codewords from a codeword storage; and
sending the plurality of codewords as a codeword data stream to the one or more computing devices for intrusion detection.