System and method for encrypted data compression
A system and method for encrypted data compression, which uses frequency analysis on data blocks within an input data stream to produce a prefix table, representing a first layer of transformation, and which applies a Burrow's-Wheeler transform (BWT) to the data inside the prefix table, representing a second layer of transformation, and which compresses the transformed data. In some implementations, the system and method may further include applying the BWT to a conditioned stream of genomic data, wherein the conditioned stream of data is accompanied by an error stream comprising the differences between the original data and the encrypted data.
1. A system for encrypted data compression, comprising:
a computing device comprising a processor and a memory;
a stream conditioner comprising a plurality of programming instructions stored in the memory which, when operating on the processor, causes the computing device to:
receive an input data stream comprising a plurality of data blocks;
analyze the input data stream to:
determine real frequencies of data blocks within the input data stream;
identify data blocks whose real frequencies deviate from ideal frequencies by more than a configured conditioning threshold;
generate a conditioned data stream having the same length as the input data stream by:
applying a conditioning rule to the identified data blocks to generate conditioned data blocks;
using the remaining data blocks from the input data stream unchanged; and
arranging the conditioned data blocks and unchanged data blocks in positions corresponding to their original positions in the input data stream;
create an error stream through sequential XOR operations, wherein:
for each position in the input data stream:
performing, a logical XOR operation between the data block in the input data stream and the corresponding data block in the conditioned data stream wherein the XOR result at each position directly forms the corresponding position of the error stream, such that the error stream is composed of the sequential XOR results; and
send the conditioned data stream and the error stream as output.
2. The system of claim 1 , further comprising a stream analyzer comprising a second plurality of programming instructions stored in the memory which, when operating on the processor, causes the computing device to:
receive an input data stream;
analyze the frequency distribution of a plurality of data blocks within the input data stream to determine whether the input data stream meets a configured threshold for data conditioning; and
if the input data stream meets or exceeds the configured threshold, send the input data stream to the stream conditioner.
3. The system of claim 2 , wherein the stream analyzer is further configured to:
analyze the frequency distribution of the plurality of data blocks within the input data stream to determine the most frequent bytes or strings of bytes that occur at the beginning of each of the plurality of data blocks;
designate the most-frequent bytes or strings of bytes as prefixes;
compile a prefix table based on the results of the frequency distribution, the prefix table comprising the designated prefixes and the block lengths; and
send the prefix table to a data transformer.
4. The system of claim 3 , further comprising the data transformer comprising a third plurality of programming instructions stored in the memory which, when operating on the processor, causes the computing device to:
receive the prefix table;
transform each of the plurality of prefixes within the prefix table by applying a Burrow's-Wheeler transform (BWT) and generating as output a plurality BWT-prefixes; and
send the plurality of BWT-prefixes as output.
5. The system of claim 3 , wherein each of the plurality of data blocks are k-mers of genomic data and the prefixes are one or more base pairs that occur at the beginning of each k-mer.
6. A method for encrypted data compression, comprising the steps of:
receiving, at a stream conditioner, an input data stream comprising a plurality of data blocks;
analyzing the input data stream to:
determine real frequencies of data blocks within the input data stream;
identify data blocks whose real frequencies deviate from ideal frequencies by more than a configured conditioning threshold;
generating a conditioned data stream having the same length as the input data stream by:
applying a conditioning rule to the data blocks to generate conditioned data blocks;
using the remaining data blocks from the input data stream unchanged;
arranging the conditioned data blocks and unchanged data blocks in positions corresponding to their original positions in the input data stream;
creating an error stream through sequential XOR operations, wherein:
for each position in the input data stream:
performing, a logical XOR operation between the data block in the input data stream and the corresponding data block in the conditioned data stream, wherein the XOR result at each position directly forms the corresponding position of the error stream, such that the error stream is composted of the sequential XOR results; and
sending the conditioned data stream and the error stream as output.
7. The method of claim 6 , further comprising the steps of:
receiving, at a stream analyzer, an input data stream;
analyzing the frequency distribution of a plurality of data blocks within the input data stream to determine whether the input data stream meets a configured threshold for data conditioning; and
if the input data stream meets or exceeds the configured threshold, sending the input data stream to the stream conditioner.
8. The method of claim 7 , further comprising the steps of:
analyzing the frequency distribution of the plurality of data blocks within the input data stream to determine the most frequent bytes or strings of bytes that occur at the beginning of each of the plurality of data blocks;
designating the most-frequent bytes or strings of bytes as prefixes;
compiling a prefix table based on the results of the frequency distribution, the prefix table comprising the designated prefixes and the block lengths; and
sending the prefix table to a data transformer.
9. The method of claim 7 , further comprising the steps of:
receiving the prefix table using the data transformer;
transforming each of the plurality of prefixes within the prefix table by applying a Burrow's-Wheeler transform (BWT) and generating as output a plurality BWT-prefixes using the data transformer; and
sending the plurality of BWT-prefixes as output using the data transformer.
10. The method of claim 7 , wherein each of the plurality of data blocks are k-mers of genomic data and the prefixes are one or more base pairs that occur at the beginning of each k-mer.