Method and system of content based dynamic data compression
View Patent ↗The present disclosure describes methods and systems that provide for content based dynamic data compression. During a training stage a plurality of training sets are compressed using multiple data compression techniques, and each training data set is associated with a most effective compression technique. Statistical features can be extracted from each of the training data sets with consideration of the most effective compression techniques and are saved as a compression parameters file. During a working stage, individual data blocks within a telemetry data set are compressed using a data compression technique selected for the data block in light of the statistical feature presented in the compression parameters file, and the compressed data blocks are written to a compressed data set, along with encoding tags that identify the compression technique used on each data block. During a consumption stage, the compressed data blocks are decompressed using techniques identified by the encoding tags.
1. A method comprising:
processing a plurality of training data sets, wherein the plurality of training data sets include at least a first training data set and a second training data set, the processing comprising:
compressing the first training data set using a first compression technique of a plurality of compression techniques;
compressing the first training data set using a second compression technique of the plurality of compression techniques;
performing a first comparison of:
compressing the first training data set using the first compression technique, with
compressing the first training data using the second compression technique;
determining, based on the first comparison, that compressing the first training data set using the first compression technique has a smaller size as compared to using the second compression technique;
compressing the second training data set using the first compression technique;
compressing the second training data set using the second compression technique;
performing a second comparison of:
compressing the second training data set using the first compression technique, with
compressing the second training data set the second compression technique;
determining, based on the second comparison, that compressing the second training data set using the second compression technique has a smaller size as compared to using the first compression technique;
associating the first training data set with the first compression technique;
associating the second training data set with the second compression technique;
creating a first table based on the first training data set;
creating a second table based on the second training data set;
creating a first matrix corresponding to the first compression technique, the first matrix comprising first rows and first columns;
creating a second matrix corresponding to the second compression technique, the second matrix comprising second rows and second columns;
wherein at least one of entry in the first matrix and the second matrix corresponds to immediately adjacent symbols in the first and second training data sets; and
sending one or more tables including the first table and the second table to one or more computer systems for use in compressing sets of data stored on each of the one or more computer systems.
2. The method of claim 1 , wherein the plurality of compression techniques comprises at least two of:
run length encoding (RLE) compression;
differential pulse code modulation (DPCM) compression;
Lempel, Ziv, Welch (LZW) compression;
Huffman code compression; and
GZIP compression.
3. The method of claim 1 wherein:
the at least one entry of the first matrix includes a total number of instances in which the immediately adjacent symbols occur in the first training data set, and
the at least one entry of the second matrix includes a total number of instances in which the immediately adjacent symbols occur in second training data set.
4. The method of claim 1 wherein the processing comprises:
compressing individual training data sets of the plurality of training data sets using the plurality of compression techniques;
determining a particular technique of the compression techniques that is most effective in producing a small file for compressing an individual training data sets of the plurality of training data sets;
associating the particular technique with the individual training data set; and
generating, for individual training data sets, a particular table based on the particular technique.
5. The method of claim 1 wherein:
at least one of the first and second tables comprises a number of rows that represents a total number of symbols included in the plurality of training data sets;
the number of rows correspond to the symbols;
each row of the number of rows comprises entries corresponding to each of the compression techniques; and
each particular entry in a row represents a probability that two immediately adjacent symbols are included in one or more training data sets of the plurality of training data sets that are mapped to the compression technique that is associated with the particular entry.
6. The method of claim 1 wherein:
at least one table of the one or more tables comprises a single row,
the single row comprising individual entries corresponding to individual ones of the compression techniques, respectively, and
individual entries representing a probability that a first two symbols of one or more training data sets of the plurality of training data sets is included in the one or more training data sets of the plurality of training sets that are associated with the compression technique corresponding to the individual entries.
7. The method according to claim 1 , wherein the processing further comprises:
determining information associated with one or more of:
a textual context of the plurality of training data sets,
a list of the plurality of compression techniques,
a definition of a data space defining symbols appearing in the plurality of training data sets,
counts of transitions between two consecutive symbols,
probabilities of the transitions between two consecutive symbols,
probabilities that a specific compression technique is used to transition between two consecutive symbols, and
probabilities of initial conditions.
8. The method of claim 1 , further comprising:
reading one or more data blocks from a working data set;
determining for each particular data block of the one or more data blocks, which particular compression technique of the plurality of compression techniques produces a smallest result when compressing each particular data block;
compressing each particular data block of the one or more data blocks using the particular compression technique determined to produce the smallest result for each particular data block; and
writing the compressed data blocks to a storage device to create a compressed data set.
9. A computing device comprising:
one or more processors; and
one or more non-transitory computer-readable storage media to store instructions executable by the one or more processors to perform operations comprising:
determining a plurality of training data sets that include at least a first training data set and a second training data set;
compressing the first training data set using a first compression technique of a plurality of compression techniques;
compressing the first training data set using a second compression technique of the plurality of compression techniques;
performing a first comparison of:
compressing the first training data set using the first compression technique, with
compressing the first training data using the second compression technique;
determining, based on the first comparison, that compressing the first training data set using the first compression technique has a smaller size as compared to using the second compression technique;
compressing the second training data set using the first compression technique;
compressing the second training data set using the second compression technique;
performing a second comparison of:
compressing the second training data set using the first compression technique, with
compressing the second training data set the second compression technique;
determining, based on the second comparison, that compressing the second training data set using the second compression technique has the smaller size as compared to using the first compression technique;
associating the first training data set with the first compression technique;
associating the second training data set with the second compression technique;
creating a first table based on the first training data set;
creating a second table based on the second training data set;
creating a first matrix corresponding to the first compression technique, the first matrix comprising first rows and first columns;
creating a second matrix corresponding to the second compression technique, the second matrix comprising second rows and second columns;
wherein at least one entry in the first matrix and the second matrix corresponds to immediately adjacent symbols in the first and second training data sets; and
sending one or more tables including the first table and the second table to one or more computer systems for use in compressing sets of data stored on each of the one or more computer systems.
10. The computing device of claim 9 , wherein:
the at least one entry of the first matrix includes a total number of instances in which the immediately adjacent symbols occur in the first training data set, and
the at least one entry of the second matrix includes a total number of instances in which the immediately adjacent symbols occur in second training data set.
11. The computing device of claim 9 , wherein the operations further comprise:
compressing individual training data sets of the plurality of training data sets using the plurality of compression techniques;
determining a particular technique of the compression techniques that is most effective in producing a small file for compressing an individual training data sets of the plurality of training data sets;
associating the particular technique with the individual training data set; and
generating, for individual training data sets, a particular table based on the particular technique.
12. The computing device of claim 9 , wherein:
at least one of the first and second tables comprises a number of rows that represent a total number of symbols in the plurality of training data sets;
the number of rows correspond to the symbols;
each row of the number of rows comprises entries corresponding to each of the compression techniques;
each particular entry in a row represents a probability that two immediately adjacent symbols are included in one or more training data sets of the plurality of training data sets that are mapped to the compression technique that is associated with the particular entry;
at least one table of the one or more tables comprises a single row;
the single row comprising individual entries corresponding to individual ones of the compression techniques, respectively; and
individual entries representing a probability that a first two symbols of one or more training data sets of the plurality of training data sets is included in the one or more training data sets of the plurality of training sets that are associated with the compression technique corresponding to the individual entries.
13. The computing device according to claim 9 , wherein the operations further comprise:
determining information associated with one or more of:
a textual context of the plurality of training data sets;
a list of the plurality of compression techniques;
a definition of a data space defining symbols appearing in the plurality of training data sets;
counts of transitions between two consecutive symbols;
probabilities of the transitions between two consecutive symbols;
probabilities that a specific compression technique is used to transition between two consecutive symbols; and
probabilities of initial conditions.
14. The computing device of claim 9 , the operations further comprise:
reading one or more data blocks from a working data set;
determining for each particular data block of the one or more data blocks, which particular compression technique of the plurality of compression techniques produces a smallest result when compressing each particular data block;
compressing each particular data block of the one or more data blocks using the particular compression technique determined to produce the smallest result for each particular data block; and
writing the compressed data blocks to a storage device to create a compressed data set.
15. One or more non-transitory computer-readable storage media to store instructions executable by one or more processors to perform operations comprising:
determining a plurality of training data sets that include at least a first training data set and a second training data set;
compressing the first training data set using a first compression technique of a plurality of compression techniques;
compressing the first training data set using a second compression technique of the plurality of compression techniques;
performing a first comparison of:
compressing the first training data set using the first compression technique, with
compressing the first training data using the second compression technique;
determining, based on the first comparison, that compressing the first training data set using the first compression technique has a smaller size as compared to using the second compression technique;
compressing the second training data set using the first compression technique;
compressing the second training data set using the second compression technique;
performing a second comparison of:
compressing the second training data set using the first compression technique, with
compressing the second training data set the second compression technique;
determining, based on the second comparison, that compressing the second training data set using the second compression technique has the smaller size as compared to using the first compression technique;
associating the first training data set with the first compression technique;
associating the second training data set with the second compression technique;
creating a first table based on the first training data set;
creating a second table based on the second training data set;
creating a first matrix corresponding to the first compression technique, the first matrix comprising first rows and first columns;
creating a second matrix corresponding to the second compression technique, the second matrix comprising second rows and second columns;
wherein at least one entry in the first matrix and the second matrix corresponds to immediately adjacent symbols in the first and second training data sets; and
sending one or more tables including the first table and the second table to one or more computer systems for use in compressing sets of data stored on each of the one or more computer systems.
16. The one or more non-transitory computer-readable storage media of claim 15 , wherein:
at least one entry of the first matrix includes a total number of instances in which the immediately adjacent symbols occur in the first training data set, and
at least one entry of the second matrix includes a total number of instances in which the immediately adjacent symbols occur in second training data set.
17. The one or more non-transitory computer-readable storage media of claim 15 , wherein the operations further comprise:
compressing individual training data sets of the plurality of training data sets using the plurality of compression techniques;
determining a particular technique of the compression techniques that is most effective in producing a small file for compressing an individual training data sets of the plurality of training data sets;
associating the particular technique with the individual training data set; and
generating, for individual training data sets, a particular table based on the particular technique.
18. The one or more non-transitory computer-readable storage media of claim 15 , wherein:
at least one of the first and second tables comprises a number of rows that represent a total number of symbols in the plurality of training data sets;
the number of rows correspond to the symbols;
each row of the number of rows comprises entries corresponding to each of the compression techniques;
each particular entry in a row represents a probability that two immediately adjacent symbols are included in one or more training data sets of the plurality of training data sets that are mapped to the compression technique that is associated with the particular entry;
at least one table of the one or more tables comprises a single row;
the single row comprising individual entries corresponding to individual ones of the compression techniques, respectively; and
individual entries representing a probability that a first two symbols of one or more training data sets of the plurality of training data sets is included in the one or more training data sets of the plurality of training sets that are associated with the compression technique corresponding to the individual entries.
19. The one or more non-transitory computer-readable storage media of claim 15 , wherein the operations further comprise:
determining information associated with one or more of:
a textual context of the plurality of training data sets;
a list of the plurality of compression techniques;
a definition of a data space defining symbols appearing in the plurality of training data sets;
counts of transitions between two consecutive symbols;
probabilities of the transitions between two consecutive symbols;
probabilities that a specific compression technique is used to transition between two consecutive symbols; and
probabilities of initial conditions.
20. The one or more non-transitory computer-readable storage media of claim 15 , wherein the operations further comprise:
reading one or more data blocks from a working data set;
determining for each particular data block of the one or more data blocks, which particular compression technique of the plurality of compression techniques produces a smallest result when compressing each particular data block;
compressing each particular data block of the one or more data blocks using the particular compression technique determined to produce the smallest result for each particular data block; and
writing the compressed data blocks to a storage device to create a compressed data set.