Adaptive delta compression for timeseries data
Example systems and methods that perform adaptive delta compression for timeseries data are described. In one implementation, one or more computer processors analyze timeseries data to identify properties about the data. The one or more computer processors select at least one compression primitive based on at least one test. A window size is dynamically changed based on the analysis results and the at least one compression primitive.
1. A method comprising:
analyzing, using one or more computer processors, timeseries data, wherein the analysis includes:
processing at least a portion of the timeseries data to generate a set of frames with scores for each of one or more compression algorithms; and
comparing the scores;
determining whether the timeseries data is compressible based on the comparing;
if the timeseries data is determined to be compressible:
testing a data compression performance of an adaptive delta compression method on the timeseries data;
responsive to the data compression performance being greater than a threshold, selecting the adaptive delta compression method as a compression primitive, wherein the adaptive delta compression method includes selecting between a delta compression method and a delta-delta compression method;
responsive to the data compression performance being less than the threshold, selecting a modified LZ4 compression method as the compression primitive;
dynamically changing a data window size based on the analysis results and the compression primitive; and
if the timeseries data is determined to be not compressible, performing no compression on the timeseries data.
2. The method of claim 1 , wherein analyzing the timeseries data includes scanning the timeseries data to determine data properties.
3. The method of claim 2 , wherein the data properties includes at least one of timestamp data, integer data, and doubles data.
4. The method of claim 2 , wherein the data properties include at least one of monotonic data, high entropy data, and low entropy data.
5. The method of claim 2 , wherein the one or more computer processors further select a data processing model based on the data properties, wherein the data processing model includes at least one of a model for strictly monotonic data, a model to detect sensor data, a model to detect financial data, and a model for dispersion and variance.
6. The method of claim 1 , wherein the compression primitive is an algorithm that receives an uncompressed text and outputs a compressed text.
7. The method of claim 6 , further comprising performing a statistical analysis of the compression primitive to determine its compression efficiency.
8. The method of claim 1 , wherein dynamically changing the window size includes defining a plurality of windows, where each of the plurality of windows is associated with a type of data.
9. The method of claim 1 , further comprising compressing the timeseries data using an LZ4 data compression algorithm.
10. The method of claim 1 , further comprising compressing the timeseries data using at least one of a delta, delta-delta, and modified LZ4 data compression algorithm.
11. The method of claim 1 , wherein the at least one compression primitive is selected from a compression primitives library.
12. The method of claim 1 , further comprising:
compressing the timeseries data using a first compression algorithm to generate a first compression result; and
compressing the timeseries data using a second compression algorithm to generate a second compression result, wherein the first and second compression results are generated simultaneously.
13. The method of claim 12 , further comprising:
analyzing the first compression result;
analyzing the second compression result; and
selecting the first compression result or the second compression result depending on the best compression result.
14. The method of claim 12 , wherein the simultaneous generation of compression results is performed using a single instruction, multiple data (SIMD) operation.
15. A method comprising:
analyzing, using one or more computer processors, a set of samples of timeseries data to identify properties associated with the data, wherein the analysis includes:
processing at least a portion of the timeseries data to generate a set of frames with scores for each of one or more compression algorithms; and
comparing the scores;
determining, based on the analysis, whether the timeseries data is compressible based on the comparing;
if the timeseries data is determined to be compressible:
selecting, using one or more computer processors, at least one compression primitive based on at least one test;
compressing the set of samples of the timeseries data using a first compression algorithm to generate a first compression result; and
compressing the set of samples of the timeseries data using a second compression algorithm to generate a second compression result, wherein the first and second compression results are generated simultaneously.
16. The method of claim 15 , further comprising dynamically changing a data window size based on the analysis results and the at least one compression primitive.
17. The method of claim 16 , wherein dynamically changing the window size includes defining a plurality of windows, where each of the plurality of windows is associated with a type of data.
18. The method of claim 15 , wherein analyzing the timeseries data includes scanning the timeseries data to identify data properties that include at least one of timestamp data, integer data, and doubles data.
19. The method of claim 15 , wherein analyzing the timeseries data includes scanning the timeseries data to identify data properties that include at least one of monotonic data, high entropy data, and low entropy data.
20. The method of claim 15 , wherein the one or more computer processors further select a data processing model based on the data properties, wherein the data processing model includes at least one of a model for strictly monotonic data, a model to detect sensor data, a model to detect financial data, and a model for dispersion and variance.
21. The method of claim 15 , wherein the first compression algorithm includes one of a delta compression algorithm, a delta-delta compression algorithm, an LZ4 compression algorithm, and a modified LZ4 compression algorithm.
22. The method of claim 15 , wherein the simultaneous generation of compression results is performed using a single instruction, multiple data (SIMD) operation.
23. A method comprising:
analyzing, using one or more computer processors, timeseries data;
testing a data compression performance of an adaptive delta compression method on the timeseries data;
responsive to the data compression performance being greater than a threshold, selecting the adaptive delta compression method as a compression primitive;
responsive to the data compression performance being less than the threshold, selecting a modified LZ4 compression method as the compression primitive;
dynamically changing a data window size based on the analysis results and the compression primitive; and
defaulting to the modified LZ4 compression method based on the data compression performance testing generating uncertain results.
24. The method of claim 23 , wherein the uncertain results are generated based on a randomness of the timeseries data.
25. The method of claim 24 , wherein a dispersion associated with the timeseries data is used as a measure of the randomness.