Systems and methods for dynamic compression of time-series data
In an embodiment, a method includes receiving, from a data source, time-series data of a time-series data stream produced by the data source. The method further includes identifying a target compression algorithm for the time-series data, wherein the target compression algorithm is linked to the data source in memory pursuant to a dynamically-variable assignment. The method also includes compressing the time-series data using the target compression algorithm and transmitting the compressed time-series data to a destination. Furthermore the method includes periodically optimizing the dynamically-variable assignment in real-time as the time-series data is received.
1. A method comprising, by a computer system:
receiving, from a data source, time-series data of a time-series data stream produced by the data source;
identifying a target compression algorithm for the time-series data, wherein the target compression algorithm is linked to the data source in memory pursuant to a dynamically-variable assignment;
compressing the time-series data using the target compression algorithm;
transmitting the compressed time-series data to a destination;
accessing a sample of the time-series data stream in relation to a sample period;
determining a time density of time-series data production by the data source over one or more intervals of the sample period, wherein the time density of time-series data production refers to a frequency with which the time-series data is produced for compression such that greater production over the one or more intervals constitutes a greater time density and less production over the one or more intervals constitutes a lower time density;
generating a time-series profile of the sample using the time density;
comparing attributes of the time-series profile to stored algorithm signatures comprising attributes of candidate compression algorithms, the comparing comprising identifying similarities between the time-series profile and the attributes of the candidate compression algorithms;
selecting a compression algorithm from among a plurality of compression algorithms based at least in part on a result of the comparing; and
causing subsequent time-series data received from the data source to be compressed using the selected compression algorithm.
2. The method of claim 1 wherein, responsive to the selected compression algorithm being different from the target compression algorithm, the causing comprises dynamically modifying the dynamically-variable assignment such that the selected compression algorithm is assigned to the data source.
3. The method of claim 1 , wherein, responsive to the selected compression algorithm being the target compression algorithm, the causing comprises compressing the subsequent time-series data using the target compression algorithm.
4. The method of claim 1 , comprising:
transmitting a request for algorithm experience data to a data processor, wherein the transmitting comprises the time-series profile;
responsive to the request, receiving algorithm performance data in relation to at least one compression algorithm;
comparing the received algorithm performance data to algorithm performance data for the target compression algorithm;
identifying based, at least in part, on a sampling and profiling of the time-series data that the at least one compression algorithm results in at least one of a higher compression ratio, a lower time to compress and a lower time to decompress; and
responsive to the identifying, causing subsequent time-series data received from the data source to be compressed using the at least one compression algorithm.
5. The method of claim 1 , wherein the method is performed in parallel with respect to a plurality of data sources.
6. The method of claim 1 , wherein:
the attributes of the time-series profile and the stored attributes of each of the candidate compression algorithms are each represented as a feature vector; and
the comparing attributes comprises determining a similarity between the feature vector of the time-series profile and the feature vector of each of the candidate compression algorithms.
7. The method of claim 6 , wherein:
the attributes of the time-series profile comprise at least one of a burst frequency and a burst size.
8. An information handling system comprising a processor, wherein the processor is operable to implement a method comprising:
receiving, from a data source, time-series data of a time-series data stream produced by the data source;
identifying a target compression algorithm for the time-series data, wherein the target compression algorithm is linked to the data source in memory pursuant to a dynamically-variable assignment;
compressing the time-series data using the target compression algorithm;
transmitting the compressed time-series data to a destination;
accessing a sample of the time-series data stream in relation to a sample period;
determining a time density of time-series data production by the data source over one or more intervals of the sample period, wherein the time density of time-series data production refers to a frequency with which the time-series data is produced for compression such that greater production over the one or more intervals constitutes a greater time density and less production over the one or more intervals constitutes a lower time density;
generating a time-series profile of the sample using the time density;
comparing attributes of the time-series profile to stored algorithm signatures comprising attributes of candidate compression algorithms, the comparing comprising identifying similarities between the time-series profile and the attributes of the candidate compression algorithms;
selecting a compression algorithm from among a plurality of compression algorithms based at least in part on a result of the comparing; and
causing subsequent time-series data received from the data source to be compressed using the selected compression algorithm.
9. The information handling system of claim 8 , wherein, responsive to the selected compression algorithm being different from the target compression algorithm, the causing comprises dynamically modifying the dynamically-variable assignment such that the selected compression algorithm is assigned to the data source.
10. The information handling system of claim 8 , wherein, responsive to the selected compression algorithm being the target compression algorithm, the causing comprises compressing the subsequent time-series data using the target compression algorithm.
11. The information handling system of claim 8 , the method comprising:
transmitting a request for algorithm experience data to a data processor, wherein the transmitting comprises the time-series profile;
responsive to the request, receiving algorithm performance data in relation to at least one compression algorithm;
comparing the received algorithm performance data to algorithm performance data for the target compression algorithm;
identifying based, at least in part, on a sampling and profiling of the time-series data that the at least one compression algorithm results in at least one of a higher compression ratio, a lower time to compress and a lower time to decompress; and
responsive to the identifying, causing subsequent time-series data received from the data source to be compressed using the at least one compression algorithm.
12. The information handling system of claim 8 , wherein the method is performed in parallel with respect to a plurality of data sources.
13. The information handling system of claim 8 , wherein:
the attributes of the time-series profile and the stored attributes of each of the candidate compression algorithms are each represented as a feature vector; and
the comparing attributes comprises determining a similarity between the feature vector of the time-series profile and the feature vector of each of the candidate compression algorithms.
14. The information handling system of claim 13 , wherein:
the attributes of the time-series profile comprise at least one of a burst frequency and a burst size.
15. A computer-program product comprising a non-transitory computer-usable medium having computer-readable program code embodied therein, the computer-readable program code adapted to be executed to implement a method comprising:
receiving, from a data source, time-series data of a time-series data stream produced by the data source;
identifying a target compression algorithm for the time-series data, wherein the target compression algorithm is linked to the data source in memory pursuant to a dynamically-variable assignment;
compressing the time-series data using the target compression algorithm;
transmitting the compressed time-series data to a destination;
periodically optimizing, at least as measured by compression ratio, the dynamically-variable assignment in real-time as the time-series data is received, the periodically optimizing comprising:
accessing a sample of the time-series data stream in relation to a sample period;
determining a time density of time-series data production by the data source over one or more intervals of the sample period, wherein the time density of time-series data production refers to a frequency with which the time-series data is produced for compression such that greater production over the one or more intervals constitutes a greater time density and less production over the one or more intervals constitutes a lower time density;
generating a time-series profile of the sample using the time density;
comparing attributes of the time-series profile to stored algorithm signatures comprising attributes of candidate compression algorithms, the comparing comprising identifying similarities between the time-series profile and the attributes of the candidate compression algorithms;
selecting a compression algorithm from among a plurality of compression algorithms based at least in part on a result of the comparing; and
causing subsequent time-series data received from the data source to be compressed using the selected compression algorithm.
16. The computer-program product of claim 15 , wherein, responsive to the selected compression algorithm being different from the target compression algorithm, the causing comprises dynamically modifying the dynamically-variable assignment such that the selected compression algorithm is assigned to the data source.
17. The computer-program product of claim 15 , wherein, responsive to the selected compression algorithm being the target compression algorithm, the causing comprises compressing the subsequent time-series data using the target compression algorithm.
18. The computer-program product of claim 15 , the method comprising:
transmitting a request for algorithm experience data to a data processor, wherein the transmitting comprises the time-series profile;
responsive to the request, receiving algorithm performance data in relation to at least one compression algorithm;
comparing the received algorithm performance data to algorithm performance data for the target compression algorithm;
identifying based, at least in part, on a sampling and profiling of the time-series data that the at least one compression algorithm results in at least one of a higher compression ratio, a lower time to compress and a lower time to decompress; and
responsive to the identifying, causing subsequent time-series data received from the data source to be compressed using the at least one compression algorithm.
19. The computer-program product of claim 15 , wherein the method is performed in parallel with respect to a plurality of data sources.
20. The computer-program product of claim 15 , wherein the attributes of the time-series profile and the stored attributes of each of the candidate compression algorithms are each represented as a feature vector.