Compression and decompression of telemetry data for prediction models
An autoregressor that compresses input data for a specific purpose. Input data is compressed using a compression/decompression framework and by accounting for a purpose of a prediction model. The compression aspect of the framework is distributed and the decompression aspect of the framework may be centralized. The compression/decompression framework and a machine learning prediction model can be centrally trained. The compressor is distributed to nodes such that the input data can be compressed and transmitted to a central node. The model and the compression/decompression framework are continually trained on new data. This allows for lossy compression and higher compression rates while maintaining low prediction error rates.
1. A method, comprising:
receiving input data at a compressor from a plurality of data sources, wherein the compressor is configured to compress the data for a purpose associated with a predictor;
generating compressed data by the compressor;
transmitting the compressed data to a central node, wherein the compressed data is received by both a decompressor and the predictor; and
retraining the compressor and the predictor based on the compressed data.
2. The method of claim 1 , further comprising decompressing the compressed data with a decompressor.
3. The method of claim 2 , further comprising retraining the decompressor.
4. The method of claim 1 , further comprising generating an error signal based on an error rate associated with the predictor and based on a loss associated with compressing the input data, wherein compressing the input data is lossy.
5. The method of claim 1 , further comprising adjusting a compression ratio of the compressor.
6. The method of claim 1 , further comprising distributing an updated compressor to each of the plurality of data sources.
7. The method of claim 1 , further comprising retraining the predictor and/or the compressor when sufficient compressed data is received from the plurality of data sources or when a prediction error rate exceeds a threshold.
8. The method of claim 1 , wherein the compressor accounts for the purpose when compressing the input data.
9. The method of claim 1 , further comprising training using the compressed data or decompressed data.
10. The method of claim 1 , wherein a compression ratio is greater than a pre-established value.
11. A non-transitory computer readable medium having stored therein instructions that are executable by one or more hardware processors to perform operations comprising:
receiving input data at a compressor from a plurality of data sources, wherein the compressor is configured to compress the data for a purpose associated with a predictor;
generating compressed data by the compressor;
transmitting the compressed data to a central node, wherein the compressed data is received by both a decompressor and the predictor; and
retraining the compressor and the predictor based on the compressed data.
12. The non-transitory computer readable medium of claim 11 , the operations further comprising decompressing the compressed data with a decompressor.
13. The non-transitory storage medium of claim 12 , the operations further comprising retraining the decompressor.
14. The non-transitory computer readable medium of claim 11 , the operations further comprising generating an error signal based on an error rate associated with the predictor and based on a loss associated with compressing the input data, wherein compressing the data is lossy.
15. The non-transitory computer readable medium of claim 11 , the operations further comprising adjusting a compression ratio of the compressor.
16. The non-transitory computer readable medium of claim 11 , the operations further comprising distributing an updated compressor to each of the plurality of data sources.
17. The non-transitory computer readable medium of claim 11 , the operations further comprising retraining the predictor and/or the compressor when sufficient compressed data is received from the plurality of data sources or when a prediction error rate exceeds a threshold.
18. The non-transitory computer readable medium of claim 11 , the operations wherein the compressor accounts for the purpose when compressing the input data.
19. The non-transitory computer readable medium of claim 11 , the operations further comprising training using the compressed data or decompressed data.
20. The non-transitory computer readable medium of claim 11 , wherein a compression ratio is greater than a pre-established value.