System and method for compressing and restoring data using multi-level autoencoders and correlation networks
Compressing and restoring data using a multi-level autoencoder and a correlation network. The multi-level autoencoder compresses (in the encoder section) and decompresses (in the decoder section) data streams, and a separate correlation network trained on groups of input data sets restores lost data by leveraging correlations between the data sets.
1. A system for compressing and restoring data, comprising:
a computing device comprising at least a memory and a processor;
a plurality of programming instructions stored in the memory and operable on the processor that, when operating on the processor, cause the computing device to:
preprocess raw data to generate a plurality of input data sets;
compress the plurality of input data sets into a plurality of compressed data sets using an encoder within a multi-layer autoencoder;
decompress the plurality of compressed data sets using a decoder located within a multi-layer autoencoder to obtain a plurality of reduced output data sets; and
process the plurality of reduced output data sets through a correlation network to recover information lost in compression by leveraging correlations between the plurality of input data sets, thereby generating a plurality of restored output data sets.
2. The system of claim 1 , wherein the multi-level autoencoder comprises an encoder and a decoder, and the encoder comprises convolutional layers, pooling layers, and activation functions.
3. The system of claim 1 , wherein the correlation network comprises convolutional layers and activation functions.
4. The system of claim 1 , wherein the plurality of data sets comprises a plurality of IoT sensor data where the incoming IoT sensor data is organized by sensor type prior to preprocessing.
5. The system of claim 1 , wherein the plurality of data sets comprises hyperspectral data.
6. A method for compressing and restoring data, comprising the steps of:
preprocessing raw data to generate a plurality of input data sets;
compressing the plurality of input data sets into a plurality of compressed data sets using an encoder within a multi-layer autoencoder;
decompressing the plurality of compressed data sets using a decoder located within a multi-layer autoencoder to obtain a plurality of reduced output data sets; and
processing the plurality of reduced output data sets through a correlation network to recover information lost in compression by leveraging correlations between the plurality of input data sets, thereby generating a plurality of restored output data sets.
7. The method of claim 6 , wherein the multi-level autoencoder comprises an encoder-decoder architecture with convolutional layers, pooling layers, and activation functions.
8. The method of claim 6 , wherein the correlation network comprises an architecture with convolutional layers and activation functions.
9. The method of claim 6 , wherein the plurality of data sets comprises a plurality of IoT sensor data where the incoming IoT sensor data is organized by sensor type prior to preprocessing.
10. The method of claim 6 , wherein the plurality of data sets comprises hyperspectral data.