System and method for federated two-stage compression with federated joint learning
A system and method for federated two-stage compression with federated joint learning. The system and method proposed allow for fast and efficient lossless data compression of a large variety of data types. The system and method have a variety of real-world applications, including deep learning solutions for telemetry, tracking, and command subsystems for satellites. Satellites and their control centers are incredibly spaced apart which makes data compression an extremely important process to transmit large sets of information in a low-latency, high-efficiency environment. The proposed system and method utilize probability prediction driven arithmetic coding which provides faster encoding times and higher compression ratios when paired with a long short-term memory system for data compression.
1. A system for federated two-stage compression with federated joint learning, comprising one or more computers with executable instructions that, when executed, cause the system to:
process input data through a Variational Autoencoder with Vector Quantization on an edge server to generate a plurality of compressed data;
transmit the plurality of compressed data to a midserver, wherein the midserver converts the plurality of compressed data into a plurality of codewords using a codebook;
transmit the plurality of codewords to a centralized server where the plurality of codewords are converted to a plurality of universal codewords using a universal codebook;
train a large codeword model core using the plurality of universal codewords; and
deploy a trained large codeword model core wherein the large codeword model core receives a plurality of input data and generates a plurality of compressed outputs.
2. The system of claim 1 , wherein the large codeword model core functions using a Transformer based architecture.
3. The system of claim 1 , wherein the Variational Autoencoder with Vector Quantization and the large codeword model core are jointly trained.
4. A method for federated two-stage compression with federated joint learning, comprising the steps of:
processing input data through a Variational Autoencoder with Vector Quantization on an edge server to generate a plurality of compressed data;
transmitting the plurality of compressed data to a midserver, wherein the midserver converts the plurality of compressed data into a plurality of codewords using a codebook;
transmitting the plurality of codewords to a centralized server where the plurality of codewords are converted to a plurality of universal codewords using a universal codebook;
training a large codeword model core using the plurality of universal codewords; and
deploying a trained large codeword model core wherein the large codeword model core receives a plurality of input data and generates a plurality of compressed outputs.
5. The method of claim 4 , wherein the large codeword model core functions using a Transformer based architecture.
6. The system of claim 4 , wherein the Variational Autoencoder with Vector Quantization and the large codeword model core are jointly trained.
7. A non-transitory, computer-readable storage media having computer-executable instructions embodied thereon that, when executed by one or more processors of a computing system employing an asset registry platform for secure, robust, and efficient blockchain management system using large codeword models, cause the computing system to:
process input data through a Variational Autoencoder with Vector Quantization on an edge server to generate a plurality of compressed data;
transmit the plurality of compressed data to a midserver, wherein the midserver converts the plurality of compressed data into a plurality of codewords using a codebook;
transmit the plurality of codewords to a centralized server where the plurality of codewords are converted to a plurality of universal codewords using a universal codebook;
train a large codeword model core using the plurality of universal codewords; and
deploy a trained large codeword model core wherein the large codeword model core receives a plurality of input data and generates a plurality of compressed outputs.
8. The media of claim 7 , wherein the large codeword model core functions using a Transformer based architecture.
9. The media of claim 7 , wherein the Variational Autoencoder with Vector Quantization and the large codeword model core are jointly trained.