System and method for federated two-stage compression with federated joint learning
A system and method for federated two-stage compression with federated joint learning. The system and method proposed allow for fast and efficient lossless data compression of a large variety of data types. The system and method have a variety of real-world applications, including deep learning solutions for telemetry, tracking, and command subsystems for satellites. Satellites and their control centers are incredibly spaced apart which makes data compression an extremely important process to transmit large sets of information in a low-latency, high-efficiency environment. The proposed system and method utilize probability prediction driven arithmetic coding which provides faster encoding times and higher compression ratios when paired with a long short-term memory system for data compression.
1. A system for federated two-stage compression with federated joint learning, comprising one or more computers with executable instructions that, when executed, cause the system to:
process input data through a compression network on an edge server to generate a plurality of compressed data;
transmit the plurality of compressed data to a midserver, wherein the midserver converts the plurality of compressed data into a plurality of codewords using a codebook;
train and deploy a lightweight codeword model on the edge server wherein the lightweight codeword model is trained using the plurality of codewords; and
compress the input data through the lightweight codeword model on the edge server to generate a second plurality of compressed data.
2. The system of claim 1 , wherein the plurality of codewords are transmitted to a cloud that converts the plurality of codewords into a plurality of codewords using a universal codebook.
3. The system of claim 2 , wherein the plurality of universal codewords are used to train a large codeword model.
4. The system of claim 2 , wherein updated models of the lightweight codeword model or the compression network may be stored in and transferred between the edge server, the midserver, and the cloud.
5. The system of claim 1 , wherein a plurality of codebooks may be stored on the midserver and used to generate codewords for a plurality of edge servers, wherein each edge server has its own specific set of codewords.
6. A method for federated two-stage compression with federated joint learning, comprising the steps of:
processing input data through a compression network on an edge server to generate a plurality of compressed data;
transmitting the plurality of compressed data to a midserver, wherein the midserver converts the plurality of compressed data into a plurality of codewords using a codebook;
training and deploy a lightweight codeword model on the edge server wherein the lightweight codeword model is trained using the plurality of codewords; and
compressing the input data through the lightweight codeword model on the edge server to generate a second plurality of compressed data.
7. The method of claim 6 , wherein the plurality of codewords are transmitted to a cloud that converts the plurality of codewords into a plurality of codewords using a universal codebook.
8. The method of claim 7 , wherein the plurality of universal codewords are used to train a large codeword model.
9. The method of claim 7 , wherein updated models of the lightweight codeword model or the compression network may be stored in and transferred between the edge server, the midserver, and the cloud.
10. The method of claim 6 , wherein a plurality of codebooks may be stored on the midserver and used to generate codewords for a plurality of edge servers, wherein each edge server has its own specific set of codewords.
11. A non-transitory, computer-readable storage media having computer-executable instructions embodied thereon that, when executed by one or more processors of a computing system employing an asset registry platform for secure, robust, and efficient blockchain management system using large codeword models, cause the computing system to:
process input data through a compression network on an edge server to generate a plurality of compressed data;
transmit the plurality of compressed data to a midserver, wherein the midserver converts the plurality of compressed data into a plurality of codewords using a codebook;
train and deploy a lightweight codeword model on the edge server wherein the lightweight codeword model is trained using the plurality of codewords; and
compress the input data through the lightweight codeword model on the edge server to generate a second plurality of compressed data.
12. The media of claim 11 , wherein the plurality of codewords are transmitted to a cloud that converts the plurality of codewords into a plurality of codewords using a universal codebook.
13. The media of claim 12 , wherein the plurality of universal codewords are used to train a large codeword model.
14. The media of claim 12 , wherein updated models of the lightweight codeword model or the compression network may be stored in and transferred between the edge server, the midserver, and the cloud.
15. The media of claim 11 , wherein a plurality of codebooks may be stored on the midserver and used to generate codewords for a plurality of edge servers, wherein each edge server has its own specific set of codewords.