System and method for data compression with protocol adaptation
A system and method for data compression with protocol adaptation, that utilizes a codebook generator which leverages one or more machine/deep learning algorithms trained on at least a plurality of protocol policies in order to generate a protocol appendix and codebook, wherein original data is encoded by an encoder according to the codebook and sent to a decoder, but instead of just decoding the data according to the codebook to reconstruct the original data, data manipulation rules such as mapping and transformation are applied at the decoding stage to transform the decoded data into protocol formatted data.
1. A system for data compression with protocol adaptation, comprising:
a computing device comprising a processor and a memory;
a codebook generator comprising a first plurality of programming instructions stored in the memory which, when operating on the processor, causes the computing device to:
receive a plurality of training data;
receive a plurality of protocol policy data;
use a subset of the training data and a subset of the protocol policy data as inputs to train a machine learning algorithm, wherein the machine learning algorithm is configured to produce a protocol appendix;
append the protocol appendix to a codebook; and
send the appended codebook to a decoder; and
the decoder comprising a second plurality of programming instructions stored in the memory which, when operating on the processor, causes the computing device to:
receive the appended codebook from the codebook generator;
receive encoded data; and
decode the encoded data using the appended codebook, wherein the decoded data is output as protocol formatted data.
2. The system of claim 1 , wherein the codebook generator is further configured to generate the codebook based on analysis of a second training data.
3. The system of claim 1 , wherein the machine learning algorithm is selected from the group consisting of decision trees, random forest, k-Nearest Neighbors and support vector machines.
4. The system of claim 1 , wherein the machine learning algorithm is a deep learning algorithm.
5. The system of claim 4 , wherein the deep learning algorithm is a neural network.
6. The system of claim 1 , wherein the plurality of protocol policy data comprises at least data format and structure, message protocol standards, data transmission and encryption, data validation and sanitation rules, error handling and reporting, data versioning, data ownership and access control, data documentation, compliance and regulations, and monitoring and auditing processes.
7. The system of claim 1 , wherein the protocol appendix is appended to the codebook in the form of bit extensions.
8. The system of claim 1 , wherein the protocol appendix is appended to the codebook in the form of a dimensional array.
9. A method for data compression with protocol adaptation, comprising the steps of:
receiving a plurality of training data;
receiving a plurality of protocol policy data;
using a subset of the training data and a subset of the protocol policy data as inputs to train a machine learning algorithm, wherein the machine learning algorithm is configured to produce a protocol appendix;
appending the protocol appendix to a codebook;
sending the appended codebook to a decoder;
receiving the appended codebook from a codebook generator;
receiving encoded data; and
decoding the encoded data using the appended codebook, wherein the decoded data is output as protocol formatted data.
10. The method of claim 9 , further comprising the step of generating the codebook based on analysis of a second training data.
11. The method of claim 9 , wherein the machine learning algorithm is selected from the group consisting of decision trees, random forest, k-Nearest Neighbors and support vector machines.
12. The method of claim 9 , wherein the machine learning algorithm is a deep learning algorithm.
13. The method of claim 12 , wherein the deep learning algorithm is a neural network.
14. The method of claim 9 , wherein the plurality of protocol policy data comprises at least data format and structure, message protocol standards, data transmission and encryption, data validation and sanitation rules, error handling and reporting, data versioning, data ownership and access control, data documentation, compliance and regulations, and monitoring and auditing processes.
15. The method of claim 9 , wherein the protocol appendix is appended to the codebook in the form of bit extensions.
16. The method of claim 9 , wherein the protocol appendix is appended to the codebook in the form of a dimensional array.