Data compression with protocol adaptation
A system and method for data compression with protocol adaptation, that utilizes a codebook generator which leverages one or more machine/deep learning algorithms trained on at least a plurality of protocol policies in order to generate a protocol appendix and codebook, wherein original data is encoded by an encoder according to the codebook and sent to a decoder, but instead of just decoding the data according to the codebook to reconstruct the original data, data manipulation rules such as mapping and transformation are applied at the decoding stage to transform the decoded data into protocol formatted data.
1. A system for data compression with protocol adaptation, comprising:
a plurality of computing devices each comprising at least a processor, a memory, and a network interface;
wherein a plurality of programming instructions stored in one or more of the memories and operating on one or more of the processors of the plurality of computing devices causes the plurality of computing devices to:
receive a plurality of training data;
receive a plurality of protocol policy data;
use a subset of the training data and a subset of the protocol policy data as inputs to train a machine learning algorithm, wherein the machine learning algorithm is configured to produce a protocol appendix;
append the protocol appendix to a codebook;
receive encoded data; and
decode the encoded data using the appended codebook, wherein the decoded data is output as protocol formatted data.
2. The system of claim 1 , wherein the system is further configured to generate the codebook based on analysis of a second training data.
3. The system of claim 1 , wherein the machine learning algorithm is selected from the group consisting of decision trees, random forest, k-Nearest Neighbors and support vector machines.
4. The system of claim 1 , wherein the machine learning algorithm is a deep learning algorithm.
5. The system of claim 4 , wherein the deep learning algorithm is a neural network.
6. The system of claim 1 , wherein the plurality of protocol policy data comprises at least data format and structure, message protocol standards, data transmission and encryption, data validation and sanitation rules, error handling and reporting, data versioning, data ownership and access control, data documentation, compliance and regulations, and monitoring and auditing processes.
7. The system of claim 1 , wherein the protocol appendix is appended to the codebook in the form of bit extensions.
8. The system of claim 1 , wherein the protocol appendix is appended to the codebook in the form of a dimensional array.
9. A method for data compression with protocol adaptation, comprising the steps of:
receiving a plurality of training data;
receiving a plurality of protocol policy data;
using a subset of the training data and a subset of the protocol policy data as inputs to train a machine learning algorithm, wherein the machine learning algorithm is configured to produce a protocol appendix;
appending the protocol appendix to a codebook;
receiving encoded data; and
decoding the encoded data using the appended codebook, wherein the decoded data is output as protocol formatted data.
10. The method of claim 9 , further comprising the step of generating the codebook based on analysis of a second training data.
11. The method of claim 9 , wherein the machine learning algorithm is selected from the group consisting of decision trees, random forest, k-Nearest Neighbors and support vector machines.
12. The method of claim 9 , wherein the machine learning algorithm is a deep learning algorithm.
13. The method of claim 12 , wherein the deep learning algorithm is a neural network.
14. The method of claim 9 , wherein the plurality of protocol policy data comprises at least data format and structure, message protocol standards, data transmission and encryption, data validation and sanitation rules, error handling and reporting, data versioning, data ownership and access control, data documentation, compliance and regulations, and monitoring and auditing processes.
15. The method of claim 9 , wherein the protocol appendix is appended to the codebook in the form of bit extensions.
16. The method of claim 9 , wherein the protocol appendix is appended to the codebook in the form of a dimensional array.
17. A computer-readable, non-transitory medium comprising a plurality of programming instructions that, when operating on a plurality of computing devices each comprising at least a processor, a memory, and a network interface, cause the plurality of computing devices to carry out the method of claim 9 .