System and method for data compaction and security with extended functionality
A system and method for highly efficient encoding of data that includes extended functionality for asymmetric encoding/decoding and network policy enforcement. In the case of asymmetric encoding/decoding the original data is encoded by an encoder according to a codebook and sent to a decoder, but the output of the decoder depends on data manipulation rules applied at the decoding stage to transform the decoded data into a different data set from the original data. In the case of network policy enforcement, a behavior appendix into the codebook, such that the encoder and/or decoder at each node of the network comply with network behavioral rules, limits, and policies during encoding and decoding.
1. A system for data conversion during decoding of data, comprising:
a computing device comprising a processor and a memory;
a codebook generator comprising a first plurality of programming instructions stored in the memory which, when operating on the processor, causes the computing device to:
receive training data for generation of a codebook;
generate a codebook from the training data, the codebook comprising a plurality of sourceblocks; and
transmit the codebook to a decoder; and
the decoder comprising a second plurality of programming instructions stored in the memory which, when operating on the processor, causes the computing device to:
receive encoded data which has been encoded into codewords based on the codebook, each codeword referring to one of the sourceblocks in the codebook;
decode the encoded data using the codebook by identifying the sourceblock in the codebook to which each codeword of the encoded data refers;
apply a rule to each sourceblock of the decoded data with which the rule is associated in the codebook to convert the sourceblock according to the rule; and
output the decoded data as a sequence of its converted and unconverted sourceblocks.
2. The system of claim 1 , wherein the rule is a mapping or transform rule, and the output of the decoded data is a mapping or transformation of the encoded data.
3. The system of claim 1 , wherein the rule is a network behavior rule, and the network behavior rule determines whether the sourceblock is decoded or not.
4. The system of claim 1 , wherein the rule is a protocol formatting rule, and the output of the decoded data is protocol formatted data.
5. A method for data conversion during decoding of data, comprising the steps of:
receiving, at a computing device, training data for generation of a codebook;
generating, using a codebook generator operating on the computing device, a codebook from the training data, the codebook comprising a plurality of sourceblocks;
transmitting the codebook to a decoder operating on the same or a different computing device;
receiving, at the decoder, encoded data which has been encoded into codeworks based on the codebook, each codeword referring to one of the sourceblocks in the codebook;
decoding the encoded data using the codebook by identifying the sourceblock in the codebook to which each codeword of the encoded data refers;
applying a rule to each sourceblock of the decoded data with which the rule is associated in the codebook to convert the sourceblock according to the rule; and
outputting the decoded data as a sequence of its converted and unconverted sourceblocks.
6. The method of claim 5 , wherein the rule is a mapping or transform rule, and the output of the decoded data is a mapping or transformation of the encoded data.
7. The method of claim 5 , wherein the rule is a network behavior rule, and the network behavior rule determines whether the sourceblock is decoded or not.
8. The method of claim 5 , wherein the rule is a protocol formatting rule, and the output of the decoded data is protocol formatted data.
9. A system for data conversion during decoding of data, comprising:
a first computing device comprising a first processor and a first memory;
a second computing device comprising a second processor and a second memory;
a codebook generator comprising a first plurality of programming instructions stored in the first memory which, when operating on the first processor, causes the first computing device to:
receive training data for generation of a codebook;
generate a codebook from the training data, the codebook comprising a plurality of sourceblocks;
transmit the codebook to a decoder operating on the second computing device; and
the decoder operating comprising a second plurality of programming instructions stored in the second memory which, when operating on the second processor, causes the second computing device to:
receive encoded data which has been encoded into codeworks based on the codebook, each codeword referring to one of the sourceblocks in the codebook;
decode the encoded data using the codebook by identifying the sourceblock in the codebook to which each codeword of the encoded data refers;
apply a rule to each sourceblock of the decoded data with which the rule is associated in the codebook to convert the sourceblock according to the rule; and
output the decoded data as a sequence of its converted and unconverted sourceblocks.
10. The system of claim 9 , wherein the rule is a mapping or transform rule, and the output of the decoded data is a mapping or transformation of the encoded data.
11. The system of claim 9 , wherein the rule is a network behavior rule, and the network behavior rule determines whether the sourceblock is decoded or not.
12. The system of claim 9 , wherein the rule is a protocol formatting rule, and the output of the decoded data is protocol formatted data.