System and method for data compaction utilizing distributed codebook encoding
A system and method for data compaction utilizing distributed codebook encoding to improve entropy encoding methods to account for, and efficiently handle, previously-unseen data in data to be compacted, allow for distributed encoding and decoding capabilities, and allow for parametrized codebook encoding methods. Training data sets are analyzed to determine the frequency of occurrence of each sourceblock in the training data sets. A mismatch probability estimate is calculated comprising an estimated frequency at which any given data sourceblock received during encoding will not have a codeword in the codebook. Further, a codebook and a behavior codebook may both be maintained or altered in a distributed fashion across multiple devices or services, for widespread, or permission-based, or parametrized codebook encoding.
1. A system for data compaction utilizing distributed codebook encoding, comprising:
a computing device comprising a processor, a memory, and a non-volatile data storage device;
a library engine comprising a first plurality of programming instructions stored in the memory which, when operating on the processor, causes the computing device to:
receive digital data to be compacted using a codebook, from a source computing device;
add new sourceblocks to a codebook based on received non-training data to be compacted;
create a behavior codebook from a set of rules, limitations, policies, or other configuration or behavioral settings for a library engine;
wherein the behavior codebook alters or determines the specific operations of the data compaction using a codebook;
encode and decode data using the codebook and behavior codebook;
return the newly encoded or decoded data to the source computing device; and
allow the distribution of the codebook to other devices for decoding of data;
wherein the codebook may be used to decode data, but may not be used to encode data properly without the behavior codebook;
wherein the behavior codebook comprises:
behavioral rules that control and prioritize which portions of the source data are encoded using specific codewords from the codebook;
configuration parameters that set limits on sourceblock sizes and types that can be compacted; and
encoding policies that determine specific operations of the data compaction process;
wherein the behavior codebook is maintained separately from but linked to the codebook, such that the behavior codebook controls how the codebook is utilized during encoding while the codebook maintains the actual codewords for encoding and decoding.
2. The system of claim 1 , wherein the source computing device is the same computing device that operates the library engine.
3. The system of claim 1 , wherein the source computing device is a mobile device, smartphone, tablet, laptop computer, desktop computer, server, or Internet-Of-Things device.
4. The system of claim 1 , wherein the library engine uses a machine learning engine to refine or optimize the codebook used for data compaction.
5. The system of claim 1 , wherein the library engine is a server that communicates with the source computing device over a network.
6. A method for data compaction utilizing distributed codebook encoding, comprising the steps of:
receiving digital data to be compacted using a codebook, from a source computing device, using a library engine;
adding new sourceblocks to a codebook based on received non-training data to be compacted, using a library engine;
creating a behavior codebook from a set of rules, limitations, policies, or other configuration or behavioral settings for a library engine, using a library engine;
wherein the behavior codebook alters or determines the specific operations of the data compaction using a codebook, using a library engine;
encoding and decode data using the codebook and behavior codebook, using a library engine;
returning the newly encoded or decoded data to the source computing device, using a library engine;
allowing the distribution of the codebook to other devices for decoding of data, using a library engine; and
wherein the codebook may be used to decode data, but may not be used to encode data properly without the behavior codebook, using a library engine;
wherein the behavior codebook comprises:
behavioral rules that control and prioritize which portions of the source data are encoded using specific codewords from the codebook;
configuration parameters that set limits on sourceblock sizes and types that can be compacted; and
encoding policies that determine specific operations of the data compaction process;
wherein the behavior codebook is maintained separately from but linked to the codebook, such that the behavior codebook controls how the codebook is utilized during encoding while the codebook maintains the actual codewords for encoding and decoding.
7. The method of claim 6 , wherein the source computing device is the same computing device that operates the library engine.
8. The method of claim 6 , wherein the source computing device is a mobile device, smartphone, tablet, laptop computer, desktop computer, server, or Internet-Of-Things device.
9. The method of claim 6 , wherein the library engine uses a machine learning engine to refine or optimize the codebook used for data compaction.
10. The method of claim 6 , wherein the library engine is a server that communicates with the source computing device over a network.