System and method for data compaction and security using multiple encoding algorithms
A system and method for encoding data using a plurality of encoding libraries. Portions of the data are encoded by different encoding libraries, depending on which library provides the greatest compaction for a given portion of the data. This methodology not only provides substantial improvements in data compaction over use of a single data compaction algorithm with the highest average compaction, but provides substantial additional security in that multiple decoding libraries must be used to decode the data. In some embodiments, each portion of data may further be encoded using different sourceblock sizes, providing further security enhancements as decoding requires multiple decoding libraries and knowledge of the sourceblock size used for each portion of the data. In some embodiments, encoding libraries may be randomly or pseudo-randomly rotated to provide additional security.
1. A system for encoding data using a plurality of codebooks comprising:
a computing device comprising a processor, a memory, and a non-volatile data storage device;
a codebook selector comprising a first plurality of programming instructions stored in the memory and operable on the processor, wherein the first plurality of programming instructions, when operating on the processor, causes the processor to:
receive a data set for encoding, the data set comprising a plurality of sourcepackets, each sourcepacket comprising a sourcepacket identifier;
select a list of codebooks for encoding of the data set, the list comprising a codebook identifier for each codebook;
for each sourcepacket in the data set:
encode the sourcepacket with each of the codebooks in the list of codebooks;
associate the sourcepacket identifier with a stored encoded sourcepacket;
associate the codebook identifier of the codebook which produced the stored encoded sourcepacket with the sourcepacket identifier; and
send the sourcepacket identifier and its associated codebook identifier to a combiner; and
the combiner comprising a second plurality of programming instructions stored in the memory and operable on the processor, wherein the second plurality of programming instructions, when operating on the processor, causes the processor to:
receive each of the sourcepacket identifiers and their associated codebook identifiers from the codebook selector; and
combine the sourcepacket identifiers and their associated codebook identifiers into a data structure comprising an ordered sequence comprising each sourcepacket identifier and its associated codebook identifier.
2. The system of claim 1 , wherein the codebooks used to encode the data are combined as a single codebook.
3. The system of claim 1 , wherein the data structure further comprises a copy of the stored encoded sourcepacket associated with each sourcepacket identifier.
4. The system of claim 1 , wherein the sourcepacket identifier is a universal unique identifier (UUID).
5. The system of claim 1 , wherein the codebook identifier is a universal unique identifier (UUID).
6. A method for encoding data using a plurality of codebooks, comprising the steps of:
receiving a data set for encoding, the data set comprising a plurality of sourcepackets, each sourcepacket comprising a sourcepacket identifier;
selecting a list of codebooks for encoding of the data set, the list comprising a codebook identifier for each codebook;
for each sourcepacket in the data set:
encoding the sourcepacket with each of the codebooks in the list of codebooks;
associating the sourcepacket identifier with thea stored encoded sourcepacket; and
associating the codebook identifier of the codebook which produced the stored encoded sourcepacket with the sourcepacket identifier; and
combining the sourcepacket identifiers and their associated codebook identifiers into a data structure comprising an ordered sequence comprising each sourcepacket identifier and its associated codebook identifier.
7. The method of claim 6 , further comprising the step of combining the codebooks used to encode the data into a single codebook.
8. The method of claim 6 , further comprising the step of including a copy of the stored encoded sourcepacket associated with each sourcepacket identifier in the data structure.
9. The method of claim 6 , wherein the sourcepacket identifier is a universal unique identifier (UUID).
10. The method of claim 6 , wherein the codebook identifier is a universal unique identifier (UUID).