System and method for data compaction and security using multiple encoding algorithms
A system and method for encoding data using a plurality of encoding libraries. Portions of the data are encoded by different encoding libraries, depending on which library provides the greatest compaction for a given portion of the data. This methodology not only provides substantial improvements in data compaction over use of a single data compaction algorithm with the highest average compaction, but provides substantial additional security in that multiple decoding libraries must be used to decode the data. In some embodiments, each portion of data may further be encoded using different sourceblock sizes, providing further security enhancements as decoding requires multiple decoding libraries and knowledge of the sourceblock size used for each portion of the data. In some embodiments, encoding libraries may be randomly or pseudo-randomly rotated to provide additional security.
1. A system for encoding data using a plurality of codebooks comprising:
a computing device comprising a processor, a memory, and a non-volatile data storage device;
a codebook selector comprising a first plurality of programming instructions stored in the memory and operable on the processor, wherein the first plurality of programming instructions, when operating on the processor, causes the processor to:
select a list of codebooks for encoding a plurality of sourcepackets, each sourcepacket comprising a sourcepacket identifier, the list comprising a codebook identifier for each codebook;
for each sourcepacket:
encode the sourcepacket with each of the codebooks in the list of codebooks;
associate the sourcepacket identifier with a stored encoded sourcepacket;
associate the codebook identifier of the codebook which produced the stored encoded sourcepacket with the sourcepacket identifier; and
send a data pair comprising the respective sourcepacket identifier and its associated codebook identifier to a combiner; and
the combiner comprising a second plurality of programming instructions stored in the memory and operable on the processor, wherein the second plurality of programming instructions, when operating on the processor, causes the processor to:
receive each data pair from the codebook selector; and
aggregate each received data pair into a data structure comprising an ordered sequence of pairs each comprising a sourcepacket identifier and its associated codebook identifier.
2. The system of claim 1 , wherein the codebooks used to encode the data are combined as a single codebook.
3. The system of claim 1 , wherein the data structure further comprises a copy of the stored encoded sourcepacket associated with each sourcepacket identifier.
4. The system of claim 1 , wherein the sourcepacket identifier is a universal unique identifier (UUID).
5. The system of claim 1 , wherein the codebook identifier is a universal unique identifier (UUID).
6. A method for encoding data using a plurality of codebooks, comprising the steps of:
selecting a list of codebooks for encoding a plurality of sourcepackets, each sourcepacket comprising a sourcepacket identifier, the list comprising a codebook identifier for each codebook;
for each sourcepacket:
encoding the sourcepacket with each of the codebooks in the list of codebooks;
associating the sourcepacket identifier with the a stored encoded sourcepacket;
associating the codebook identifier of the codebook which produced the stored encoded sourcepacket with the sourcepacket identifier; and
sending a data pair comprising the respective sourcepacket identifier and its associated codebook identifier to a combiner;
using the combiner:
receiving the data pairs at the combiner; and
aggregating each received data pair into a data structure comprising an ordered sequence of pairs each comprising a sourcepacket identifier and its associated codebook identifier.
7. The method of claim 6 , further comprising the step of combining the codebooks used to encode the data into a single codebook.
8. The method of claim 6 , further comprising the step of including a copy of the stored encoded sourcepacket associated with each sourcepacket identifier in the data structure.
9. The method of claim 6 , wherein the sourcepacket identifier is a universal unique identifier (UUID).
10. The method of claim 6 , wherein the codebook identifier is a universal unique identifier (UUID).