System and methods for secure storage for data deduplication
A system and methods for secure storage for data deduplication comprising a data deconstruction engine, a data reconstruction engine, a library manager, a reference codebook, and a codeword storage which performs simultaneous compaction and deduplication of data sets. A data set may be comprised of one or more sourcepackets which may be optimally deconstructed into a plurality of sourceblocks and wherein each sourceblock may be compared against a reference codebook that contains key-value pairs of a sourceblock and its associated reference code in order to determine if a received sourceblock is a duplicate of data already stored within the reference codebook. Non-duplicate sourceblocks can have a reference code algorithmically created and stored in the reference codebook, thereby ensuring that when a duplicate sourceblock is received, it will not be stored as duplicated data.
1. A system for secure storage for data deduplication of compacted data, comprising:
at least one reference codebook comprising key-value pairs of data;
a data deconstruction engine comprising at least a processor, a memory, and a plurality of programming instructions stored in the memory and operable on the processor of a computing device, wherein the plurality of programming instructions, when operating on the processor, cause the processor to:
receive a sourcepacket from a data source, the sourcepacket comprising a plurality of data to be stored and encoded;
deconstruct the incoming data into a plurality of deconstructed sourceblocks;
send the plurality of deconstructed sourceblocks to the library manager for comparison with sourceblocks already contained in the reference codebook; and
receive a reference code for each of the deconstructed sourceblocks from the library manager; and
a library manager comprising at least a processor, a memory, and a plurality of programming instructions stored in the memory and operable on the processor of a computing device, wherein the first plurality of programming instructions, when operating on the processor, cause the processor to:
receive the plurality of deconstructed sourceblocks from the data deconstruction engine;
perform data deduplication by comparing each of the deconstructed sourceblocks with sourceblocks already contained in the reference codebook;
for each of the plurality of deconstructed sourceblocks, return the reference code to the data deconstruction engine, when the respective received deconstructed sourceblock is a duplicate of an existing sourceblock in the reference codebook; and
for each received deconstructed sourceblock that is not present in the codebook:
create a new, unique reference code for the respective deconstructed sourceblock
store both the respective deconstructed sourceblock and the associated reference code in the reference codebook as a key-value pair; and
return the new reference code to the data deconstruction engine.
2. The system of claim 1 , wherein the data deconstruction engine is further configured to:
create a multiplicity of codeword pairs for storage or transmission of the data, each of which contains at least a reference code to a sourceblock in the library, and may contain additional information about the location of the reference code within the data; and
store the codeword pairs on a data storage device.
3. A method for secure storage for data deduplication of compacted data, comprising the steps of:
receiving a sourcepacket from a data source, the sourcepacket comprising a plurality of data to be stored and encoded;
deconstructing the incoming data into a plurality of deconstructed sourceblocks;
sending the plurality of deconstructed sourceblocks to the library manager for comparison with sourceblocks already contained in the reference codebook;
receiving a reference code for each of the deconstructed sourceblocks from a library manager;
receiving the plurality of deconstructed sourceblocks from a data deconstruction engine;
performing data deduplication by comparing each of the deconstructed sourceblocks with sourceblocks already contained in the reference codebook;
for each received deconstructed sourceblock, returning the reference code to the data deconstruction engine, when the respective received deconstructed sourceblock is a duplicate of an existing sourceblock in the reference codebook;
for each received deconstructed sourceblock that is not present in the codebook:
creating a new, unique reference code for the respective deconstructed sourceblock;
storing both the respective deconstructed sourceblock and the associated reference code in the reference codebook as a key-value pair; and
returning the new reference code to the data deconstruction engine.
4. The method of claim 3 , further comprising the steps of:
creating a multiplicity of codeword pairs for storage or transmission of the data, each of which contains at least a reference code to a sourceblock in the library, and may contain additional information about the location of the reference code within the data; and
storing the codeword pairs on a data storage device.