System and method for data compaction and encryption of anonymized datasets
A system and method for encoding anonymized dataset. A dataset may be pre-processed by dividing into a plurality of sourceblocks at all reasonable sourceblock lengths, and then counting how many times each sourceblock occurs in the dataset, resulting in a tally record of tokens and their count value. This tally record may then be anonymized and transmitted as an anonymized tally record to a data deconstruction engine which combined with a library manager creates a codebook and performs optimization techniques on the codebook. The received anonymized tally record may be parsed into individual tokens by identifying the tokens with the highest count value. The tokens may then be sent, in descending order of count value, to the library manger where each token may be assigned a codeword. Then a half-backed codebook is created using the tokens and each token's unique codeword, before sending the half-backed codebook to a system user.
1. A system for encoding anonymized data comprising:
a computing device comprising a processor, a memory, and a non-volatile data storage device;
a data deconstruction engine comprising a first plurality of programming instructions stored in the memory and operable on the processor, wherein the first plurality of programming instructions, when operating on the processor, causes the processor to:
receive an anonymized tally record from a user, the anonymized tally record comprising a plurality of anonymized sourceblocks and for each anonymized sourceblock a tally value indicating the number of times the anonymized sourceblock occurs in a data source, wherein each of the plurality of anonymized sourceblocks is associated with a fixed sourceblock length, and wherein multiple sourceblock lengths may be included in the anonymized tally record;
select a sourceblock length;
for each sourceblock length:
parse the anonymized sourceblocks to identify, in descending order, the anonymized sourceblocks with the highest tally value;
send the anonymized sourceblocks, in descending order, to a library manager;
receive a half-backed codebook from the library manager; and
transmit the half-backed codebook to the user; and
a library manager comprising a second plurality of programming instructions stored in the memory and operable on the processor, wherein the second plurality of programming instructions, when operating on the processor, causes the processor to:
receive anonymized sourceblocks, in descending order, from a data deconstruction engine;
for each sourceblock length:
assign a unique codeword to each of the received anonymized sourceblocks in the same order as they are received;
determine the sourceblock length that resulted in the best compaction;
create a half-backed codebook, the half-backed codebook comprising a plurality of codeword pairs, wherein each codeword pair comprises an anonymized sourceblock and its associated unique codeword; and
send the half-backed codebook to the data deconstruction engine.
2. The system of claim 1 , wherein the data deconstruction engine receives two or more anonymized tally records.
3. The system of claim 2 , wherein the two or more anonymized tally records are used to create a combined half-backed codebook, the combined half-backed codebook comprising codeword pairs from each of the anonymized tally records, selected from the sourceblock length of each anonymized tally record that resulted in the best compaction.
4. The system of claim 3 , wherein a stencil is created and transmitted to the user with the combined half-backed codebook to assist the user to extract codeword pairs from the combined half-backed codebook.
5. The system of claim 2 , wherein a hybrid stencil is created to synthesize multiple half-backed codebooks into a hybrid synthesized codebook.
6. A method for encoding anonymized data comprising:
receiving an anonymized tally record from a user, the anonymized tally record comprising a plurality of anonymized sourceblocks and for each anonymized sourceblock a tally value indicating the number of times the anonymized sourceblock occurs in a data source, wherein each of the plurality of anonymized sourceblocks is associated with a fixed sourceblock length, and wherein multiple sourceblock lengths may be included in the anonymized tally record;
selecting a sourceblock length;
for each sourceblock length:
parsing the anonymized sourceblocks to identify, in descending order, the anonymized sourceblocks with the highest tally value;
sending the anonymized sourceblocks, in descending order, to a library manager;
receiving a half-backed codebook from the library manager;
transmitting the half-backed codebook to the user;
receiving anonymized sourceblocks, in descending order, from a data deconstruction engine;
for each sourceblock length:
assigning a unique codeword to each of the received anonymized sourceblocks in the same order as they are received;
determining the sourceblock length that resulted in the best compaction;
creating a half-backed codebook, the half-backed codebook comprising a plurality of codeword pairs, wherein each codeword pair comprises an anonymized sourceblock and its associated unique codeword; and
sending the half-backed codebook to the data deconstruction engine.
7. The method of claim 6 , wherein the data deconstruction engine receives two or more anonymized tally records.
8. The method of claim 7 , wherein the two or more anonymized tally records are used to create a combined half-backed codebook, the combined half-backed codebook comprising codeword pairs from each of the anonymized tally records, selected from the sourceblock length of each anonymized tally record that resulted in the best compaction.
9. The method of claim 8 , wherein a stencil is created and transmitted to the user with the combined half-backed codebook to assist the user to extract codeword pairs from the combined half-backed codebook.
10. The method of claim 7 , wherein a hybrid stencil is created to synthesize multiple half-backed codebooks into a hybrid synthesized codebook.