System and method for data compaction and encryption of anonymized data records
A system and method for data compaction and encryption of anonymized data records. A dataset may be pre-processed by dividing into a plurality of sourceblocks at all reasonable sourceblock lengths, and then counting how many times each sourceblock occurs in the dataset, resulting in a tally record of tokens and their count value. This tally record may then be anonymized and transmitted to a data deconstruction engine which combined with a library manager creates a codebook and performs optimization techniques on the codebook. The received anonymized tally record may be parsed into individual tokens by identifying the tokens with the highest count value. The tokens may then be sent, in descending order of count value, to the library manger where each token may be assigned a codeword. A half-backed codebook is then created using the tokens and each token's unique codeword, before sending the half-backed codebook to a system user.
1. A system for data compaction and encryption of anonymized data records, comprising:
a computing device comprising a processor, a memory, and a non-volatile data storage device;
a data deconstruction engine comprising a first plurality of programming instructions stored in the memory and operable on the processor, wherein the first plurality of programming instructions, when operating on the processor, causes the processor to:
receive an anonymized tally record from a user, the anonymized tally record comprising a plurality of anonymized sourceblocks and for each anonymized sourceblock a tally value indicating the number of times the anonymized sourceblock occurs in a data source;
for each sourceblock:
parse the anonymized sourceblocks to identify the anonymized sourceblocks with the highest tally value;
send the anonymized sourceblocks to a library manager;
receive a half-backed codebook from the library manager; and
transmit the half-backed codebook to the user; and
a library manager comprising a second plurality of programming instructions stored in the memory and operable on the processor, wherein the second plurality of programming instructions, when operating on the processor, causes the processor to:
receive anonymized sourceblocks from a data deconstruction engine;
for each sourceblock:
assign a unique codeword to each of the received anonymized;
create a half-backed codebook, the half-backed codebook comprising a plurality of codeword pairs, wherein each codeword pair comprises an anonymized sourceblock and its associated unique codeword; and
send the half-backed codebook to the data deconstruction engine.
2. The system of claim 1 , wherein the data deconstruction engine receives two or more anonymized tally records.
3. The system of claim 2 , wherein the two or more anonymized tally records are used to create a combined half-backed codebook, the combined half-backed codebook comprising codeword pairs from each of the anonymized tally records, selected from the sourceblock of each anonymized tally record that resulted in the best compaction.
4. The system of claim 3 , wherein a stencil is created and transmitted to the user with the combined half-backed codebook to assist the user to extract codeword pairs from the combined half-backed codebook.
5. The system of claim 2 , wherein a hybrid stencil is created to synthesize multiple half-backed codebooks into a hybrid synthesized codebook.
6. A method for data compaction and encryption of anonymized data records, comprising the steps of:
receiving an anonymized tally record from a user, the anonymized tally record comprising a plurality of anonymized sourceblocks and for each anonymized sourceblock a tally value indicating the number of times the anonymized sourceblock occurs in a data source, wherein each of the plurality of anonymized sourceblocks is associated with a fixed sourceblock length, and wherein multiple sourceblock lengths may be included in the anonymized tally record;
for each sourceblock:
parsing the anonymized sourceblocks to identify the anonymized sourceblocks with the highest tally value;
sending the anonymized sourceblocks to a library manager;
receiving a half-backed codebook from the library manager;
transmitting the half-backed codebook to the user;
receiving anonymized sourceblocks from a data deconstruction engine;
for each sourceblock:
assigning a unique codeword to each of the received anonymized sourceblocks;
creating a half-backed codebook, the half-backed codebook comprising a plurality of codeword pairs, wherein each codeword pair comprises an anonymized sourceblock and its associated unique codeword; and
sending the half-backed codebook to the data deconstruction engine.
7. The method of claim 6 , wherein the data deconstruction engine receives two or more anonymized tally records.
8. The method of claim 7 , wherein the two or more anonymized tally records are used to create a combined half-backed codebook, the combined half-backed codebook comprising codeword pairs from each of the anonymized tally records, selected from the sourceblock of each anonymized tally record that resulted in the best compaction.
9. The method of claim 8 , wherein a stencil is created and transmitted to the user with the combined half-backed codebook to assist the user to extract codeword pairs from the combined half-backed codebook.
10. The method of claim 7 , wherein a hybrid stencil is created to synthesize multiple half-backed codebooks into a hybrid synthesized codebook.