System and method for random-access manipulation of compacted data files
A system and method for random-access manipulation of compacted data files, utilizing a reference codebook, a random-access engine, a data deconstruction engine, and a data deconstruction engine. The system may receive a data query pertaining to a data read or data write request, wherein the data file to be read from or written to is a compacted data file. A random-access engine may facilitate data manipulation processes by transforming the codebook into a hierarchical representation and then traversing the representation scanning for specific codewords associated with a data query request. In an embodiment, an estimator module is present and configured to utilize cardinality estimation to determine a starting codeword to begin searching the compacted data file for the data associated with the data query. The random-access engine may encode the data to be written, insert the encoded data into a compacted data file, and update the codebook as needed.
1 . A system for random-access manipulation of a compacted data file, comprising:
a computing device comprising a memory, a processor, and a data storage device;
a random access engine comprising a first plurality of programming instructions that, when operating on the processor, cause the computing device to:
receive a data search query directed to the data file, wherein the compacted file is compacted using a codebook;
process the file using the codebook to identify a codeword corresponding to the query; and
send the codeword to a decoder;
an estimator module comprising a third plurality of programming instructions that, when operating on the processor, cause the computing device to:
estimate a starting location in the compacted data file;
refine the starting location by:
identifying codeword boundaries within the compacted data file;
determining whether a sequence starting at the estimated starting location corresponds to a codeword boundary;
if the estimated starting location does correspond to a codeword boundary, use the estimated starting location as the starting location; and
if the estimated starting location does not correspond to a codeword boundary, advance the estimated starting location one bit and repeat the determining step.
2 . The system of claim 1 , further comprising a deconstruction engine comprising a second plurality of programming instructions that, when operating on the processor, cause the computing device to:
deconstruct a data stream into data sourceblocks;
create the data file by compacting the data sourceblocks using a codebook; and
send the reference codebook to the random access engine.
3 . The system of claim 1 , wherein the estimator module sends the codeword boundary to the random access engine.
4 . The system of claim 3 , wherein the random access engine uses the codeword boundary to begin traversing the codebook.
5 . A method for random-access manipulation of a compacted data file, comprising the steps of:
receiving a data search query directed to the data file, wherein the compacted file is compacted using a codebook;
processing the file using a codebook to identify a codeword corresponding to the query; and
sending the codeword to a decoder;
estimating, using the estimator module comprising a third plurality of programming instructions that, when operating on the processor, cause the computing device to:
estimating a starting location in the data file;
refining the starting location by:
identifying codeword boundaries within the compacted data file;
determining whether a sequence starting at the estimated starting location corresponds to a codeword boundary;
if the estimated starting location does correspond to a codeword boundary, use the estimated starting location as the starting location; and
if the estimated starting location does not correspond to a codeword boundary, advance the estimated starting location one bit and repeat the determining step.
6 . The method of claim 5 , further comprising the steps of:
deconstructing a data stream into data sourceblocks;
creating the compacted data file by compacting the data sourceblocks using a codebook; and
sending the results to the random access engine.
7 . The method of claim 5 , wherein the codeword boundary estimate is used as a point to begin traversing the codebook.
8 . One or more non-transitory computer-storage media having computer-executable instructions embodied thereon that, when executed by one or more processors of a computing system employing a system for random-access manipulation of a compacted data file, cause the computing system to perform the method of claim 5 .
9 . The media of claim 8 , further comprising a deconstruction engine comprising a second plurality of programming instructions stored in the memory and operable on the processor, wherein the second plurality of programming instructions, when operating on the processor, cause the computing device to:
deconstruct a data stream into data sourceblocks;
create the compacted data file by compacting the data sourceblocks using a codebook; and
send the reference codebook to the random access engine.
10 . The media of claim 8 , wherein the random access engine uses the codeword boundary as point to begin traversing the codebook.