System and method for random-access manipulation of compacted data files
A system and method for random-access manipulation of compacted data files, utilizing a reference codebook, a random-access engine, a data deconstruction engine, and a data deconstruction engine. The system may receive a data query pertaining to a data read or data write request, wherein the data file to be read from or written to is a compacted data file. A random-access engine may facilitate data manipulation processes by accessing a reference codebook associated with the compacted data file, a frequency table used to construct the reference codebook, and data query details. A data read request is supported by random-access search capabilities that may enable the locating and decoding of the bits corresponding to data query details. A random-access engine facilitates data write processes. The random-access engine may encode the data to be written, insert the encoded data into a compacted data file, and update the codebook as needed.
1. A system for random-access manipulation of compacted data files, comprising:
a computing device comprising a memory, a processor, and a non-volatile data storage device;
at least one optimized hash-based reference codebook;
a reconstruction engine comprising a first plurality of programming instructions stored in the memory and operable on the processor, wherein the first plurality of programming instructions, when operating on the processor, cause the computing device to:
retrieve a plurality of reference codes from a compacted data file;
decode each of the plurality of retrieved reference codes by, for each retrieved reference code, retrieving the chunklet associated with the respective reference code from the reference codebook; and
provide the retrieved chunklets as a data stream to the requesting process; and
a random access engine comprising a second plurality of programming instructions stored in the memory and operating on the processor, wherein the second plurality of programming instructions, when operating on the processor, cause the computing device to:
receive a data search query;
estimate, using an estimator module, a first starting bit location in the compacted data file;
refine the first starting bit location by:
determining whether a bit sequence starting at the first starting bit location corresponds to a codeword boundary and, if not, traversing the reference codebook until a codeword boundary is located at a new starting bit;
traversing from the new starting bit until a start codeword corresponding to the beginning of the data search query is identified; and
sending the first start codeword and a plurality of immediately following codewords from the compacted data file to the reconstruction engine for decoding.
2. The system of claim 1 , further comprising a deconstruction engine comprising a third plurality of programming instructions stored in the memory and operable on the processor, wherein the third plurality of programming instructions, when operating on the processor, cause the computing device to:
deconstruct a data stream into a plurality of data chunklets;
encode the data stream using the reference codebook by:
retrieving a reference code for each chunklet from the reference codebook; and
where there is no reference code for a first chunklet, generating a hash code as a new reference code and storing the first chunklet and its newly-created reference code in the reference codebook.
3. A method for random-access manipulation of compacted data files, comprising the steps of:
retrieving, at a reconstruction engine comprising a first plurality of programming instructions stored in a memory and operable on a processor, wherein the first plurality of programming instructions, a plurality of reference codes from a compacted data file;
decoding each of the plurality of retrieved reference codes by, for each retrieved reference code, retrieving the chunklet associated with the respective reference code from the reference codebook;
providing the retrieved chunklets as a data stream to the requesting process;
receiving, at a random access engine comprising a second plurality of programming instructions stored in the memory and operating on the processor, a data search query;
estimating, using an estimator module of the random access engine, a first starting bit location in the compacted data file;
refining the first starting bit location by determining whether a bit sequence starting at the first starting bit location corresponds to a codeword boundary and, if not, traversing the reference codebook until a codeword boundary is located at a new starting bit;
traversing from the new starting bit until a start codeword corresponding to the beginning of the data search query is identified; and
sending the first start codeword and a plurality of immediately following codewords from the compacted data file to the reconstruction engine for decoding.
4. The method of claim 3 , further comprising the steps of:
deconstructing, at a deconstruction engine comprising a third plurality of programming instructions stored in the memory and operable on the processor, a data stream into a plurality of data chunklets;
encoding the data stream using the reference codebook by retrieving a reference code for each chunklet from the reference codebook; and
where there is no reference code for a first chunklet, generating a hash code as a new reference code and storing the first chunklet and the its newly-created reference code in the reference codebook.