System and method for error-resilient data reduction
A system and method for error-resilient data reduction, utilizing a phase detector, a data requestor, a multi-phase trainer, a reconstruction engine, a deconstruction engine, and one or more reference codebooks. A multi-phase trainer may be used to train the reconstruction and deconstruction engines on various phase sourceblocks in order recover quickly from corrupted data files that cause the phase alignment of the sourceblocks to become out of phase. A phase detector may determine when the sourceblocks get out of phase and when the return to in-phase by checking if a predetermined threshold probability of correct encoding is met. Data requestor may request for retransmission only the data that was received out of phase.
1. A system for error-resilient data reduction, comprising:
a computing device comprising a memory, a processor, and a non-volatile data storage device;
at least one reference codebook comprising key-value pairs of data;
a data deconstruction engine comprising a first plurality of programming instructions stored in the memory of, and operating on a processor of, the computing device, wherein the first plurality of programming instructions, when operating on the processor, cause the computing device to:
train an encoding algorithm on sourceblocks at multiple phases, wherein each phase comprises a distinct starting bit offset of a respective sourceblock;
receive data from a data source;
deconstruct the incoming data into a plurality of chunklets;
encode the data using the encoding algorithm and the reference codebook; and
send the encoded data to a data reconstruction engine; and
a data reconstruction engine comprising a second plurality of programming instructions stored in the memory of, and operating on a processor of, the computing device, wherein the second plurality of programming instructions, when operating on the processor, cause the computing device to:
receive encoded data;
decode the data using the key-value pairs stored within the reference codebook;
determine if the decoded data has exceeded a predetermined threshold probability that the decoded data was properly encoded, wherein the decoded data is in-phase if the threshold is exceeded and the decoded data is out-of-phase if the threshold is not exceeded; and
request retransmission of out-of-phase data.
2. The system of claim 1 , wherein the threshold probability is determined using logistic regression.
3. The system of claim 1 , wherein multiple phases refer to byte-phase sourceblocks with an offset.
4. The system of claim 3 , wherein the offset is an integer value in the inclusive range of 1 to 7.
5. A method for error-resilient data reduction, comprising the steps of:
training an encoding algorithm on sourceblocks at multiple phases, wherein each phase comprises a distinct starting bit offset of a respective sourceblock;
receiving data from a data source;
deconstructing the incoming data into a plurality of chunklets;
encoding the chunklets using an encoding algorithm and a reference codebook;
sending the encoded data to a data reconstruction engine;
receiving encoded data;
decoding the data using key-value pairs stored within the reference codebook;
determining if the decoded data has exceeded a predetermined threshold probability that the decoded data was properly encoded, wherein the decoded data is in-phase if the threshold is exceeded and the decoded data is out-of-phase if the threshold is not exceeded; and
requesting retransmission of out-of-phase data.
6. The method of claim 5 , wherein the threshold probability is determined using logistic regression.
7. The method of claim 5 , wherein multiple phases refer o byte-phase sourceblocks with an offset.
8. The method of claim 7 , wherein the offset is an integer value in the inclusive range of 1 to 7.