Data error control
Multiple corruptions and/or erasures in data storage or data communication systems are corrected. An encoder generates M of parity fields from N data channels. Each item of the generated parity fields is the result of simple exclusive-or operations on one item from one or more data fields and possibly one item from one or more of the other parity fields. A decoder can regenerate as many as M missing or corrupted fields of either data or parity using combinations of correct and/or previously corrected items as inputs using M independent parity equations to solve for and correct each missing or corrupted item in turn.
1. A computerized method for data error protection comprising:
receiving source data;
representing the source data as N channels of source data, each including a series of I entries;
storing the N series of I entries in a buffer; and
encoding, based on the N series of I entries in the buffer, the N channels of source data to form M channels of encoded data such that the M channels of encoded data and the N channels of source data satisfy at least M×I independent parity equations, at least I entries of some of the M channels of encoded data each being an argument to at least M of the parity equations, and at least N+1 entries, from the stored series of entries and the previously encoded entries, contribute to one or more of the parity equations.
2. The computerized method of claim 1 wherein encoding the N channels comprises augmenting the N channels of source data with M channels of parity data.
3. The computerized method of claim 2 wherein each entry of the N channels of source data is an argument to M of the parity equations.
4. The computerized method of claim 2 wherein in at least some of the M channels of parity data comprise at least I entries in the channel are each an argument to at least 2 of the parity equations.
5. The computerized method of claim 4 wherein in at least some of the M channels of parity data comprise at least I entries in the channel are each an argument to at least M of the parity equations.
6. The computerized method of claim 2 wherein each channel of parity data is associated with a class of parity equations, the class of parity equations being associated with a set of the N+M channels that contribute arguments to the parity equations and relative offsets of entries in the channels that contribute to any of the equations in the class.
7. The computerized method of claim 6 wherein the offsets for at least some of the classes of parity equations are such that the relative offsets of the entries form a diagonal line across the N channels of source data when arranged as an array of N columns by I rows.
8. The computerized method of claim 1 wherein representing the source data as N channels of source data includes concatenating a series of extra entries with the series of I entries.
9. The computerized method of claim 8 wherein encoding the N channels of source data to form N+M channels of encoded data includes applying at least some of the parity equations to extra entries and entries representing the source data to determine entries of the encoded data.
10. The computerized method of claim 8 wherein the extra entries have zero values.
11. The computerized method of claim 8 wherein encoding the data further comprises determining the extra entries as a key to the encoded data.
12. The computerized method of claim 1 wherein the encoding of the data enables reconstruction of the source data from any N of the N+M channels of encoded data.
13. The computerized method of claim 1 wherein the encoding of the data enables detection of errors in M−I of the N+M channels of encoded data.
14. The computerized method of claim 1 wherein accepting the source data includes accepting the source data as one or more data streams, representing the one or more data streams as N channels, and encoding the data includes processing the accepted data using a cross-convolutional encoding approach across the N channels to produce the N+M channels of encoded data as a stream of data.
15. The computerized method of claim 1 wherein any two of the parity equations share at most one entry of the N+M channels as an argument.
16. Software stored on non-transitory computer-readable tangible media comprising instructions for causing a data processing system to:
receive source data;
represent the source data as N channels of source data, each including a series of I entries;
store the N series of I entries in a buffer; and
encode, based on the N series of I entries in the buffer, the N channels of source data to form M channels of encoded data such that the M channels of encoded data and the N channels of source data satisfy at least M×I independent parity equations, at least I entries of some of the M channels of encoded data each being an argument to at least M of the parity equations, and at least N+1 entries, from the stored series of entries and the previously encoded entries, contribute to one or more of the parity equations.
17. A computerized method for data error protection comprising:
receiving source data;
representing the source data as N channels of source data, each including a series of I entries;
storing the N series of I entries in a buffer; and
iteratively encoding, based on the N series of I entries in the buffer, the N channels of source data to form M channels of encoded data, at one or more iterations applying one or more parity equations to one or more entries in the N series of I entries and to one or more previously constructed entries in the M channels of encoded data to construct a parity entry, the M channels of encoded data and N channels of source data satisfying at least M×I independent parity equations, at least I entries of some of the M channels of encoded data each contributing to at least M of the parity equations, and at least N+1 entries, from the stored series of entries and the previously encoded entries, contribute to one or more of the parity equations.
18. A computerized method for distributed storage of data comprising:
receiving source data;
representing the source data as N channels of source data, each including a series of I entries;
storing the N channels of source data in a buffer;
iteratively encoding the N channels of source data to form N+M channels of encoded data, at one or more iterations applying one or more parity equations to one or more entries from the N channels and to one or more entries from previously encoded channels to construct a parity entry, the N+M channels of encoded data satisfying at least M×I independent parity equations, wherein at least I entries of some of the N+M channels of encoded data each being an argument to at least M of the parity equations, and at least N+1 entries, from the stored series of entries and the previously encoded entries, contribute to one or more of the parity equations; and
distributing the N+M channels of encoded data to separate storages.