Distributed checksum computation
Data is divided into parts and each part provided to a different processor. Each processor processes the provided data part to produce a partial CRC result. The partial CRC results from each of the different processors are XORed to produce a CRC of the data.
1. A method comprising:
dividing data into parts and providing each part to a memory buffer of a different machine data processor, each data part comprising a string of bits, the string of bits comprising sequences of bits not representing data values separating portions of the string representing data values;
each processor processing a provided data part to produce a partial CRC result; and
xoring partial CRC results from each of the different processors to produce a CRC of the data.
2. The method of claim 1 , wherein each processor processing a provided data part to produce a partial CRC result further comprises:
xoring a plurality of CRCs of bit strings having one or more set bits, where each set bit of the bit strings corresponds to a set bit of the provided data part.
3. The method of claim 1 , wherein each processor processing a provided data part to produce a partial CRC result further comprises:
each processor XOR multiplying a CRC for a bit string having a single set bit with the provided data part.
4. The method of claim 3 , wherein each processor XOR multiplying a CRC for a bit string having a single set bit with the provided data part further comprises:
each processor generating a CRC of upper bits of the result of the XOR multiplying, and XORing the CRC of the upper bits with lower bits of the result of XOR multiplying.
5. The method of claim 3 , wherein each processor XOR multiplying a CRC for a bit string having a single set bit with the provided data part further comprises:
a position of the single set bit in the bit string varying according to a length of a sequence of bits not representing data values between portions of the data part representing data values.
6. The method of claim 1 , wherein each data part comprising a string of bits, the string of bits comprising sequences of bits not representing data values separating portions of the string representing data values further comprises:
each processor XOR multiplying a CRC for a bit string having a single set bit with the provided data part, where the single bit is set in a position that reflects the length of the sequence of bits separating the portions of the string of bits representing data values.
7. The method of claim 6 , wherein each processor XOR multiplying a CRC for a bit string having a single set bit with the provided data part further comprises:
each processor generating a CRC of upper bits of the result of the XOR multiplying, and XORing the CRC of the upper bits with lower bits of the result of XOR multiplying.
8. The method of claim 6 , wherein each processor XOR multiplying a CRC for a bit string having a single set bit with the provided data part further comprises:
a position of the single set bit in the bit string varying according to a length of a sequence of bits not representing data values between portions of the data part representing data values.
9. The method of claim 1 , wherein each processor processing a provided data part to produce a partial CRC result further comprises:
each provided data part comprising a plurality of pixel data.
10. The method of claim 1 , wherein each processor processing a provided data part to produce a partial CRC result further comprises:
each provided part comprising a portion of a distributed data file.
11. The method of claim 1 , wherein each processor processing a provided data part to produce a partial CRC result further comprises:
each processor repeatedly applying a CRC forward operation to produce the partial CRC result.