Generating integrity information in a vast storage system
A method includes encoding data via erasure coding to produce a plurality of data slices. The method further includes determining a plurality of identifiers corresponding to the data. The method further includes generating integrity information based on the plurality of identifiers by performing a cyclic redundancy check. The method further includes storing the plurality of data slices, the plurality of identifiers, and the integrity information in a storage system.
1. A method comprises:
encoding data via erasure coding to produce a plurality of data slices;
determining a plurality of identifiers corresponding to the data;
generating integrity information based on the plurality of identifiers by performing a cyclic redundancy check; and
storing the plurality of data slices, the plurality of identifiers, and the integrity information in a storage system.
2. The method of claim 1 , wherein the data is encoded in accordance with a width, and wherein a corresponding decoding process can accommodate a number of failures equal to the width minus an error coding parameter utilized to encode the data.
3. The method of claim 2 , wherein the width is greater than a size of the data.
4. The method of claim 1 , further comprising:
performing data storage integrity verification by periodically retrieving data slices of the plurality of data slices from the storage system to verify whether one or more data slices of the plurality of data slices have been corrupted.
5. The method of claim 1 , wherein the plurality of identifiers identify a virtual memory space that maps to storage units of the storage system.
6. The method of claim 1 , wherein the plurality of identifiers are determined in conjunction with determining a plurality of virtual memory addresses corresponding to the plurality of data slices, wherein each virtual memory address of the plurality of virtual memory addresses is associated with a physical address, and wherein the integrity information is generated based on the plurality of virtual memory addresses.
7. The method of claim 1 , wherein storing the plurality of data slices, the plurality of identifiers, and the integrity information in the storage system includes sending the plurality of data slices, the plurality of identifiers, and the integrity information to a plurality of storage units of the storage system for storage therein.
8. The method of claim 7 , wherein the plurality of data slices and the plurality of identifiers and the integrity information are sent to the plurality of storage units based on operational health of the plurality of storage units.
9. The method of claim 1 , wherein determining the integrity information further comprises:
generating data file integrity information for at least some of the plurality of identifiers; and
generating the integrity information based on the data file integrity information.
10. The method of claim 1 , further comprising:
sending first list digest requests to at least first and second storage units of the storage system, the first list digest requests indicating a first range of slice identifiers to include in a first list digest;
receiving, in response to the first list digest requests, first and second list digest responses from the first and second storage units;
determining whether the first list digest response compares favorably to the second list digest response; and
in response to an unfavorable determination, sending second list digest requests to the first and second storage units, the second list digest requests indicating a second range of slice identifiers to include in second list digests.
11. A computer comprises:
an interface;
a memory; and
a processing module operable to:
encode data via erasure coding to produce a plurality of data slices;
determine a plurality of identifiers corresponding to the data;
generate integrity information based on the plurality of identifiers by performing a cyclic redundancy check; and
store the plurality of data slices, the plurality of identifiers, and the integrity information in a storage system.
12. The computer of claim 11 , wherein the data is encoded in accordance with a width, and wherein a corresponding decoding process can accommodate a number of failures equal to the width minus an error coding parameter algorithm utilized to encode the data.
13. The computer of claim 12 , wherein the width is greater than a size of the data.
14. The computer of claim 11 , wherein the processing module further functions to:
perform data storage integrity verification by periodically retrieving data slices of the plurality of data slices from the storage system to verify whether one or more data slices of the plurality of data slices have been corrupted.
15. The computer of claim 11 , wherein the plurality of identifiers identify a virtual memory space that maps to storage units of the storage system.
16. The computer of claim 11 , wherein the plurality of identifiers are determined in conjunction with determining a plurality of virtual memory addresses corresponding to the plurality of data slices, wherein each virtual memory address of the plurality of virtual memory addresses is associated with a physical address, and wherein the integrity information is generated based on the plurality of virtual memory addresses.
17. The computer of claim 11 , wherein storing the plurality of data slices, the plurality of identifiers, and the integrity information in the storage system includes sending the plurality of data slices, the plurality of identifiers and the integrity information to a plurality of storage units of the storage system for storage therein.
18. The computer of claim 17 , wherein the plurality of data slices and the plurality of identifiers and the integrity information are sent to the plurality of storage units based on operational health of the plurality of storage units.
19. The computer of claim 11 , wherein determining the integrity information further comprises:
generating data file integrity information for at least some of the plurality of identifiers; and
generating the integrity information based on the data file integrity information.
20. The computer of claim 11 , wherein the data is dispersed storage error encoded in accordance with dispersed storage error coding parameters to produce the plurality of data slices.