Utilizing integrity information to determine corruption in a vast storage system
A method includes determining a plurality of identifiers based on a data retrieval request. Integrity information is generated based on determining the plurality of identifiers. Stored integrity information corresponding to the data retrieval request is compared with the integrity information. When the stored integrity information compares unfavorably with the integrity information, corruption associated with the plurality of identifiers is determined.
1. A method comprises:
determining a plurality of identifiers based on a data retrieval request;
generating integrity information based on determining the plurality of identifiers;
comparing stored integrity information corresponding to the data retrieval request with the integrity information; and
when the stored integrity information compares unfavorably with the integrity information, determining corruption associated with the plurality of identifiers.
2. The method of claim 1 , wherein the plurality of identifiers are associated with at least one data slice generated via an encoding process in accordance with a width, and wherein a corresponding decoding process can accommodate a number of failures equal to the width minus an error coding parameter of the encoding process.
3. The method of claim 1 , further comprising:
performing data storage integrity verification by periodically retrieving data slices from a storage system to determine whether one or more data slices have been corrupted.
4. The method of claim 1 , wherein the plurality of identifiers identify a virtual memory space that maps to storage units of a storage system.
5. The method of claim 1 , wherein the plurality of identifiers are determined in conjunction with determining a plurality of virtual memory addresses.
6. The method of claim 5 , wherein each virtual memory address of the plurality of virtual memory addresses is associated with a physical address, and wherein the integrity information is generated based on the plurality of virtual memory addresses.
7. The method of claim 1 , further comprising:
receiving the stored integrity information corresponding to the data retrieval request.
8. The method of claim 1 , further comprising:
decoding a plurality of data slices corresponding to the plurality of identifiers when the stored integrity information compares favorably with the integrity information.
9. The method of claim 1 , further comprising:
performing a set level integrity check when the stored integrity information compares unfavorably with the integrity information to identify at least one set of data slices having an error; and
for each set of data slices identified as having an error, performing an individual encoded slice name integrity check to identify at least one data slice having an error in the at least one set of data slices.
10. The method of claim 9 , further comprising:
identifying a proper subset of a plurality of data slices corresponding to the plurality of identifiers by excluding ones of the plurality of data slices identified as having an error; and
reproducing a data file by dispersed storage error decoding the proper subset of the plurality of data slices when the proper subset of the plurality of data slices includes at least a decode threshold number of data slices.
11. The method of claim 1 , further comprising:
determining a plurality of sets of slice names in accordance with the data retrieval request, wherein the plurality of identifiers are determined based on the plurality of sets of slice names, wherein the plurality of sets of slice names correspond to a plurality of data slices stored in a set of storage units that were dispersed storage error encoded utilizing dispersed storage error coding parameters.
12. The method of claim 1 , further comprising:
rebuilding at least one data slice associated with the plurality of identifiers based on determining the corruption.
13. A computer comprises:
a memory; and
a processing module operable to:
determine a plurality of identifiers based on a data retrieval request;
generate integrity information based on determining the plurality of identifiers;
compare stored integrity information corresponding to the data retrieval request with the integrity information; and
when the stored integrity information compares unfavorably with the integrity information, determine corruption associated with the plurality of identifiers.
14. The computer of claim 13 , wherein the plurality of identifiers are associated with at least one data slice generated via an encoding process in accordance with a width, and wherein a corresponding decoding process can accommodate a number of failures equal to the width minus an error coding parameter of the encoding process.
15. The computer of claim 13 , wherein the processing module further functions to:
performing data storage integrity verification by periodically retrieving data slices from a storage system to determine whether one or more data slices have been corrupted.
16. The computer of claim 13 , wherein the plurality of identifiers identify a virtual memory space that maps to storage units of a storage system.
17. The computer of claim 13 , wherein the plurality of identifiers are determined in conjunction with determining a plurality of virtual memory addresses.
18. The computer of claim 17 , wherein each virtual memory address of the plurality of virtual memory addresses is associated with a physical address, and wherein the integrity information is generated based on the plurality of virtual memory addresses.
19. The computer of claim 13 , wherein the processing module further functions to:
rebuild at least one data slice associated with the plurality of identifiers based on determining the corruption.
20. The computer of claim 13 , wherein the processing module further functions to:
decode a plurality of data slices corresponding to the plurality of identifiers when the stored integrity information compares favorably with the integrity information.