Generating multiple sets of integrity information in a vast storage system
A method includes storing a plurality of data in a storage system. A plurality of identifiers corresponding to the plurality of data is determined and the plurality of identifiers are stored in the storage system. A first set of integrity information corresponding to a first system storage level is generated for the plurality of data by performing a first set of cyclic redundancy checks and the first set of integrity information is stored in the storage system. A second set of integrity information corresponding to a second system storage level is generated for the plurality of data and the second set of integrity information is stored in the storage system.
1. A method comprising:
storing a plurality of data in a storage system;
determining a plurality of identifiers corresponding to the plurality of data;
generating, for the plurality of data, a first set of integrity information corresponding to a first system storage level by performing a first set of cyclic redundancy checks;
generating, for the plurality of data, a plurality of corresponding data structures that each include:
a corresponding one of the plurality of identifiers; and
corresponding integrity information of the first set of integrity information;
storing the first set of integrity information and the plurality of identifiers in the storage system via storage of the plurality of corresponding data structures;
generating, for the plurality of data, a second set of integrity information corresponding to a second system storage level by performing a second set of cyclic redundancy checks; and
storing the second set of integrity information in the storage system.
2. The method of claim 1 , wherein the plurality of data is generated based on erasure coding.
3. The method of claim 1 , wherein the plurality of identifiers are associated with at least one data slice generated via an encoding process in accordance with a width, and wherein a corresponding decoding process can accommodate a number of failures equal to the width minus an error coding parameter of the encoding process.
4. The method of claim 1 , wherein the plurality of identifiers identify a virtual memory space that maps to storage units of the storage system.
5. The method of claim 1 , wherein the plurality of identifiers are determined in conjunction with determining a plurality of virtual memory addresses.
6. The method of claim 5 , wherein each virtual memory address of the plurality of virtual memory addresses is associated with a physical address, and wherein the integrity information is generated based on the plurality of virtual memory addresses.
7. The method of claim 1 , further comprising:
performing periodic data storage integrity verification based on accessing at least one of the second set of integrity information in the storage system;
wherein performing the periodic data storage integrity verification is based on periodically determining whether any of the plurality of data has been corrupted.
8. The method of claim 7 , further comprising:
rebuilding at least some of the plurality of data based on determining the at least some of the plurality of data has been corrupted.
9. The method of claim 1 , wherein the storage system is configured to store the data via a plurality of system storage levels that includes the first system storage level and the second system storage level.
10. The method of claim 1 , wherein the first set of integrity information is generated based on a first plurality of data portions of the plurality of data having first data structuring corresponding to the first system storage level, and wherein the second set of integrity information is generated based on a second plurality of data portions of the plurality of data having second data structuring corresponding to the second system storage level.
11. The method of claim 10 , wherein each of the first plurality of data portions include multiple ones of the second plurality of data portions.
12. A computer comprises:
a memory; and
a processing module operable to:
store a plurality of data in a storage system;
determine a plurality of identifiers corresponding to the plurality of data;
generate, for the plurality of data, a first set of integrity information corresponding to a first system storage level by performing a first set of cyclic redundancy checks;
generate, for the plurality of data, a plurality of corresponding data structures that each include:
a corresponding one of the plurality of identifiers; and
corresponding integrity information of the first set of integrity information;
store the first set of integrity information and the plurality of identifiers in the storage system via storage of the plurality of corresponding data structures;
generate, for the plurality of data, a second set of integrity information corresponding to a second system storage level by performing a second set of cyclic redundancy checks; and
store the second set of integrity information in the storage system.
13. The computer of claim 12 , wherein the plurality of identifiers are associated with at least one data slice generated via an encoding process in accordance with a width, and wherein a corresponding decoding process can accommodate a number of failures equal to the width minus an error coding parameter of the encoding process.
14. The computer of claim 12 , wherein the plurality of identifiers identify a virtual memory space that maps to storage units of the storage system.
15. The computer of claim 12 , wherein the plurality of identifiers are determined in conjunction with determining a plurality of virtual memory addresses.
16. The computer of claim 15 , wherein each virtual memory address of the plurality of virtual memory addresses is associated with a physical address, and wherein the integrity information is generated based on the plurality of virtual memory addresses.
17. The computer of claim 12 , further comprising:
performing periodic data storage integrity verification based on accessing at least one of the second set of integrity information in the storage system;
wherein performing the periodic data storage integrity verification is based on periodically determining whether any of the plurality of data has been corrupted.
18. The computer of claim 17 , further comprising:
rebuilding at least some of the plurality of data based on determining the at least some of the plurality of data has been corrupted.
19. A storage system comprises:
a plurality of storage units; and
at least one processing module operable to:
store a plurality of data;
determine a plurality of identifiers corresponding to the plurality of data;
generate, for the plurality of data, a first set of integrity information corresponding to a first system storage level by performing a first set of cyclic redundancy checks;
generate, for the plurality of data, a plurality of corresponding data structures that each include:
a corresponding one of the plurality of identifiers; and
corresponding integrity information of the first set of integrity information;
store the first set of integrity information and the plurality of identifiers in the storage system via storage of the plurality of corresponding data structures;
generate, for the plurality of data, a second set of integrity information corresponding to a second system storage level by performing a second set of cyclic redundancy checks; and
store the second set of integrity information in the storage system.
20. The storage system of claim 19 , wherein the plurality of data is generated based on erasure coding.