Integrity processing in a dispersed storage network
A distributed storage integrity system in a dispersed storage network includes a scanning agent and a control unit. The scanning agent identifies an encoded data slice that requires rebuilding, wherein the encoded data slice is one of a plurality of encoded data slices generated from a data segment using an error encoding dispersal function. The control unit retrieves at least a number T of encoded data slices needed to reconstruct the data segment based on the error encoding dispersal function. The control unit is operable to reconstruct the data segment from at least the number T of the encoded data slices and generate a rebuilt encoded data slice from the reconstructed data segment. The scanning agent is located in a storage unit and the control unit is located in the storage unit or in a storage integrity processing unit, a dispersed storage processing unit or a dispersed storage managing unit.
1. A processing unit in a storage network, comprises:
a memory including a rebuild queue that stores indicators of data slices for rebuilding; and
at least one rebuild module utilizing processing hardware operable to:
select one of the data slices from the rebuild queue based on a priority of a selected data slice associated with the select one of the data slices, wherein the selected data slice is one of a number of a plurality of encoded data slices of an encoded data segment;
reconstruct the encoded data segment from at least a threshold number of the number of the plurality of encoded data slices of the encoded data segment to generate a reconstructed data segment; and
generate a rebuilt encoded data slice corresponding to the selected data slice from the reconstructed data segment.
2. The processing unit of claim 1 , further comprising:
a network interface for interfacing with a plurality of storage units in the storage network; and
at least one scanning agent operable to:
scan an address range of encoded data slices stored in the plurality of storage units;
identify an encoded data slice having an address within the address range for rebuilding to generate an identified data slice.
3. The processing unit of claim 2 , wherein the address range of encoded data slices corresponds to a portion of a virtual storage network addressing space.
4. The processing unit of claim 3 , wherein the portion of the virtual storage network addressing space is a portion of a user vault, wherein the user vault includes a virtual storage network address space associated with a user device.
5. The processing unit of claim 3 , wherein the portion of the virtual storage network addressing space corresponds to a portion of at least one pillar of a user vault, wherein the user vault includes a virtual storage network address space associated with a user device and wherein a number of pillars of the user vault equals the number of encoded data slices of an encoded data segment for the user vault set in accordance data storage parameters and an error coding scheme.
6. The processing unit of claim 1 , wherein the at least one rebuild module is further operable to:
determine a slice identifier of the selected data slice, wherein the slice identifier includes a vault identifier that identifies a user vault from a physical address table; and
store the slice identifier of the identified slice in the rebuild queue as one of the data slices for rebuilding.
7. The processing unit of claim 1 , wherein the encoded data segment is one of a plurality of encoded data segments of a data object and wherein the priority is associated with the data object.
8. The processing unit of claim 7 , wherein the priority associated with the data object includes at least one of: a priority indicator for the data object, a priority indicator of a user associated with the data object, a priority indicator for a data object type, or a priority indicator based on a data slice age.
9. The processing unit of claim 1 , wherein the threshold number corresponds to a threshold number of the number of the plurality of encoded data slices needed to reconstruct the encoded data segment based on an error encoding dispersal function, wherein the number of the plurality of encoded data slices of the encoded data segment is equal to a threshold number of the number of the plurality of encoded data slices.
10. The processing unit of claim 9 , wherein the rebuild module is operable to reconstruct the encoded data segment from at least the threshold number of the plurality of encoded data slices of by:
decoding at least the threshold number of the plurality of encoded data slices using a decoding matrix in accordance with an error encoding dispersal function.
11. The processing unit of claim 10 , wherein the rebuild module is operable to decode at least the threshold number of the plurality of encoded data slices using a decoding matrix by:
deslicing at least the threshold number of the plurality of encoded data slices to generate a portion of an encoded data segment; and
multiplying the portion of the encoded data segment with the decoding matrix, wherein the decoding matrix is an M×R matrix, wherein the decoding matrix includes a number R of rows equal to the number of the plurality of encoded data slices and includes a number M of columns equal to the threshold number of the plurality of encoded data slices needed to reconstruct the data segment based on the error encoding dispersal function.
12. A device in a storage network comprising:
a network interface for interfacing with a plurality of storage units in the storage network;
a memory storing operational instructions and a rebuild queue that stores data slices for rebuilding; and
at least one rebuild module utilizing processing circuitry configured to execute the operational instructions, wherein the operational instructions cause the processing circuitry to:
select one of the data slices from the rebuild queue based on a priority of a selected data slice associated with the select one of the data slices, wherein the selected data slice is one of a number of a plurality of encoded data slices of an encoded data segment;
reconstruct the encoded data segment from at least a threshold number of the number of the plurality of encoded data slices of the encoded data segment to generate a reconstructed data segment; and
generate a rebuilt encoded data slice corresponding to the selected data slice from the reconstructed data segment.
13. The device of claim 12 , further comprising at least one scanning agent operable to:
scan an address range of encoded data slices stored in the plurality of storage units;
identify an encoded data slice having an address within the address range for rebuilding to generate an identified data slice.
14. The device of claim 13 , wherein the address range of encoded data slices corresponds to a portion of a virtual storage network addressing space.
15. The device of claim 14 , wherein the portion of the virtual storage network addressing space is a portion of a user vault, wherein the user vault includes a virtual storage network address space associated with a user device.
16. The device of claim 14 , wherein the portion of the virtual storage network addressing space corresponds to a portion of at least one pillar of a user vault, wherein the user vault includes a virtual storage network address space associated with a user device and wherein a number of pillars of the user vault equals the number of encoded data slices of an encoded data segment for the user vault set in accordance data storage parameters and an error coding scheme.
17. A method comprising:
selecting one of a plurality of data slices from a rebuild queue based on a priority of a selected data slice associated with the select one of the plurality of data slices, wherein the selected data slice is one of a number of a plurality of encoded data slices of an encoded data segment;
reconstruct the encoded data segment from at least a threshold number of the number of the plurality of encoded data slices of the encoded data segment to generate a reconstructed data segment; and
generate a rebuilt encoded data slice corresponding to the selected data slice from the reconstructed data segment.
18. The method of claim 17 , further comprising:
scanning an address range of encoded data slices stored in a plurality of storage units; and
identifying an encoded data slice having an address within the address range for rebuilding to generate an identified data slice.
19. The method of claim 18 , wherein the address range of encoded data slices corresponds to a portion of a virtual storage network addressing space.
20. The method of claim 19 , wherein the portion of the virtual storage network addressing space is a portion of a user vault, wherein the user vault includes a virtual storage network address space associated with a user device.