Method and apparatus for nested dispersed storage
View Patent ↗A method begins by a DS processing module generating a plurality of encoded slices from a data segment using an error encoding function. The method continues with the DS processing module identifying a plurality of DS storage units for storing the plurality of encoded slices. The method continues with the DS processing module selecting an encoded slice of the plurality of encoded slices for sub-slicing using a sub-slicing encoding function to produce a selected encoded slice. The method continues with the DS processing module outputting the plurality of encoded slices to the plurality of DS storage units. The method continues with the DS processing module outputting a command to a DS storage unit of the plurality of DS storage units corresponding to the selected encoded slice, wherein the command includes an instruction to sub-slice the selected encoded slice.
1. A method comprises:
encoding, by a distributed storage (DS) processing module, a data segment using a first layer of dispersed storage error encoding to produce a plurality of encoded slices;
identifying, by the DS processing module, storage units for storing the plurality of encoded slices;
selecting, by the DS processing module, an encoded slice of the plurality of encoded slices for a second layer of dispersed storage error encoding to produce a selected encoded slice;
outputting, by the DS processing module, the plurality of encoded slices to the storage units, wherein the storage units store the plurality of encoded data slices;
outputting, by the DS processing module, a command to a storage unit of the storage units corresponding to the selected encoded slice, wherein the command includes an instruction to encoding the selected encoded slice using the second layer of dispersed storage error encoding; and
encoding, by the storage unit, the selected encoded slice using the second layer of dispersed storage error encoding to produce a plurality of encoded sub-slices.
2. The method of claim 1 further comprises:
maintaining, by the DS processing module, a list of the storage units for storing the plurality of encoded slices less the selected encoded slice; and
maintaining, by the DS processing module, in the list the storage units receiving the command.
3. The method of claim 1 , wherein the command further comprises at least one of:
an instruction for storing plurality of encoded sub-slices in other-storage units; and
identity of the second layer of dispersed storage error encoding.
4. The method of claim 1 , wherein the selecting the encoded slice comprises:
determining a security threshold for the plurality of encoded slices based on a read threshold of the error encoding function; and
selecting a set of encoded slices from the plurality of encoded slices based on the security threshold, wherein the set of encoded slices includes the selected encoded slice.
5. The method of claim 1 , wherein the selecting the encoded slice comprises at least one of:
determining retrieval latency for each of the storage units;
determining priority of the selected encoded slice;
determining a data type of the data segment;
determining availability of each of the storage units; and
determining cost of storing an encoded slice of the plurality of encoded slices in each of the storage units.
6. The method of claim 1 further comprises:
outputting a second command to the storage units less the storage unit, wherein the second command includes an instruction not to perform the second layer of dispersed storage error encoding.
7. The method of claim 1 further comprises at least one of:
receiving, from the DS storage unit, information regarding the plurality of encoded sub-slices; and
receiving, from the DS storage unit, information regarding other storage units identified for storing the plurality of encoded sub-slices.
8. A distributed storage (DS) processing system comprises:
a first processing module operable to:
encode a data segment using a first layer of dispersed storage error encoding to produce a plurality of encoded slices;
identify storage units for storing the plurality of encoded slices;
select an encoded slice of the plurality of encoded slices for a second layer of dispersed storage error encoding to produce a selected encoded slice;
output, via a first network interface, the plurality of encoded slices to the storage units, wherein the storage units store the plurality of encoded data slices;
output, via the first network interface, a command to a storage unit of the storage units corresponding to the selected encoded slice, wherein the command includes an instruction to encoding the selected encoded slice using the second layer of dispersed storage error encoding;
a second processing module associated with the storage unit operable to encode the selected encoded slice using the second layer of dispersed storage error encoding to produce a plurality of encoded sub-slices.
9. The DS processing system of claim 8 further comprises:
memory, wherein the first processing module is further operable to:
maintain, in the memory, a list of the storage units for storing the plurality of encoded slices less the selected encoded slice; and
maintain in the list the storage unit receiving the command.
10. The DS processing system of claim 8 , wherein the command further comprises at least one of:
an instruction for storing plurality of encoded sub-slices in other storage units; and
identity of the second layer of dispersed storage error encoding.
11. The DS processing system of claim 8 , wherein the first processing module further functions to select the encoded slice by:
determining a security threshold for the plurality of encoded slices based on a read threshold of the error encoding function; and
selecting a set of encoded slices from the plurality of encoded slices based on the security threshold, wherein the set of encoded slices includes the selected encoded slice.
12. The DS processing system of claim 8 , wherein the first processing module further functions to select the encoded slice by at least one of:
determining retrieval latency for each of the storage units;
determining priority of the selected encoded slice;
determining a data type of the data segment;
determining availability of each of the storage units; and
determining cost of storing an encoded slice of the plurality of encoded slices in each of the storage units.
13. The DS processing system of claim 8 , wherein the first processing module further functions to:
output, via the network interface, a second command to the storage units less the DS storage unit, wherein the second command includes an instruction not to perform the second layer of dispersed storage error encoding.
14. The DS processing system of claim 8 , wherein the first processing module further functions to receive from the storage unit via the network interface at least one of:
information regarding the second layer of dispersed storage error encoding;
information regarding the plurality of encoded sub-slices; and
information regarding other storage units identified for storing the plurality of encoded sub-slices.