Data storage system and method by shredding and deshredding
View Patent ↗A system and method for data storage by shredding and deshredding of the data allows for various combinations of processing of the data to provide various resultant storage of the data. Data storage and retrieval functions include various combinations of data redundancy generation, data compression and decompression, data encryption and decryption, and data integrity by signature generation and verification. Data shredding is performed by shredders and data deshredding is performed by deshredders that have some implementations that allocate processing internally in the shredder and deshredder either in parallel to multiple processors or sequentially to a single processor. Other implementations use multiple processing through multi-level shredders and deshredders. Redundancy generation includes implementations using non-systematic encoding, systematic encoding, or a hybrid combination. Shredder based tag generators and deshredder based tag readers are used in some implementations to allow the deshredders to adapt to various versions of the shredders.
1. A method for decoding encoded data elements retrieved from storage units by use of at least one processor comprising:
transmitting a retrieval request for a plurality of encoded data elements to a plurality of storage units; wherein each encoded data element of the plurality of encoded data elements is associated with a data object;
retrieving metadata from a set of storage units of the plurality of storage units, wherein the metadata includes information regarding the plurality of encoded data elements, and further wherein the information is sufficient to determine one or more transforms of a set of transforms used to create each encoded data element of the plurality of encoded data elements, wherein each transform of the set of transforms is different from other transforms in the set of transforms; and
based on the information regarding the plurality of encoded data elements, performing one or more of:
retrieving a threshold number of encoded data elements of the plurality of encoded data elements from the set of storage units of the plurality of storage units, wherein the threshold number of encoded data elements is a number of encoded data elements of the plurality of encoded data elements sufficient to recover a first piece of data and further wherein the number of encoded data elements of the plurality of encoded data elements sufficient to recover the first piece of data is less than a total number of encoded data slices associated with the data object; and
decoding the threshold number of encoded data elements retrieved to recover the data object, wherein the data object is divided into a set of separate pieces of data and wherein the set of separate pieces of data includes the piece of data.
2. The method of claim 1 , further including:
retrieving and decoding another threshold number of encoded data elements to produce a second piece of data; and
combining the second piece of data with the first piece of data to recover the data.
3. The method of claim 1 , further including selecting the set of storage units of the plurality of storage units for retrieving the threshold number of encoded data elements, wherein the set of storage units includes the storage unit for retrieving the metadata.
4. The method of claim 1 , wherein systematic decoding is utilized to decode the threshold number of encoded data elements.
5. The method of claim 1 , wherein non-systematic decoding is utilized to decode the threshold number of encoded data elements.
6. The method of claim 1 , wherein a hybrid combination of systematic decoding and non-systematic decoding is utilized to decode the threshold number of encoded data elements.
7. The method of claim 1 , wherein the metadata includes information pertaining to relationships among different encoded data elements of the plurality of encoded data elements.
8. The method of claim 1 , wherein the one or more transforms collectively form a matrix.
9. The method of claim 1 , wherein the one or more transforms includes information sufficient to identify at least one of an encryption function, a sequencing function, and parameters that have changed, in the generation of the plurality of encoded data elements.
10. The method of claim 1 , wherein the metadata includes one or more information pertaining to:
data geometries;
data relationships;
data nesting;
structure;
coding structure;
data alignment;
size and length of data; and
data hash.
11. An apparatus for decoding encoded data elements retrieved from storage units comprising:
an interface; and
at least one processor configured with the interface to:
transmit a retrieval request for a plurality of encoded data elements to a plurality of storage units; wherein each encoded data element of the plurality of encoded data elements is associated with a data object;
retrieve metadata from a storage unit of the plurality of storage units, wherein the metadata includes information sufficient to determine one or more transforms of a set of transforms used to create each encoded data element of the plurality of encoded data elements, wherein each transform of the set of transforms is different from other transforms in the set of transforms; and
based on the information regarding the plurality of encoded data elements, perform one or more of:
retrieve a threshold number of encoded data elements of the plurality of encoded data elements from a set of storage units of the plurality of storage units, wherein the threshold number of encoded data elements is a number of encoded data elements of the plurality of encoded data elements sufficient to recover a first piece of data and further wherein the number of encoded data elements of the plurality of encoded data elements sufficient to recover the first piece of data is less than a total number of encoded data slices associated with the data object; and
decode the threshold number of encoded data elements retrieved to recover the data object, wherein the data object is divided into a set of separate pieces of data and wherein the set of separate pieces of data includes the piece of data.
12. The apparatus of claim 11 , wherein the at least one processor also retrieves and decodes another threshold number of encoded data elements to produce a second piece of data and combines the second piece of data with the first piece of data to recover the data.
13. The apparatus of claim 11 , wherein the at least one processor selects the set of storage units from the plurality of storage units to retrieve the threshold number of encoded data elements, wherein the set of storage units includes the storage unit with the metadata.
14. The apparatus of claim 11 , wherein systematic decoding is utilized to decode the threshold number of encoded data elements.
15. The apparatus of claim 11 , wherein non-systematic decoding is utilized to decode the threshold number of encoded data elements.
16. The apparatus of claim 11 , wherein a hybrid combination of systematic decoding and non-systematic decoding is utilized to decode the threshold number of encoded data elements.
17. The apparatus of claim 11 , wherein the metadata includes information pertaining to relationship among different encoded data elements of the plurality of encoded data elements.
18. The apparatus of claim 11 , wherein the one or more transforms collectively form a matrix.
19. The apparatus of claim 18 , wherein the one or more transforms includes information sufficient to identify at least one of an encryption function, a sequencing function, and parameters that have changed, in the generation of the plurality of encoded data elements.
20. The apparatus of claim 11 , wherein the metadata includes one or more information pertaining to:
data geometries;
data relationships;
data nesting;
structure;
coding structure;
data alignment;
size and length of data; and
data hash.
21. A computer-readable medium comprising instructions executable by a processor and when executed, instructing the processor to:
transmit a retrieval request for a plurality of encoded data elements to a plurality of storage units;
wherein each encoded data element of the plurality of encoded data elements is associated with a data object;
retrieve metadata from a storage unit of the plurality of storage units, wherein the metadata includes information sufficient to determine one or more transforms of a set of transforms used to create each encoded data element of the plurality of encoded data elements, wherein each transform of the set of transforms is different from other transforms in the set of transforms; and
based on the information regarding the plurality of encoded data elements, perform one or more of:
retrieve a threshold number of encoded data elements of the plurality of encoded data elements from a set of storage units of the plurality of storage units, wherein the threshold number of encoded data elements is a number of encoded data elements of the plurality of encoded data elements sufficient to recover a first piece of data and further wherein the number of encoded data elements of the plurality of encoded data elements sufficient to recover the first piece of data is less than a total number of encoded data slices associated with the data object; and
decode the threshold number of encoded data elements retrieved to recover the data object, wherein the data object is divided into a set of separate pieces of data and wherein the set of separate pieces of data includes the piece of data.
22. The computer-readable medium of claim 21 , wherein the instructions instruct the processor to use at least one of systematic decoding, non-systematic decoding and a hybrid combination of systematic decoding and non-systematic decoding, to decode the threshold number of encoded data elements.