Storage conditioning with intelligent rebuild
View Patent ↗Storage conditioning for a data storage system having D data storage devices (DSDs) is provided. E erasure codes (ECs) for an object are stored in the system, D>E. A map of d E-sized vectors of the D DSDs is produced. A DSD appears in e vectors. The ratio d/e is the reduced form of D/E. A hash value is produced for the object. A destination vector for storing the ECs is selected using the hash value according to a pre-determined, substantially uniform distribution. A compromised vector affected by a first DSD becoming unable to store ECs is identified. An intact vector that is not affected by the first DSD is identified. A complete set of ECs is produced from an incomplete set of ECs in the compromised vector and is distributed to the intact vector then copied back when the compromised vector is once again intact.
1. A computerized method performed in a computer storage system, comprising:
determining a number D of data storage devices (DSDs) in the computer storage system that are available to store protection data items produced by an automated durability process;
determining a number E of DSDs used by the automated durability process to protect data stored by the computer storage system;
upon determining that D is greater than E:
producing a reduced form d:e of a ratio D:E, where d and e are numbers;
producing a partitioning of the D DSDs, where the partitioning organizes the D DSDs into d sets of devices, where a member of the D DSDs appears in e of the d sets of devices;
associating a range of hash values with a member of the d sets of DSDs, where the range of hash values are associated with a hash function used to hash an item presented to a key/value store for storage;
receiving an object to be stored in the key/value store;
receiving a hash value for the object from the hash function and establishing the hash value as the key value for the key/value store;
receiving an E-sized set of protection data items for the object from the automated durability process;
selecting a first member of the d sets of DSDs to store the E-sized set of protection data items based, at least in part, on the hash value; and
storing the E-sized sets of protection data items on DSDs in the first member of the d sets of DSDs.
2. The computerized method of claim 1 , where the computer storage system is a single apparatus.
3. The computerized method of claim 1 , where the computer storage system is two or more apparatus.
4. The computerized method of claim 1 , where the computer storage system includes at least one storage device located in the cloud.
5. The computerized method of claim 1 , where the automated durability process runs in the computer storage system.
6. The computerized method of claim 1 , where the automated durability process runs at least partially in the cloud.
7. The computerized method of claim 1 , where the hash function runs in the computer storage system.
8. The computerized method of claim 1 , where the hash function runs at least partially in the cloud.
9. The computerized method of claim 1 , where the automated durability process is an erasure code (EC) process.
10. The computerized method of claim 1 , where the automated durability process is a redundant array of independent disks (RAID) process.
11. The computerized method of claim 1 , where the automated durability process is a replication process.
12. The computerized method of claim 1 , where storing the E-sized set of protection data items includes storing one member of the E-sized set of protection data items per DSD in the first member of the d sets of DSDs.
13. The computerized method of claim 1 , comprising:
upon detecting that a threshold number of DSDs in the D DSDs have become unavailable to store protection data items produced by the automated durability process:
identifying a source member of the d sets of DSDs having a greatest number of DSDs that are unavailable to store protection data items produced by the automated durability process;
receiving a complete reconstructed set of protection data items from the automated durability process, where the complete reconstructed set of protection data items is produced from an incomplete set of protection data items stored on DSDs that are available in the source member;
identifying a target member of the d sets of DSDs having a complete set of DSDs that are available to store protection data items produced by the automated durability process;
storing the complete reconstructed set of protection data items on DSDs in the target member; and
storing redirection data that controls a request for data from the source member to be redirected to the target member.
14. The computerized method of claim 13 , comprising:
upon detecting that the identified member of the D DSDs has become available to store protection data items produced by the automated durability process:
reading the complete reconstructed set of protection data items from the target member;
writing the complete reconstructed set of protection data items read from the target member to the source member; and
deactivating the redirection data.
15. The computerized method of claim 14 , comprising:
deleting the complete reconstructed set of protection data items from the target member.
16. The computerized method of claim 1 , where the hash function and the partitioning are configured to cause DSDs in the D DSDs to operate within 10% of an average utilization of the D DSDs.
17. The computerized method of claim 1 , where the automated durability process operates independent of the value for D.