Distributed object storage system with dynamic spreading
Systems and methods for distributed storage systems using dynamic spreading policies are described. Distributed storage elements may be accessed using various storage paths through hierarchically organized storage system components to store data blocks corresponding to data elements. A higher priority hierarchical spreading policy is selected for determining the storage elements to receive the data blocks. If the first hierarchical spreading policy is determined not to have been met, a lower priority hierarchical spreading policy is selected and the data blocks are stored according to the lower priority hierarchical spreading policy. Data block stored at the lower priority hierarchical spreading policy may automatically be migrated to the higher priority hierarchical spreading policy.
1. A storage system, comprising:
a controller configured to:
access a plurality of storage elements using a plurality of storage paths through a plurality of hierarchical system components;
select, for a selected data element, a first hierarchical spreading policy from a plurality of hierarchical spreading policies for determining a subset of storage elements from the plurality of storage elements to store a plurality of data blocks corresponding to the selected data element;
determine, for the selected data element, that the first hierarchical spreading policy has not been met;
select, responsive to determining that the first hierarchical spreading policy has not been met, a second hierarchical spreading policy from the plurality of hierarchical spreading policies; and
store, using the second hierarchical spreading policy, the plurality of data blocks corresponding to the selected data element to a selected subset of storage elements from the plurality of storage elements.
2. The storage system of claim 1 , further comprising:
an encoder configured to disassemble a data object into a predetermined number of redundant data blocks;
a block spreader configured to distribute, using the second hierarchical spreading policy, the predetermined number of redundant data blocks to the plurality of storage elements;
a block clusterer configured to retrieve a subset of the predetermined number of redundant data blocks from the plurality of storage elements; and
a decoder configured to assemble the data object from the subset of the predetermined number of redundant data blocks.
3. The storage system of claim 1 , further comprising:
a storage policy engine configured to:
store the plurality of hierarchical spreading policies; and
assign a priority order for the plurality of hierarchical spreading policies, wherein the first hierarchical spreading policy has a higher priority order than the second hierarchical spreading policy.
4. The storage system of claim 3 , wherein:
the first hierarchical spreading policy corresponds to availability of the plurality of hierarchical system components at each hierarchy level of a plurality of hierarchy levels; and
the second hierarchical spreading policy corresponds to downgraded availability of the plurality of hierarchical system components at a selected hierarchy level of the plurality of hierarchy levels.
5. The storage system of claim 4 , wherein the plurality of hierarchical system components comprises at least two types of system components selected from:
a storage node;
a storage rack;
a storage cluster;
a storage site;
a network switch;
a network;
a subnetwork; and
a storage system.
6. The storage system of claim 3 , wherein:
the first hierarchical spreading policy corresponds to a percentage of available storage elements along a first set of storage paths from the plurality of storage paths;
the second hierarchical spreading policy corresponds to a percentage of available storage elements along a second set of storage paths from the plurality of storage paths; and
the second set of storage paths includes fewer storage paths than the first set of storage paths.
7. The storage system of claim 1 , further comprising:
a storage manager configured to:
attempt to store the plurality of data blocks to the subset of storage elements selected from the plurality of storage elements; and
determine a quantity of failed attempts to store the plurality of data blocks using the first hierarchical spreading policy, wherein the controller determines that the first hierarchical spreading policy has not been met based on the quantity of failed attempts meeting a policy failure threshold.
8. The storage system of claim 1 , further comprising:
a blacklist engine configured to identify blacklisted storage elements among the plurality of storage elements, wherein the controller determines that the first hierarchical spreading policy has not been met based on the blacklisted storage elements reducing the plurality of storage elements available below an availability threshold for the first hierarchical spreading policy.
9. The storage system of claim 8 , wherein:
the blacklist engine is further configured to identify blacklisted storage elements based on failure of hierarchical system components; and
the controller determines that the first hierarchical spreading policy has not been met based on failure of at least one hierarchical system component reducing the plurality of hierarchical system components available at a selected hierarchy level below a hierarchical availability threshold for the first hierarchical spreading policy.
10. The storage system of claim 1 , wherein the controller is further configured to:
disable, based on an operating condition, the first hierarchical spreading policy;
store, for each data element stored using the second hierarchical spreading policy, reduced spreading identifiers in metadata;
identify each data element with the reduced spreading identifier; and
migrate, using the first hierarchical spreading policy, each identified data element to an expanded subset of storage elements from the plurality of storage elements.
11. A computer-implemented method, comprising:
accessing a plurality of storage elements using a plurality of storage paths through a plurality of hierarchical system components;
selecting, for a selected data element, a first hierarchical spreading policy from a plurality of hierarchical spreading policies for determining a subset of storage elements from the plurality of storage elements to store a plurality of data blocks corresponding to the selected data element;
determining, for the selected data element, that the first hierarchical spreading policy has not been met;
selecting, responsive to determining that the first hierarchical spreading policy has not been met, a second hierarchical spreading policy from the plurality of hierarchical spreading policies; and
storing, using the second hierarchical spreading policy, the plurality of data blocks corresponding to the selected data element to a selected subset of storage elements from the plurality of storage elements.
12. The computer-implemented method of claim 11 , further comprising:
storing the plurality of hierarchical spreading policies; and
assigning a priority order for the plurality of hierarchical spreading policies, wherein the first hierarchical spreading policy has a higher priority order than the second hierarchical spreading policy.
13. The computer-implemented method of claim 12 , wherein:
the first hierarchical spreading policy corresponds to availability of the plurality of hierarchical system components at each hierarchy level of a plurality of hierarchy levels; and
the second hierarchical spreading policy corresponds to downgraded availability of the plurality of hierarchical system components at a selected hierarchy level of the plurality of hierarchy levels.
14. The computer-implemented method of claim 13 , wherein the plurality of hierarchical system components comprises at least two types of system components selected from:
a storage node;
a storage rack;
a storage cluster;
a storage site;
a network switch;
a network;
a subnetwork; and
a storage system.
15. The computer-implemented method of claim 12 , wherein:
the first hierarchical spreading policy corresponds to a percentage of available storage elements along a first set of storage paths from the plurality of storage paths;
the second hierarchical spreading policy corresponds to a percentage of available storage elements along a second set of storage paths from the plurality of storage paths; and
the second set of storage paths includes fewer storage paths than the first set of storage paths.
16. The computer-implemented method of claim 11 , further comprising:
attempting to store the plurality of data blocks to the subset of storage elements selected from the plurality of storage elements; and
determining a quantity of failed attempts to store the plurality of data blocks using the first hierarchical spreading policy, wherein determining that the first hierarchical spreading policy has not been met is based on the quantity of failed attempts meeting a policy failure threshold.
17. The computer-implemented method of claim 11 , further comprising:
identifying blacklisted storage elements among the plurality of storage elements, wherein determining that the first hierarchical spreading policy has not been met is based on the blacklisted storage elements reducing the plurality of storage elements available below an availability threshold for the first hierarchical spreading policy.
18. The computer-implemented method of claim 11 , further comprising:
identifying blacklisted storage elements among the plurality of storage elements based on failure of hierarchical system components, wherein determining that the first hierarchical spreading policy has not been met is based on failure of at least one hierarchical system component reducing the plurality of hierarchical system components available at a selected hierarchy level below a hierarchical availability threshold for the first hierarchical spreading policy.
19. The computer-implemented method of claim 11 , further comprising:
disabling, based on an operating condition, the first hierarchical spreading policy;
storing, for each data element stored using the second hierarchical spreading policy, reduced spreading identifiers in metadata;
identifying each data element with the reduced spreading identifiers; and
migrating, using the first hierarchical spreading policy, each identified data element to an expanded subset of storage elements from the plurality of storage elements.
20. A system, comprising:
a plurality of storage elements configured to store a plurality of data blocks;
a plurality of hierarchical system components configured to define a plurality of storage paths for accessing the plurality of storage elements;
means for accessing the plurality of storage elements using the plurality of storage paths through the plurality of hierarchical system components;
means for selecting, for a selected data element, a first hierarchical spreading policy from a plurality of hierarchical spreading policies for determining a subset of storage elements from the plurality of storage elements to store a plurality of data blocks corresponding to the selected data element;
means for determining, for the selected data element, that the first hierarchical spreading policy has not been met;
means for selecting, responsive to determining that the first hierarchical spreading policy has not been met, a second hierarchical spreading policy from the plurality of hierarchical spreading policies;
means for storing, using the second hierarchical spreading policy, the plurality of data blocks corresponding to the selected data element to a selected subset of storage elements from the plurality of storage elements; and
means for migrating, using the first hierarchical spreading policy, the plurality of data blocks stored using the second hierarchical spreading policy.