Dynamically configuring erasure code redundancy and distribution
View Patent ↗Example apparatus and methods monitor conditions in a tiered storage system. The conditions monitored may include the availability of different numbers and types of devices including an erasure code based object storage system. The conditions monitored may also include the availability and type of devices available to the erasure code based object storage system. A redundancy policy for storing an item using the erasure code based object storage system may be determined based on the conditions. Erasure codes associated with the item may then be stored in the erasure code based object storage system as controlled, at least in part, by the redundancy policy. The redundancy policy for the erasure codes may be updated dynamically in response to changing conditions on the tiered storage system.
1. A non-transitory computer-readable storage medium storing computer-executable instructions that when executed by a computer cause the computer to perform a method, the method comprising:
accessing status data concerning a storage device in a tiered archive system, where the status data identifies an availability of the storage device, a capacity of the storage device, or a reliability of the storage device, where the tiered archive system includes an erasure code based object store or a non-object storage device, where the tiered archive system includes a plurality of devices arranged in different tiers, where the different tiers provide different levels of redundancy at different costs, and where accessing the status data includes receiving the status data from the storage device using a push model without querying the device;
upon determining that the tiered archive system includes a threshold number of non-object storage devices:
identifying, based at least in part on the status data, a first N/M redundancy policy for storing a message using the erasure code based object store and the non-object storage device, where the first N/M redundancy policy identifies a minimum number N−M of erasure codes needed to reconstruct the message and a maximum number N of erasure codes to be generated for the message, M being less than N, M and N being integers; and
selectively causing a first erasure code associated with the message to be stored on the erasure code based object store, and selectively causing a second erasure code associated with the message to be stored on the non-object storage device;
upon determining that the tiered archive system includes less than the threshold number of non-object storage devices:
identifying, based at least in part on the status data, a second N/M redundancy policy for storing the message using the erasure code based object store, where the second N/M redundancy policy identifies a minimum number N−M of erasure codes needed to reconstruct the message and a maximum number N of erasure codes to be generated for the message, M being less than N, M and N being integers; and
selectively causing an erasure code associated with the message to be stored on the erasure code based object store.
2. The non-transitory computer-readable storage medium of claim 1 , where accessing the availability data includes querying the storage device.
3. The non-transitory computer-readable storage medium of claim 1 , the method comprising generating the erasure code.
4. The non-transitory computer-readable storage medium of claim 1 , where identifying the N/M redundancy policy is based, at least in part, a user defined rule.
5. The non-transitory computer-readable storage medium of claim 1 , where identifying the N/M redundancy policy is based, at least in part, on an automated rule.
6. The non-transitory computer-readable storage medium of claim 5 , where the automated rule identifies a minimum number of erasure codes to be present on a minimum number of different devices associated with the erasure code based object store.
7. The non-transitory computer-readable storage medium of claim 1 , the method comprising:
accessing updated status data concerning the storage device, where accessing the updated status data includes receiving the updated status data from the storage device using a push model without querying the device;
identifying, based at least in part on the updated status data, an updated N/M redundancy policy;
identifying, based at least in part on the updated status data and on the updated N/M redundancy policy, an updated object store device to store an erasure code associated with the message, and
selectively causing the erasure code to be stored on the updated object store device according to the updated N/M redundancy policy.
8. The non-transitory computer-readable storage medium of claim 7 , the method comprising:
accessing the updated status data periodically,
accessing the updated status data upon detecting a changed condition in the tiered distributed storage system, or
accessing the updated status data as controlled by a control signal.