Multiple erasure codes for distributed storage
Embodiments relate to a system with multiple erasure codes, and selecting and encoding for a write file with one of the codes to mitigate costs associated with storage recovery. The codes include a fast recovery code for frequently accessed data and a higher storage efficiency code for less frequently accessed data. State data is tracked to ascertain frequency of access to the file. One of the erasure codes is dynamically selected based on the tracked data, with the focus of the code select to lower recovery costs, and the data is encoded with the selected erasure code. Accordingly, the original coding of the write file is subject to change based on the tracked state data.
1. A method for dynamically selecting an erasure code in a storage system, comprising:
tracking a state of data access;
adaptively selecting between at least two erasure codes based on the tracked data access state, including a fast recovery code for frequently accessed data and a higher storage efficiency code for less frequently accessed data;
dynamically converting between the at least two erasure codes based on a global state;
dynamically converting a set of one or more fast erasure coded files to a compact erasure code responsive to the global state exceeding a threshold or the tracked data access state demonstrating a decrease; and
encoding the data with the selected erasure code.
2. The method of claim 1 , further comprising dynamically converting between the at least two erasure codes based on the tracked data state.
3. The method of claim 1 , wherein the global state includes system storage capacity.
4. The method of claim 1 , further comprising dynamically converting a set of one or more compact encoded data files to a fast erasure code responsive to the tracked data access state demonstrating an increase in data access and the global state being below a threshold.
5. The method of claim 1 , wherein the data access state includes data selected from the group consisting of: last modification time, size, read counts, coding state, and combinations thereof.
6. The method of claim 1 , wherein the two erasure codes may be employed with a product code and a local reconstruction code family.
7. A computer program product for dynamically selecting an erasure code in a storage system, the computer program product comprising a computer readable storage device having program code embodied therewith, the program code executable by a processing unit to:
track a state of data access;
adaptively selecting between at least two erasure codes based on the tracked data access state, including a fast recovery code for frequently accessed data and a higher storage efficiency code for less frequently accessed data;
dynamically convert between the at least two erasure codes based on a global state;
dynamically convert a set of one or more fast erasure coded files to a compact erasure code responsive to the global state exceeding a threshold or the tracked data access state demonstrating a decrease; and
encode the data with the selected erasure code.
8. The computer program product of claim 7 , further comprising program code to dynamically convert between the at least two erasure codes based on the tracked data state.
9. The computer program product of claim 7 , wherein the global state includes system storage capacity.
10. The computer program product of claim 7 , further comprising program code to dynamically convert a set of one or more compact encoded data files to a fast erasure code responsive to the tracked data access state demonstrating an increase in data access and the global state being below a threshold.
11. The computer program product of claim 7 , further comprising program code to select an initial erasure code for data blocks based on a file state and encode the data blocks with the initial erasure code, and convert the encoded data blocks to a second erasure code different from the initial erasure code, wherein the conversion is based on both the file state and a global state.
12. The computer program product of claim 7 , wherein the data access state includes data selected from the group consisting of: last modification time, size, read counts, coding state, and combinations thereof.
13. The computer program product of claim 7 , wherein the two erasure codes may be employed with a product code and a local reconstruction code family.
14. A computer system comprising:
a processor in communication with data storage;
a tool in communication with the processor to dynamically select an erasure code, including:
a system states module to track a state of data access;
an adaptive coding module to adaptively select between at least two erasure codes based on the data access state tracked by the system states module, including a fast recovery code for frequently accessed data and a higher storage efficiency code for less frequently accessed data, the adaptive coding module to dynamically convert between the at least two erasure code based on a global state and dynamically convert a set of one or more fast erasure coded files to a compact erasure code responsive to the system states module having a global state exceeding a threshold or the tracked data access state demonstrating a decrease; and
a coding module to encode the data with the erasure code selected by the adaptive coding module.
15. The system of claim 14 , further comprising the adaptive coding module to dynamically convert between at least two erasure codes based on the tracked data state.
16. The system of claim 14 , wherein the global state includes system storage capacity.
17. The system of claim 14 , further comprising the adaptive coding module to dynamically convert a set of one or more compact encoded data files to a fast erasure code responsive to the system states module demonstrating an increase in data access and the global state being below a threshold.