Content-aware data recovery method for elastic cloud storage
A method for avoiding recovery and storage of useless data is disclosed. The operations comprise: determining that a subset of data fragments out of a plurality of data fragments have become unavailable, the plurality of data fragments forming a data chunk, wherein the plurality of data fragments are protected by additional redundancy data; determining whether all of data that were stored in the unavailable subset of data fragments when the unavailable subset of data fragments were available would be useless if recovered using remaining available data fragments of the data chunk and the redundancy data; and in response to determining that all of the data that were stored in the unavailable subset of data fragments would be useless if recovered, generating new redundancy data for protecting the remaining available data fragments of the data chunk without recovering the data that were stored in the unavailable subset of data fragments.
1. A method for avoiding recovery and storage of useless data, comprising:
determining that a subset of data fragments out of a plurality of data fragments have become unavailable, the plurality of data fragments forming a data chunk, wherein the plurality of data fragments are protected by additional redundancy data;
determining whether all of data that were stored in the unavailable subset of data fragments when the unavailable subset of data fragments were available would be useless if recovered using remaining available data fragments of the data chunk and the redundancy data; and
in response to determining that all of the data that were stored in the unavailable subset of data fragments would be useless if recovered, generating new redundancy data for protecting the remaining available data fragments of the data chunk without recovering the data that were stored in the unavailable subset of data fragments.
2. The method of claim 1 , wherein determining whether all of the data that were stored in the unavailable subset of data fragments would be useless comprises determining whether all user objects associated with the unavailable subset of data fragments have been deleted.
3. The method of claim 1 , wherein whether all of the data that were stored in the unavailable subset of data fragments would be useless is determined based on reference tables.
4. The method of claim 1 , wherein the new redundancy data are generated based on erasure coding.
5. The method of claim 1 , wherein the new redundancy data are generated in part with predetermined fake data, wherein the predetermined fake data is used whenever data from the unavailable subset of data fragments would be needed in the new redundancy data generation.
6. The method of claim 5 , wherein the predetermined fake data are not stored.
7. The method of claim 5 , wherein the predetermined fake data are one of: 1) all 0's, or 2) all 1's.
8. A non-transitory machine-readable medium having instructions stored therein which, when executed by a processor, cause the processor to perform data protection operations, the operations comprising:
determining that a subset of data fragments out of a plurality of data fragments have become unavailable, the plurality of data fragments forming a data chunk, wherein the plurality of data fragments are protected by additional redundancy data;
determining whether all of data that were stored in the unavailable subset of data fragments when the unavailable subset of data fragments were available would be useless if recovered using remaining available data fragments of the data chunk and the redundancy data; and
in response to determining that all of the data that were stored in the unavailable subset of data fragments would be useless if recovered, generating new redundancy data for protecting the remaining available data fragments of the data chunk without recovering the data that were stored in the unavailable subset of data fragments.
9. The non-transitory machine-readable medium of claim 8 , wherein determining whether all of the data that were stored in the unavailable subset of data fragments would be useless comprises determining whether all user objects associated with the unavailable subset of data fragments have been deleted.
10. The non-transitory machine-readable medium of claim 8 , wherein whether all of the data that were stored in the unavailable subset of data fragments would be useless is determined based on reference tables.
11. The non-transitory machine-readable medium of claim 8 , wherein the new redundancy data are generated based on erasure coding.
12. The non-transitory machine-readable medium of claim 8 , wherein the new redundancy data are generated in part with predetermined fake data, wherein the predetermined fake data is used whenever data from the unavailable subset of data fragments would be needed in the new redundancy data generation.
13. The non-transitory machine-readable medium of claim 12 , wherein the predetermined fake data are not stored.
14. The non-transitory machine-readable medium of claim 12 , wherein the predetermined fake data are one of: 1) all 0's, or 2) all 1's.
15. A data processing system, comprising:
a processor; and
a memory coupled to the processor storing instructions which, when executed by the processor, cause the processor to perform data protection operations, the operations including:
determining that a subset of data fragments out of a plurality of data fragments have become unavailable, the plurality of data fragments forming a data chunk, wherein the plurality of data fragments are protected by additional redundancy data;
determining whether all of data that were stored in the unavailable subset of data fragments when the unavailable subset of data fragments were available would be useless if recovered using remaining available data fragments of the data chunk and the redundancy data; and
in response to determining that all of the data that were stored in the unavailable subset of data fragments would be useless if recovered, generating new redundancy data for protecting the remaining available data fragments of the data chunk without recovering the data that were stored in the unavailable subset of data fragments.
16. The data processing system of claim 15 , wherein determining whether all of the data that were stored in the unavailable subset of data fragments would be useless comprises determining whether all user objects associated with the unavailable subset of data fragments have been deleted.
17. The data processing system of claim 15 , wherein whether all of the data that were stored in the unavailable subset of data fragments would be useless is determined based on reference tables.
18. The data processing system of claim 15 , wherein the new redundancy data are generated based on erasure coding.
19. The data processing system of claim 15 , wherein the new redundancy data are generated in part with predetermined fake data, wherein the predetermined fake data is used whenever data from the unavailable subset of data fragments would be needed in the new redundancy data generation.
20. The data processing system of claim 19 , wherein the predetermined fake data are not stored.
21. The data processing system of claim 19 , wherein the predetermined fake data are one of: 1) all 0's, or 2) all 1's.