Data deduplication with adaptive erasure code redundancy
Example apparatus and methods combine erasure coding with data deduplication to simultaneously reduce the overall redundancy in data while increasing the redundancy of unique data. In one embodiment, an efficient representation of a data set is produced by deduplication. The efficient representation reduces duplicate data in the data set. Redundancy is then added back into the data set using erasure coding. The redundancy that is added back in adds protection to the unique data associated with the efficient representation. How much redundancy is added back in and what type of redundancy is added back in may be controlled based on an attribute (e.g., value, reference count, symbol size, number of symbols) of the unique data. Decisions concerning how much and what type of redundancy to add back in may be adapted over time based, for example, on observations of the efficiency of the overall system.
1. A non-transitory computer-readable storage medium storing computer-executable instructions that when executed by a computer control the computer to perform a method, the method comprising:
generating, by a processor, a characterization of segments of a plurality of segments with failure probabilities associated with the segments that indicate a likelihood of failure of the segments;
parsing, by a processor, the plurality of segments to generate a plurality of original chunks;
deduplicating, by a processor, the plurality of original chunks to generate a plurality of unique chunks;
grouping, by a processor, the plurality of unique chunks to form grouped chunks;
encoding, by a processor, the grouped chunks to generate a desired number of erasure code symbols that satisfy the failure probabilities associated with the segments; and
selectively storing members of the desired number of erasure code symbols on a number of storage devices.
2. The non-transitory computer-readable storage medium of claim 1 , further comprising:
calculating, by a processor, attributes of unique chunks of the plurality of unique chunks based, at least in part, on the failure probabilities; and where the encoding is based, at least in part, on the attributes being constrained by the failure probabilities.
3. The non-transitory computer-readable storage medium of claim 2 , where importance is an attribute of the attributes, and where importance is calculated based on a reference count to the data in a unique chunk.
4. The non-transitory computer-readable storage medium of claim 2 , where the attributes are variable, and where the desired number of erasure code symbols changes when an attribute changes.
5. The non-transitory computer-readable storage medium of claim 2 , where the number of storage devices is based, at least in part, on the attributes of the unique chunks.
6. The non-transitory computer-readable storage medium of claim 1 , where an erasure code symbol represents a number of connections between at least one segment and an erasure code symbol.
7. The non-transitory computer-readable storage medium of claim 1 , where the encoding is based on information stored in a generator matrix, and the method further comprising:
identifying, by a processor, attributes of unique chunks of the plurality of unique chunks; and
manipulating, by a processor, the generator matrix to control, at least in part, a size or composition of an erasure code symbol of the desired number of erasure code symbols.
8. A method comprising:
generating, by a processor, a characterization of segments of a plurality of segments with failure probabilities associated with the segments that indicate a likelihood of failure of the segments;
parsing, by a processor, the plurality of segments to generate a plurality of original chunks,
deduplicating, by a processor, the plurality of original chunks to generate a plurality of unique chunks;
calculating, by a processor, attributes of unique chunks of the plurality of unique chunks based, at least in part, on the failure probabilities associated with corresponding segments;
grouping, by a processor, the plurality of unique chunks to form a grouped chunk;
encoding, by a processor, the grouped chunk to generate a desired number of erasure code symbols based, at least in part, on the attributes being constrained by the failure probabilities; and
selectively storing members of the desired number of erasure code symbols on a number of storage devices, where the number of storage devices is based, at least in part, on the attributes for the unique chunks.
9. The method of claim 8 , where the attributes include an importance of a unique chunk, a size of the unique chunk, or an age of the unique chunk.
10. The method of claim 8 , where importance is an attribute, and where importance is calculated, by a processor, based on a reference count to the data in a unique chunk.
11. The method of claim 8 , where the attributes are variable, and where the desired number of erasure code symbols changes when an attribute changes.
12. The method of claim 8 , where the desired number of erasure code symbols is greater than or equal to a minimum threshold of erasure code symbols but less than the total possible erasure codes.
13. The method of claim 12 , where the desired number of erasure code symbols, the minimum threshold of erasure code symbols, and the total possible erasure codes are directly proportional to a variation of the attributes.
14. The method of claim 8 , where the encoding is based on information stored in a generator matrix, the method further comprising:
identifying, by a processor, attributes of unique chunks of the plurality unique chunks; and
manipulating, by a processor, the generator matrix to control, at least in part, a size or composition of an erasure code symbol of the desired number of erasure code symbols.
15. An apparatus, comprising:
a number of storage devices; and
a processor configured to:
generate a characterization of segments of a plurality of segments with failure probabilities associated with the segments that indicate a likelihood of failure of segments;
parse the plurality of segments to generate a plurality of original chunks,
deduplicate the plurality of original chunks to generate a plurality of unique chunks;
calculate attributes of unique chunks of the plurality of unique chunks based, at least in part, on the failure probabilities;
group the plurality of unique chunks to form at least one grouped chunk;
encode the at least one grouped chunk to generate a desired number of erasure code symbols that satisfy the failure probabilities associated with the segments, where the encoding is based, at least in part, on the attributes being constrained by the failure probabilities; and
selectively store members of the desired number of erasure code symbols on the number of storage devices, where the number of storage devices is based, at least in part, on the attributes for the unique chunks.
16. The apparatus of claim 15 , where importance is an attribute of the attributes, and where importance is calculated based on a reference count to the data in a unique chunk.
17. The apparatus of claim 15 , where the attributes are variable, and where the desired number of erasure code symbols changes when an attribute changes.
18. The apparatus of claim 15 , where an erasure code symbol represents a number of connections between at least one segment and an erasure code symbol.
19. The apparatus of claim 15 , where the encoding is based on information stored in a generator matrix, and where the processor is further configured to:
identify attributes of unique chunks of the plurality unique chunks; and
manipulate the generator matrix to control, at least in part, a size or composition of an erasure code symbol of the desired number of erasure code symbols.
20. The apparatus of claim 19 , where a non-zero entry in the generator matrix represents a node/edge probability distribution.