Reference set construction for data deduplication
By way of example, a data storage system may comprise, a non-transitory storage device storing data blocks in chunks, and a storage logic coupled to the non-transitory storage device that manages storage of data on the storage device. The storage logic is executable to receive a data stream for storage in a non-transitory storage device, the data stream including one or more data blocks, analyze the data stream to determine a domain, retrieve a pre-configured reference set based on the domain, and deduplicate the one or more data blocks of the data stream using the pre-configured reference set.
1. A data storage system, comprising:
a non-transitory storage device storing data blocks in chunks;
a processor coupled to the non-transitory storage device, the processor configured to:
receive a data stream for storage in the non-transitory storage device, the data stream including one or more data blocks;
analyze the data stream to determine a domain;
retrieve a pre-configured reference set based on the domain, the pre-configured reference set defined by a group of data blocks; and
deduplicate the one or more data blocks of the data stream using the pre-configured reference set by being further configured to:
encode, responsive to the one or more blocks in the data stream and the group of data blocks in the reference set meeting a similarity criterion, the one or more blocks based on the reference set,
aggregate, responsive to:
(a) the one or more blocks in the data stream and the group of data blocks in the reference set not meeting the similarity criterion, and
(b) the one or more blocks in the data stream having a threshold similarity to each other,
the one or more blocks into an aggregated set of data blocks comprising data blocks different from the data blocks in the pre-configured reference set, and
generate a new reference set using the set of data blocks.
2. The data storage system of claim 1 , wherein the processor is further configured to:
evaluate performance of the pre-configured reference set in deduplicating data blocks of one or more prior data streams;
assign a performance score to the pre-configured reference set based on the evaluated performance; and
generate a ranking for the pre-configured reference set for the domain relative to other reference sets associated with the domain based on the performance score, wherein retrieving the pre-configured reference set includes retrieving the pre-configured reference set based further on the ranking of the pre-configured reference set.
3. The data storage system of claim 2 , wherein the performance of the pre-configured reference set is evaluated based on a deduplication ratio.
4. The data storage system of claim 2 , wherein the performance of the pre-configured reference set is evaluated based on a deduplication speed.
5. The data storage system of claim 1 , wherein the domain is a range of physical area of the non-transitory storage device that hosts data blocks with similar characteristics.
6. The data storage system of claim 1 , wherein the processor is further configured to:
generate the pre-configured reference set as a synthetic reference set for data block sequences that repeat at a rate above a predetermined threshold;
identify a duplicated data block sequence;
retrieve the synthetic reference set based on the duplicated data block sequence; and
deduplicate the identified duplicated data block sequence using the synthetic reference set.
7. The data storage system of claim 1 , wherein the processor is further configured to:
assign a use count variable to the pre-configured reference set;
update the use count variable when the pre-configured reference set is used to deduplicate the data blocks; and
store the pre-configured reference set including the use count variable in the non-transitory storage device.
8. A method comprising:
receiving a data stream for storage in a non-transitory storage device, the data stream including one or more data blocks;
analyzing the data stream to determine a domain;
retrieving a pre-configured reference set based on the domain, the pre-configured reference set defined by a group of data blocks; and
deduplicating the one or more data blocks of the data stream using the pre-configured reference set, the deduplicating comprising:
encoding, responsive to the one or more blocks in the data stream and the group of data blocks in the reference set meeting a similarity criterion, the one or more blocks based on the reference set,
aggregating, responsive to:
(a) the one or more blocks in the data stream and the group of data blocks in the reference set not meeting the similarity criterion, and
(b) the one or more blocks in the data stream having a threshold similarity to each other,
the one or more blocks into an aggregated set of data blocks comprising data blocks different from the data blocks in the pre-configured reference set; and
generating a new reference set using the aggregated set of data blocks.
9. The method of claim 8 , further comprising:
evaluating performance of the pre-configured reference set in deduplicating data blocks of one or more prior data streams;
assigning a performance score to the pre-configured reference set based on the evaluated performance; and
generating a ranking for the pre-configured reference set for the domain relative to other reference sets associated with the domain based on the performance score, wherein retrieving the pre-configured reference set includes retrieving the pre-configured reference set based further on the ranking of the pre-configured reference set.
10. The method of claim 9 , wherein the performance of the pre-configured reference set is evaluated based on a deduplication ratio.
11. The method of claim 9 , wherein the performance of the pre-configured reference set is evaluated based on a deduplication speed.
12. The method of claim 8 , wherein the domain is a range of physical area of the non-transitory storage device that hosts data blocks with similar characteristics.
13. The method of claim 8 , further comprising:
generating the pre-configured reference set as a synthetic reference set for data block sequences that repeat at a rate above a predetermined threshold;
identifying a duplicated data block sequence;
retrieving the synthetic reference set based on the duplicated data block sequence; and
deduplicating the identified duplicated data block sequence using the synthetic reference set.
14. The method of claim 8 , further comprising:
assigning a use count variable to the pre-configured reference set;
updating the use count variable when the pre-configured reference set is used to deduplicate the data blocks; and
storing the pre-configured reference set including the use count variable in the non-transitory storage device.
15. A data storage system comprising:
means for receiving a data stream for storage in a non-transitory storage device, the data stream including one or more data blocks;
means for analyzing the data stream to determine a domain;
means for retrieving a pre-configured reference set based on the domain, the pre-configured reference set defined by a group of data blocks; and
means for deduplicating the one or more data blocks of the data stream using the pre-configured reference set, the means for deduplicating comprising:
means for encoding, responsive to the one or more blocks in the data stream and the group of data blocks in the reference set meeting a similarity criterion, the one or more blocks based on the reference set,
means for aggregating, responsive to:
(a) the one or more blocks in the data stream and the group of data blocks in the reference set not meeting the similarity criterion, and
(b) the one or more blocks in the data stream having a threshold similarity to each other,
the one or more blocks into an aggregated set of data blocks comprising data blocks different from the data blocks in the pre-configured reference set, and
means for generating a new reference set using the set of data blocks.
16. The data storage system of claim 15 , further comprising:
means for evaluating performance of the pre-configured reference set in deduplicating data blocks of one or more prior data streams;
means for assigning a performance score to the pre-configured reference set based on the evaluated performance; and
means for generating a ranking for the pre-configured reference set for the domain relative to other reference sets associated with the domain based on the performance score, wherein retrieving the pre-configured reference set includes retrieving the pre-configured reference set based further on the ranking of the pre-configured reference set.
17. The data storage system of claim 16 , wherein the performance of the pre-configured reference set is evaluated based on a deduplication ratio.
18. The data storage system of claim 16 , wherein the performance of the pre-configured reference set is evaluated based on a deduplication speed.
19. The data storage system of claim 15 , wherein the domain is a range of physical area of the non-transitory storage device that hosts data blocks with similar characteristics.