Data deduplication (dedup) management
Aspects of the present disclosure relate to data deduplication (dedup) techniques for storage arrays. At least one input/output (IO) operations in an IO workload received by a storage array can be identified. Each of the IOs can relate to a data track of the storage array. a probability of the at least one IO being similar to a previous stored IO can be determined. A data deduplication (dedup) operation can be performed on the at least one IO based on the probability. The probability can be less than one hundred percent (100%).
1. An apparatus having at least one processor configured to:
identify two or more input/output (IO) operations in an IO workload received by a storage array, wherein each IO relates to a data track of the storage array, wherein identifying the two or more IO operations includes identifying a set of sequential IOs in the IO workload, and wherein identifying the set of sequential IOs includes performing a time dimensional comparison between the IO workload's IOs and their corresponding data tracks, wherein the time dimensional comparison includes performing a time series comparison using the set of sequential IOs' corresponding receive timestamp and their related data tracks' corresponding write timestamps;
determine a probability of the at least one IO being similar to a previous stored IO; and
perform a data deduplication (dedup) operation on the at least one IO based on the probability, wherein the probability is less than one hundred (100) percent.
2. The apparatus of claim 1 further configured to:
generate one or more data track fingerprints representing data of each data track identified by the set of sequential IOs; and
store the one or more data track fingerprints in a searchable data structure.
3. The apparatus of claim 2 further configured to analyze a temporal dimension of each data track identified by the set of sequential IOs.
4. The apparatus of claim 3 further configured to convert a plurality of unique fingerprints representing the set of sequential IOs into a time series fingerprint, wherein each time-series fingerprint represents a time step representing each data track related to the set of sequential IOs.
5. The apparatus of claim 4 further configured to compare the time series fingerprints with corresponding previously stored set of sequential IOs' time series fingerprints.
6. The apparatus of claim 5 further configured to perform a comparison technique between at least a subset of tracks related to the time series fingerprints and a corresponding subset of tracks related to the previously stored set of sequential IOs' time series fingerprints.
7. The apparatus of claim 6 further configured to select a number of tracks to include in a subset track comparison based on a threshold statistical relevancy.
8. The apparatus of claim 6 further configured to determine the probability of the similarity between the at least one set of sequential IOs and the previously previous stored set of sequential IOs based on a number of matching tracks identified by the comparison technique.
9. The apparatus of claim 6 further configured to perform the comparison technique using a time series comparison system, wherein the time series comparison system includes dynamic time warp (DTW) processor instructions.
10. The apparatus of claim 8 further configured to perform the data dedup operation on the at least one set of sequential IOs based on an analysis of the similarity probability and a probability threshold.
11. A method comprising:
identifying two or more input/output (IO) operations in an IO workload received by a storage array, wherein each IO relates to a data track of the storage array, wherein identifying the two or more IO operations includes identifying a set of sequential IOs in the IO workload, and wherein identifying the set of sequential IOs includes performing a time dimensional comparison between the IO workload's IOs and their corresponding data tracks, wherein the time dimensional comparison includes performing a time series comparison using the set of sequential IOs' corresponding receive timestamp and their related data tracks' corresponding write timestamps;
determining a probability of the at least one IO being similar to a previous stored IO; and
performing a data deduplication (dedup) operation on the at least one IO based on the probability, wherein the probability is less than one hundred (100) percent.
12. The method of claim 11 further comprising:
generating one or more data track fingerprints representing the data of each data track identified by the set of sequential IOs; and
storing the one or more data track fingerprints in a searchable data structure.
13. The method of claim 12 further comprising analyzing a temporal dimension of each data track identified by the set of sequential IOs.
14. The method of claim 13 further comprising converting a plurality of unique fingerprints representing the set of sequential IOs into a time series fingerprint, wherein each time-series fingerprint represents a time step representing each data track related to the set of sequential IOs.
15. The method of claim 14 further comprising comparing the time series fingerprints with corresponding previously stored set of sequential IOs' time series fingerprints.
16. The method of claim 15 further comprising performing a comparison technique between at least a subset of tracks related to the time series fingerprint and a corresponding subset of tracks related to the previously stored set of sequential IOs' time series fingerprints.
17. The method of claim 16 further comprising selecting a number of tracks to include in a subset track comparison based on a threshold statistical relevancy.
18. The method of claim 16 further comprising determining the probability of the similarity between the at least one set of sequential IOs and the previously previous stored set of sequential IOs based on a number of matching tracks identified by the comparison technique.
19. The method of claim 16 further comprising performing the comparison technique using a time series comparison system, wherein the time series comparison system includes dynamic time warp (DTW) time series comparison algorithm.
20. The method of claim 18 further comprising performing the data dedup operation on the at least one set of sequential IOs based on an analysis of the similarity probability and a probability threshold.