Techniques for optimizing data reduction by understanding application data
Techniques for determining data reduction options may include: receiving first data reduction information regarding compression and deduplication of chunks of a data set; determining, in accordance with the first data reduction information for the data set, first settings denoting whether compression and deduplication are enabled or disabled for the data set; receiving, during a first time period when the first settings are effective, writes directed to the data set; receiving second data reduction information regarding compression and deduplication of chunks of the data set modified by writes during the first time period; and determining, in accordance with the second plurality of data reduction statistics for the data set, second settings denoting whether compression and deduplication are enabled or disabled for the data set. Data reduction information about the data set may include histograms regarding compressed sizes and detected duplications and may include data reduction information regarding different data set regions.
1. A method of determining data reduction options for data sets comprising:
receiving first data reduction information for a data set, wherein the first data reduction information includes first compression information regarding compression of chunks of the data set, wherein the first data reduction information includes first deduplication information regarding deduplication of chunks of the data set;
determining, in accordance with the first data reduction information for the data set, first settings denoting whether compression is enabled or disabled for the data set and denoting whether deduplication is enabled or disabled for the data set;
receiving, during a first time period when the first settings are effective, a plurality of writes directed to the data set, wherein the plurality of writes modify a plurality of chunks of the data set;
receiving second data reduction information for the data set, wherein the second data reduction information includes second compression information regarding compression of the plurality of chunks of the data set modified by the plurality of writes during the first time period, wherein the second data reduction information includes second deduplication information regarding deduplication of the plurality of chunks of the data set modified by the plurality of writes during the first time period; and
determining, in accordance with the second data reduction information for the data set, second settings denoting whether compression is enabled or disabled for the data set and denoting whether deduplication is enabled or disabled for the data set.
2. The method of claim 1 , further comprising:
receiving, during a second time period when the second settings are effective, a second plurality of writes directed to the data set, wherein the second plurality of writes modify a second plurality of chunks of the data set;
receiving third data reduction information for the data set, wherein the third data reduction information includes third compression information regarding compression of the second plurality of chunks of the data set modified by the second plurality of writes during the second time period, wherein the third data reduction information includes third deduplication information regarding deduplication of the second plurality of chunks of the data set modified by the second plurality of writes during the second time period; and
determining, in accordance with the third data reduction information for the data set, third settings denoting whether compression is enabled or disabled for the data set and denoting whether deduplication is enabled or disabled for the data set.
3. The method of claim 1 , wherein the first compression information includes a histogram denoting frequencies of different compressed chunk sizes in the data set; and wherein the second compression information includes a histogram denoting frequencies of different compressed chunk sizes for the first plurality of chunks modified by the first plurality of writes during the first time period.
4. The method of claim 1 , wherein the first compression information identifies compressed chunk sizes for chunks at different logical locations or offsets in the data set, and wherein the second compression information includes identifies compressed chunk sizes for the first plurality of chunks modified by the first plurality of writes during the first time period.
5. The method of claim 1 , wherein the first deduplication information includes a histogram denoting frequencies of duplicates at different logical locations or offsets in the data set, and wherein the second deduplication information includes a histogram denoting frequencies of duplicates at different logical locations or offsets in the data set modified by the first plurality of writes during the first time period.
6. The method of claim 1 , wherein deduplication when enabled is performed as processing of an I/O path when writing to the data set.
7. The method of claim 1 , wherein compression when enabled is performed as processing of the I/O path when writing to the data set.
8. The method of claim 1 , wherein deduplication when enabled is not performed as processing of an I/O path when writing to the data set.
9. The method of claim 1 , wherein compression when enabled is not performed as processing of the I/O path when writing to the data set.
10. The method of claim 1 , further comprising:
displaying, on a user interface at a first point in time based on the first data reduction information, a diagram illustrating data reduction benefits for the data set;
providing, in accordance with the first data reduction information for the data set, recommended data reduction options for the data set; and
receiving, via the user interface, a first input identifying the first settings as current data reduction options selected for the data set.
11. The method of claim 10 , further comprising:
displaying, on the user interface at a second point in time based on the second data reduction information, a diagram illustrating revised data reduction benefits for the data set based on chunks modified by the first plurality of writes in the first time period
providing, in accordance with the second data reduction information for the data set, recommended data reduction options for the data set; and
receiving, via the user interface, a second input identifying the second settings as revised data reduction options selected for the data set.
12. The method of claim 1 , wherein the first settings indicate, for each of a plurality of data portions of the data set, whether compression is enabled or disabled individually for said each portion and whether deduplication is enabled or disabled individually for said each portion.
13. The method of claim 1 , wherein the data set is a first data set included as a first portion of a second data set larger than the first data set, and the method includes:
partitioning the second data set into a plurality of portions each having a different logical address subrange of locations in the second data set.
14. The method of claim 1 , further comprising:
taking a snapshot of the data set at the end of the first time period, wherein the snapshot is a point in time logical copy of the data set at the end of the first time period after the plurality of writes are issued to the data set; and
determining the second reduction information using the snapshot of the data set.
15. A system comprising:
a processor; and
a memory comprising code stored thereon that, when executed, perform a method of determining data reduction options for data sets comprising:
receiving first data reduction information for a data set, wherein the first data reduction information includes first compression information regarding compression of chunks of the data set, wherein the first data reduction information includes first deduplication information regarding deduplication of chunks of the data set;
determining, in accordance with the first data reduction information for the data set, first settings denoting whether compression is enabled or disabled for the data set and denoting whether deduplication is enabled or disabled for the data set;
receiving, during a first time period when the first settings are effective, a plurality of writes directed to the data set, wherein the plurality of writes modify a plurality of chunks of the data set;
receiving second data reduction information for the data set, wherein the second data reduction information includes second compression information regarding compression of the plurality of chunks of the data set modified by the plurality of writes during the first time period, wherein the second data reduction information includes second deduplication information regarding deduplication of the plurality of chunks of the data set modified by the plurality of writes during the first time period; and
determining, in accordance with the second data reduction information for the data set, second settings denoting whether compression is enabled or disabled for the data set and denoting whether deduplication is enabled or disabled for the data set.
16. A computer readable medium comprising code stored thereon that, when executed, performs a method of determining data reduction options for data sets comprising:
receiving first data reduction information for a data set, wherein the first data reduction information includes first compression information regarding compression of chunks of the data set, wherein the first data reduction information includes first deduplication information regarding deduplication of chunks of the data set;
determining, in accordance with the first data reduction information for the data set, first settings denoting whether compression is enabled or disabled for the data set and denoting whether deduplication is enabled or disabled for the data set;
receiving, during a first time period when the first settings are effective, a plurality of writes directed to the data set, wherein the plurality of writes modify a plurality of chunks of the data set;
receiving second data reduction information for the data set, wherein the second data reduction information includes second compression information regarding compression of the plurality of chunks of the data set modified by the plurality of writes during the first time period, wherein the second data reduction information includes second deduplication information regarding deduplication of the plurality of chunks of the data set modified by the plurality of writes during the first time period; and
determining, in accordance with the second data reduction information for the data set, second settings denoting whether compression is enabled or disabled for the data set and denoting whether deduplication is enabled or disabled for the data set.
17. The computer readable medium of claim 16 , wherein the method further comprises:
receiving, during a second time period when the second settings are effective, a second plurality of writes directed to the data set, wherein the second plurality of writes modify a second plurality of chunks of the data set;
receiving third data reduction information for the data set, wherein the third data reduction information includes third compression information regarding compression of the second plurality of chunks of the data set modified by the second plurality of writes during the second time period, wherein the third data reduction information includes third deduplication information regarding deduplication of the second plurality of chunks of the data set modified by the second plurality of writes during the second time period; and
determining, in accordance with the third data reduction information for the data set, third settings denoting whether compression is enabled or disabled for the data set and denoting whether deduplication is enabled or disabled for the data set.
18. The computer readable medium of claim 16 , wherein the first compression information includes a histogram denoting frequencies of different compressed chunk sizes in the data set; and wherein the second compression information includes a histogram denoting frequencies of different compressed chunk sizes for the first plurality of chunks modified by the first plurality of writes during the first time period.
19. The computer readable medium of claim 16 , wherein the first compression information identifies compressed chunk sizes for chunks at different logical locations or offsets in the data set, and wherein the second compression information includes identifies compressed chunk sizes for the first plurality of chunks modified by the first plurality of writes during the first time period.
20. The computer readable medium of claim 16 , wherein the first deduplication information includes a histogram denoting frequencies of duplicates at different logical locations or offsets in the data set, and wherein the second deduplication information includes a histogram denoting frequencies of duplicates at different logical locations or offsets in the data set modified by the first plurality of writes during the first time period.