Heat indices for file systems and block storage
View Patent ↗Techniques and mechanisms are provided to allow for selective optimization, including deduplication and/or compression, of portions of files and data blocks. Data access is monitored to generate a heat index for identifying sections of files and volumes that are frequently and infrequently accessed. These frequently used portions may be left non-optimized to reduce or eliminate optimization I/O overhead. Infrequently accessed portions can be more aggressively optimized.
1. A method, comprising:
monitoring access including read access to a plurality of portions of a first file;
determining whether a first portion of the first file as less frequently accessed than a second portion of the first file;
receiving a request to optimize the first file;
based on a determination that the first portion of the first file is less frequently accessed than the second portion of the first file, performing optimization on the first portion of the first file more aggressively than on the second portion of the first file to achieve a higher reduction ratio on the first portion of the first file than on the second portion of the first file, wherein optimization comprises deduplication of data in the first file.
2. The method of claim 1 , wherein access comprises processor read and write access.
3. The method of claim 1 , wherein monitoring access to a plurality of portions comprises maintaining access time information.
4. The method of claim 1 , wherein monitoring access comprises maintaining a heat index.
5. The method of claim 1 , wherein optimization further comprises compression.
6. The method of claim 1 , wherein the second portion of the first file is left unoptimized.
7. The method of claim 1 , wherein a third portion of the first file is accessed more frequently than the second portion of the first file.
8. The method of claim 1 , wherein an optimization scale is applied to determine the aggressiveness of optimization based on the frequency of access.
9. The method of claim 1 , wherein a head index is used to determine the aggressiveness of optimization based on the frequency of access.
10. The method of claim 1 , wherein more aggressive optimization results in smaller file sizes.
11. An apparatus, comprising:
a processor configured to monitor access including read access to a plurality of portions of a first file and determine whether a first portion of the first file as less frequently accessed than a second portion of the first file;
an interface configured to receive a request to optimize the first file;
wherein, based on a determination that the first portion of the first file is less frequently accessed than the second portion of the first file, optimization is performed on the first portion of the first file more aggressively than on the second portion of the first file to achieve a higher reduction ratio on the first portion of the first file than on the second portion of the first file, wherein optimization comprises deduplication of data in the first file.
12. The apparatus of claim 11 , wherein access comprises processor read and write access.
13. The apparatus of claim 11 , wherein monitoring access to a plurality of portions comprises maintaining access time information.
14. The apparatus of claim 11 , wherein monitoring access comprises maintaining a heat index.
15. The apparatus of claim 11 , wherein optimization further comprises compression.
16. The apparatus of claim 11 , wherein the second portion of the first file is left unoptimized.
17. The apparatus of claim 11 , wherein a third portion of the first file is accessed more frequently than the second portion of the first file.
18. The apparatus of claim 11 , wherein an optimization scale is applied to determine the aggressiveness of optimization based on the frequency of access.
19. The apparatus of claim 11 , wherein a head index is used to determine the aggressiveness of optimization based on the frequency of access.
20. A non-transitory computer readable medium, comprising:
computer code for monitoring access including read access to a plurality of portions of a first file;
computer code for determining whether a first portion of the first file as less frequently accessed than a second portion of the first file;
computer code for receiving a request to optimize the first file;
computer code for performing, based on a determination that the first portion of the first file is less frequently accessed than the second portion of the first file, optimization on the first portion of the first file more aggressively than on the second portion of the first file to achieve a higher reduction ratio on the first portion of the first file than on the second portion of the first file, wherein optimization comprises deduplication of data in the first file.