Heat indices for file systems and block storage
Techniques and mechanisms are provided to allow for selective optimization, including deduplication and/or compression, of portions of files and data blocks. Data access is monitored to generate a heat index for identifying sections of files and volumes that are frequently and infrequently accessed. These frequently used portions may be left non-optimized to reduce or eliminate optimization I/O overhead. Infrequently accessed portions can be more aggressively optimized.
1. A method, comprising:
maintaining a heat index during optimization of each portion of a file;
determining that access to a first portion of the file corresponds to a low heat index;
determining that access to a second portion of the file corresponds to a high heat index; and
performing optimization on the first portion and the second portion of the file using an optimization scale based on the heat indices for each section during optimization, wherein the first portion of the file is optimized more than the second portion of the file.
2. The method of claim 1 , wherein access comprises processor read and write access.
3. The method of claim 1 , further comprising monitoring access to a plurality of portions comprises maintaining access time information.
4. The method of claim 1 , wherein a high heat index corresponds to a timestamp of access indicating a more recent access when compared to a timestamp of access corresponding to a low heat index.
5. The method of claim 1 , wherein optimization comprises compression.
6. The method of claim 1 , wherein the second portion of the file is left unoptimized.
7. The method of claim 1 , wherein a third portion of the file is accessed more frequently than the second portion of the file.
8. The method of claim 1 , wherein the optimization scale determines a reduction rate during optimization.
9. The method of claim 1 , wherein a high heat index corresponds to a higher frequency of access when compared with the frequency of access of a low heat index.
10. The method of claim 1 , wherein optimization comprises deduplication.
11. An apparatus, comprising:
memory; and
a processor configured to:
maintain a heat index in memory during optimization of each portion of a file;
determine that access to a first portion of the file corresponds to a low heat index;
determine that access to a second portion of the file corresponds to a high heat index; and
perform optimization on the first portion and the second portion of the file using an optimization scale based on the heat indices for each section during optimization, wherein the first portion of the file is optimized more than the second portion of the file.
12. The apparatus of claim 11 , wherein access comprises processor read and write access.
13. The apparatus of claim 11 , wherein the processor is further configured to monitor access to a plurality of portions comprises maintaining access time information.
14. The apparatus of claim 11 , wherein a high heat index corresponds to a timestamp of access indicating a more recent access when compared to a timestamp of access corresponding to a low heat index.
15. The apparatus of claim 11 , wherein optimization comprises deduplication or compression.
16. The apparatus of claim 11 , wherein the second portion of the file is left unoptimized.
17. The apparatus of claim 11 , wherein a third portion of the file is accessed more frequently than the second portion of the file.
18. The apparatus of claim 11 , wherein the optimization scale determines a reduction rate during optimization.
19. The apparatus of claim 11 , wherein a heat index is used to determine reduction rate during optimization based on the frequency of access.
20. A non-transitory computer readable medium comprising instructions for causing a processor and memory to execute a method, the method comprising:
maintaining a heat index during optimization of each portion of a file;
determining that access to a first portion of the file corresponds to a low heat index;
determining that access to a second portion of the file corresponds to a high heat index; and
performing optimization on the first portion and the second portion of the file using an optimization scale based on the heat indices for each section during optimization, wherein the first portion of the file is optimized more than the second portion of the file.