Mass data storage system and method of operating thereof
There are provided a mass storage system comprising a control layer operatively coupled to a physical storage space and operable to interface with one or more clients and to present to said clients a plurality of logical volumes. The method of operating the storage system comprises: dividing one or more logical volumes into a plurality of statistical segments with predefined size; assigning to each given statistical segment a corresponding activity level characterizing statistics of I/O activity with regard to data portions within the given statistical segment, said statistics collected over a plurality of cycles of fixed counting length; and managing one or more data storage processes in the storage system (e.g. a background defragmentation process, a background garbage collection process, a destage management process, etc.) using said activity level.
1. A method of operating a storage system comprising a control layer operable to interface with one or more clients and to present to said clients a plurality of logical volumes, said control layer comprising a cache memory and further operatively coupled to a physical storage space comprising a plurality of disk drives, the method comprising:
dividing one or more logical volumes into a plurality of statistical segments with predefined size;
assigning to each given statistical segment a corresponding activity level characterizing statistics of I/O activity with regard to data portions within the given statistical segment, said statistics collected over a plurality of cycles of fixed counting length; and
managing one or more data storage processes in the storage system using said activity level.
2. The method of claim 1 wherein said activity level of a given statistical segment is characterized by an activity vector assigned to said given statistical segment, said activity vector characterized by at least one value obtained during a current cycle and by at least one value related to I/O statistics collected during at least one of the previous cycles.
3. The method of claim 1 wherein at least one of said one or more data storage processes is selected from a group comprising a background defragmentation process; a background garbage collection process; and a destage management process.
4. The method of claim 3 further comprising:
selecting one or more statistical segments with activity levels fitting a first selection criterion related a given data storage process; and
executing said given data storage process for one or more sets of data portions corresponding to the selected statistical segments, wherein said one or more sets of data portions fit a second selection criterion related a given data storage process.
5. The method of claim 1 further comprising:
selecting one or more statistical segments with activity levels fitting a first defragmentation criterion; and
providing defragmentation of one or more sets of data portions corresponding to the selected statistical segments, wherein said one or more sets of data portions fit a second defragmentation criterion.
6. The method of claim 5 further comprising:
reading to the cache memory said one or more sets of data portions to yield data portions to be defragmented;
consolidating data portions cached in the cache memory and corresponding to the same statistical segment into one or more consolidated write requests, wherein said data portions to be defragmented constitute at least part of data portions in the consolidated write requests; and
enabling writing said one or more consolidated write requests to one or more disk drives.
7. The method of claim 5 wherein data portions constituting a given set of data portions are selected from a group comprising all data portions within a given stripe in a RAID group, all data portions within two or more consecutive stripes in a RAID group, and all data portions within two or more logically-related stripes in one or more RAID groups.
8. The method of claim 6 wherein a given consolidated write request corresponds to a stripe in a RAID group.
9. The method of claim 6 wherein writing the consolidated write request is provided in a write-out-of-place technique.
10. The method of claim 5 wherein said activity level of a given statistical segment is characterized by an activity vector assigned to said given statistical segment, said activity vector characterized by at least one value obtained during a current cycle and by at least one value related to I/O statistics collected during at least one of the previous cycles, and wherein at least one of the values characterizing the activity vector is related to additional physical data space required resulting from the defragmentation of data portions within the statistical segment.
11. The method of claim 5 further comprising handling a constantly updated list of statistical segments matching said first defragmentation criterion.
12. The method of claim 1 further comprising:
selecting one or more statistical segments with activity levels fitting a first garbage collection criterion; and
providing re-arrangement of one or more sets of data portions corresponding to the selected statistical segments, wherein said re-arrangement enables releasing at least physical storage space corresponding to padded or outdated data portions within said one or more sets of data portions.
13. The method of claim 12 further comprising:
reading to the cache memory said one or more sets of data portions to yield data portions to be re-arranged;
consolidating data portions cached in the cache memory and corresponding to the selected statistical segments into one or more consolidated write requests, wherein said data portions to be re-arranged constitute at least part of data portions in the consolidated write requests;
enabling writing said one or more consolidated write requests to one or more disk drives in write-out-of-place technique; and
upon writing the consolidated write request, releasing the physical space corresponding to said one or more sets of data portions.
14. The method of claim 12 wherein data portions constituting a given set of data portions are selected from a group comprising all data portions within a given stripe in a RAID group, all data portions within two or more consecutive stripes in a RAID group, and all data portions within two or more logically-related stripes in one or more RAID groups, and wherein a given consolidated write request corresponds to a stripe in a RAID group.
15. The method of claim 1 wherein the cache memory is operable to handle dirty data by managing a replacement sequence, the method further comprising:
configuring the replacement sequence to comprise a destage pool operable to handle data portions to be used for further generation of one or more consolidated write requests;
dividing the destage pool into a plurality of destage bins;
handling data portions corresponding to segments with similar access patterns within a common destage bin, wherein two or more segments have similar access patterns if distance between respective activity levels matches a predefined criteria;
consolidating data portions cached in the cache memory and corresponding to the same destage bin into one or more consolidated write requests; and
enabling writing said one or more consolidated write requests to one or more disk drives.
16. The method of claim 15 wherein said pool is located at the bottom of the replacement sequence.
17. A storage system comprising a physical storage space comprising a plurality of disk drives and operatively coupled to a control layer operable to interface with one or more clients and to present to said clients a plurality of logical volumes, wherein said control layer comprises a cache memory and is further operable:
to divide one or more logical volumes into a plurality of statistical segments with predefined size;
to assign to each given statistical segment a corresponding activity level characterizing statistics of I/O activity with regard to data portions within the given statistical segment, said statistics collected over a plurality of cycles of fixed counting length; and
to manage one or more data storage processes in the storage system using said activity levels.
18. The storage system of claim 17 wherein said activity level of a given statistical segment is characterized by an activity vector assigned to said given statistical segment, said activity vector characterized by at least one value obtained during a current cycle and by at least one value related to I/O statistics collected during at least one of the previous cycles.
19. The storage system of claim 17 wherein at least one of said one or more data storage processes is selected from a group comprising a background defragmentation process; a background garbage collection process; and a destage management process.
20. The storage system of claim 19 wherein the storage control layer is further operable:
to select one or more statistical segments with activity levels fitting a first selection criterion related a given data storage process; and
to execute said given data storage process for one or more sets of data portions corresponding to the selected statistical segments, wherein said one or more sets of data portions fit a second selection criterion related a given data storage process.
21. The storage system of claim 18 wherein the storage control layer is further operable to select one or more statistical segments with activity levels fitting a first defragmentation criterion; and wherein the storage control layer further comprises a defragmentation module operatively coupled to the cache memory and operable to enable defragmentation of one or more sets of data portions corresponding to the selected statistical segments, wherein said one or more sets of data portions fit a second defragmentation criterion.
22. The storage system of claim 21 wherein the storage control layer is further operable to:
enable reading to the cache memory said one or more sets of data portions to yield data portions to be defragmented;
to consolidate data portions cached in the cache memory and corresponding to the same statistical segment into one or more consolidated write requests, wherein said data portions to be defragmented constitute at least part of data portions in the consolidated write requests; and
to enable writing said one or more consolidated write requests to one or more disk drives.
23. The storage system of claim 18 wherein said activity level of a given statistical segment is characterized by an activity vector assigned to said given statistical segment, said activity vector characterized by at least one value obtained during a current cycle and by at least one value related to I/O statistics collected during at least one of the previous cycles, and wherein at least one of the values characterizing the activity vector is related to additional physical data space required resulting from the defragmentation of data portions within the statistical segment.
24. The storage system of claim 21 wherein the storage control layer further comprises a defragmentation module, said defragmentation module is operable to handle a constantly updated list of statistical segments matching said first defragmentation criterion.
25. The storage system of claim 18 wherein the storage control layer is further operable to select one or more statistical segments with activity levels fitting a first garbage collection criterion; and to provide re-arrangement of one or more sets of data portions corresponding to the selected statistical segments, wherein said re-arrangement enables releasing at least physical storage space corresponding to padded or outdated data portions within said one or more sets of data portions.
26. The storage system of claim 25 wherein the storage control layer is further operable to read to the cache memory said one or more sets of data portions to yield data portions to be re-arranged;
to consolidate data portions cached in the cache memory and corresponding to the selected statistical segments into one or more consolidated write requests, wherein said data portions to be re-arranged constitute at least part of data portions in the consolidated write requests;
to enable writing said one or more consolidated write requests to one or more disk drives in write-out-of-place technique; and
upon writing the consolidated write request, to release the physical space corresponding to said one or more sets of data portions.
27. A computer program product comprising a non-transitory computer readable medium storing computer readable program code for a computer operating a storage system comprising a control layer is operable to interface with one or more clients and to present to said clients a plurality of logical volumes, said control layer comprising a cache memory and further operatively coupled to a physical storage space comprising a plurality of disk drives, the computer program product comprising:
computer readable program code for causing the computer to divide one or more logical volumes into a plurality of statistical segments with predefined size;
computer readable program code for causing the computer to assign to each given statistical segment a corresponding activity level characterizing statistics of I/O activity with regard to data portions within the given statistical segment, said statistics collected over a plurality of cycles of fixed counting length; and
computer readable program code for causing the computer to manage one or more data storage processes in the storage system using said activity levels.