Handling data with different lifetime characteristics in stream-aware data storage equipment
Techniques for handling data with different lifetime characteristics in stream-aware data storage systems. The data storage systems can include a file system that has a log-based architecture design, and can employ one or more solid state drives (SSDs) that provide log-based data storage, which can include a data log divided into a series of storage segments. The techniques can be employed in the data storage systems to control the placement of data in the respective segments of the data log based at least on the lifetime of the data, significantly reducing the processing overhead associated with performing garbage collection functions within the SSDs.
1. A method of handling data with different lifetime characteristics in a data storage system, comprising:
dividing a data log into a plurality of data log divisions, the plurality of data log divisions including at least a first data log division, and a second data log division, each of the first data log division and the second data log division including a first storage segment and a second storage segment;
determining similar lifetimes of a plurality of groups of data blocks in at least one first incoming data flow, the plurality of groups of data blocks including at least a first group of data blocks determined to have shortest similar lifetimes, and a second group of data blocks determined to have mid-range similar lifetimes;
performing a first plurality of data log operations in the first data log division, the first plurality of data log operations including:
sequentially writing the first group of data blocks determined to have the shortest similar lifetimes as log structured data to the first storage segment of the first data log division, the first group of data blocks being modified in at least one second incoming data flow;
sequentially writing the modified first group of data blocks as log structured data to the second storage segment of the first data log division; and
in response to sequentially writing the modified first group of data blocks as log structured data to the second storage segment of the first data log division, invalidating each of the first group of data blocks written to the first storage segment of the first data log division, and erasing the invalidated data blocks from the first storage segment of the first data log division to make storage space within the first data log division available for reuse; and
performing a second plurality of data log operations in the second data log division, the second plurality of data log operations including:
sequentially writing the second group of data blocks determined to have the mid-range similar lifetimes as log structured data to the first storage segment of the second data log division, the second group of data blocks being modified in the at least one second incoming data flow;
sequentially writing the modified second group of data blocks as log structured data to the second storage segment of the second data log division; and
in response to sequentially writing the modified second group of data blocks as log structured data to the second storage segment of the second data log division, invalidating each of the second group of data blocks written to the first storage segment of the second data log division, and erasing the invalidated data blocks from the first storage segment of the second data log division to make storage space within the second data log division available for reuse.
2. The method of claim 1 wherein the groups of data blocks determined to have similar lifetimes include a third group determined to have longest similar lifetimes, and wherein the method further comprises:
monitoring lifetime characteristics of each data block in the first group, the second group, and the third group, the lifetime characteristics providing locality information for the data blocks in each of the first group, the second group, and the third group.
3. The method of claim 2 further comprising:
associating a stream identifier (ID) with each data block in the first group determined to have the shortest similar lifetimes, the second group determined to have the mid-range similar lifetimes, and the third group determined to have the longest similar lifetimes based on the monitored lifetime characteristics.
4. The method of claim 3 wherein the plurality of data log divisions includes a third data log division, the third data log division including a first storage segment, and wherein the method further comprises:
sequentially writing the third group of data blocks determined to have the longest similar lifetimes as log structured data to the first storage segment of the third data log division.
5. The method of claim 4 further comprising:
for each of the first group, the second group, and the third group sequentially written to the first storage segment of the first data log division, the first storage segment of the second data log division, and the first storage segment of the third data log division, respectively, storing metadata containing information about the data blocks, the information including the stream ID, the locality information, and a segment ID of the respective segment.
6. The method of claim 5 further comprising:
copying the third group determined to have the longest similar lifetimes to archival data storage.
7. The method of claim 5 further comprising:
having invalidated each of the second group of data blocks, combining or consolidating any valid data blocks remaining in the first storage segment of the second data log division;
copying the valid data blocks to a next available storage segment of the second data log division; and
erasing the invalidated data blocks from the first storage segment of the second data log division.
8. The method of claim 7 further comprising:
performing one of (i) comingling valid data blocks having the shorter similar lifetimes with other valid data blocks having the shorter similar lifetimes from one or more different streams, and (ii) comingling valid data blocks having the longer similar lifetimes with other valid data blocks having the longer similar lifetimes from the one or more different streams.
9. A data storage system, comprising:
a memory; and
processing circuitry configured to execute program instructions out of the memory to:
divide a data log into a plurality of data log divisions, the plurality of data log divisions including at least a first data log division, and a second data log, each of the first data log division and the second data log division including a first storage segment and a second storage segment;
determine similar lifetimes of a plurality of groups of data blocks in at least one first incoming data flow, the plurality of groups of data blocks including at least a first group of data blocks determined to have shortest similar lifetimes, and a second group of data blocks determined to have mid-range similar lifetimes;
perform a first plurality of data log operations in the first data log division, the first plurality of data log operations including:
sequentially writing the first group of data blocks determined to have the shortest similar lifetimes as log structured data to the first storage segment of the first data log division, the first group of data blocks being modified in at least one second incoming data flow;
sequentially writing the modified first group of data blocks as log structured data to the second storage segment of the first data log division; and
in response to sequentially writing the modified first group of data blocks as log structured data to the second storage segment of the first data log division, invalidating each of the first group of data blocks written to the first storage segment of the first data log division, and erasing the invalidated data blocks from the first storage segment of the first data log division to make storage space within the first data log division available for reuse; and
perform a second plurality of data log operations in the second data log division, the second plurality of data log operations including:
sequentially writing the second group of data blocks determined to have the mid-range similar lifetimes as log structured data to the first storage segment of the second data log division, the second group of data blocks being modified in the at least one second incoming data flow;
sequentially writing the modified second group of data blocks as log structured data to the second storage segment of the second data log division; and
in response to sequentially writing the modified second group of data blocks as log structured data to the second storage segment of the second data log division, invalidating each of the second group of data blocks written to the first storage segment of the second data log division, and erasing the invalidated data blocks from the first storage segment of the second data log division to make storage space within the second data log division available for reuse.
10. The data storage system of claim 9 wherein the groups of data blocks determined to have similar lifetimes include a third group determined to have longest similar lifetimes, and wherein the processing circuitry is further configured to execute the program instructions out of the memory to monitor lifetime characteristics of each data block in the first group, the second group, and the third group, wherein the lifetime characteristics provide locality information for the data blocks in each of the first group, the second group, and the third group.
11. The data storage system of claim 10 wherein the processing circuitry is further configured to execute the program instructions out of the memory to associate a stream identifier (ID) with each data block in the first group determined to have the shortest similar lifetimes, the second group determined to have the mid-range similar lifetimes, and the third group determined to have the longest similar lifetimes based on the monitored lifetime characteristics.
12. The data storage system of claim 11 wherein the plurality of data log divisions includes a third data log division, the third data log division including a first storage segment, and wherein the processing circuitry is further configured to execute the program instructions out of the memory to sequentially write the third group of data blocks determined to have the longest similar lifetimes as log structured data to the first storage segment of the third data log division.
13. The data storage system of claim 12 wherein the processing circuitry is further configured to execute the program instructions out of the memory, for each of the first group, the second group, and the third group sequentially written to the first storage segment of the first data log division, the first storage segment of the second data log division, and the first storage segment of the third data log division, respectively, to store metadata containing information about the data blocks, the information including the stream ID, the locality information, and a segment ID of the respective segment.
14. The data storage system of claim 13 wherein the processing circuitry is further configured to execute the program instructions out of the memory to copy the third group determined to have the longest similar lifetimes to archival data storage.
15. The data storage system of claim 13 wherein the processing circuitry is further configured to execute the program instructions out of the memory to:
having invalidated each of the second group of data blocks, combine or consolidate any valid data blocks remaining in the first storage segment of the second data log division;
copy the valid data blocks to a next available storage segment of the second data log division; and
erase the invalidated data blocks from the first storage segment of the second data log division.
16. The data storage system of claim 15 wherein the processing circuitry is further configured to execute the program instructions out of the memory to perform one of (i) comingling valid data blocks having the shorter similar lifetimes with other valid data blocks having the shorter similar lifetimes from one or more different streams, and (ii) comingling valid data blocks having the longer similar lifetimes with other valid data blocks having the longer similar lifetimes from the one or more different streams.
17. A computer program product including a set of non-transitory, computer-readable media having instructions that, when executed by control circuitry of a computerized apparatus, cause the control circuitry to perform a method of handling data with different lifetime characteristics in a data storage system, the method comprising:
dividing a data log into a plurality of data log divisions, the plurality of data log divisions including at least a first data log division, and a second data log, each of the first data log division and the second data log division including a first storage segment and a second storage segment;
determining similar lifetimes of a plurality of groups of data blocks in at least one first incoming data flow, the plurality of groups of data blocks including at least a first group of data blocks determined to have shortest similar lifetimes, and a second group of data blocks determined to have mid-range similar lifetimes;
performing a first plurality of data log operations in the first data log division, the first plurality of data log operations including:
sequentially writing the first group of data blocks determined to have the shortest similar lifetimes as log structured data to the first storage segment of the first data log division, the first group of data blocks being modified in at least one second incoming data flow;
sequentially writing the modified first group of data blocks as log structured data to the second storage segment of the first data log division; and
in response to sequentially writing the modified first group of data blocks as log structured data to the second storage segment of the first data log division, invalidating each of the first group of data blocks written to the first storage segment of the first data log division, and erasing the invalidated data blocks from the first storage segment of the first data log division to make storage space within the first data log division available for reuse; and
performing a second plurality of data log operations in the second data log division, the second plurality of data log operations including:
sequentially writing the second group of data blocks determined to have the mid-range similar lifetimes as log structured data to the first storage segment of the second data log division, the second group of data blocks being modified in the at least one second incoming data flow;
sequentially writing the modified second group of data blocks as log structured data to the second storage segment of the second data log division; and
in response to sequentially writing the modified second group of data blocks as log structured data to the second storage segment of the second data log division, invalidating each of the second group of data blocks written to the first storage segment of the second data log division, and erasing the invalidated data blocks from the first storage segment of the second data log division to make storage space within the second data log division available for reuse.
18. The computer program product of claim 17 wherein the groups of data blocks determined to have similar lifetimes include a third group determined to have longest similar lifetimes, and wherein the method further comprises:
monitoring lifetime characteristics of each data block in the first group, the second group, and the third group, the lifetime characteristics providing locality information for the data blocks in each of the first group, the second group, and the third group.