User stream aware file systems with user stream detection
Techniques for handling multiple data streams in stream-aware data storage systems. The data storage systems can detect multiple sub-streams in an incoming stream of data, form a group of data blocks corresponding to each respective sub-stream, and associate, bind, and/or assign a stream ID to each data block in the respective sub-stream. The data storage systems can write each group of data blocks having the same stream ID to the same segment of a data log in one or more non-volatile storage devices, and manage and/or maintain, in persistent data storage, attribute information pertaining to the groups of data blocks in the respective sub-streams relative to time periods during which the respective groups of data blocks were written and/or received. The techniques can improve the detection of multiple sub-streams in an incoming stream of data, and improve the management of attribute information pertaining to data blocks in the respective sub-streams.
1. A method of handling multiple data streams in a stream-aware data storage system, comprising:
identifying one or more data sub-streams in an incoming data stream based on at least one attribute of data elements in the respective data sub-streams;
forming one or more groups of data elements from the respective data sub-streams;
writing the one or more groups of data elements as log structured data to one or more segments of a data log; and
maintaining, in persistent data storage, information pertaining to the at least one attribute of the data elements in the respective data sub-streams relative to time periods during which the respective groups of data elements were written.
2. The method of claim 1 wherein the identifying of one or more data sub-streams in the incoming data stream is based on one or more of a temporal locality of the data elements, a spatial locality of the data elements, a type of each data element, and a port number through which each data element is received.
3. The method of claim 1 further comprising:
associating a stream identifier (ID) to each data element in each data sub-stream.
4. The method of claim 3 wherein the writing of the one or more groups of data elements as log structured data to one or more segments of the data log includes writing each group of data elements having the same stream ID to the same segment of the data log.
5. The method of claim 1 further comprising:
generating a digest for each data element in each respective data sub-stream,
thereby generating a plurality of digests for a plurality of data elements, respectively, in the respective data sub-stream.
6. The method of claim 5 further comprising:
forming a group of digests from the plurality of digests; and
associating a stream identifier (ID) to each digest in the group of digests.
7. The method of claim 6 further comprising:
writing the group of digests as a data stream of log structured data to a segment of the data log.
8. The method of claim 7 further comprising:
maintaining, in persistent data storage, information pertaining to (i) the at least one attribute of the data elements in the respective data sub-stream, and (ii) the respective digests in the data stream, relative to a time period during which each of a respective group of data elements from the respective data sub-stream and the group of digests from the data stream were written to the data log.
9. A data storage system, comprising:
a memory; and
processing circuitry configured to execute program instructions out of the memory to:
identify one or more data sub-streams in an incoming data stream based on at least one attribute of data elements in the respective data sub-streams;
form one or more groups of data elements from the respective data sub-streams;
write the one or more groups of data elements as log structured data to one or more segments of a data log; and
maintain, in persistent data storage, information pertaining to the at least one attribute of the data elements in the respective data sub-streams relative to time periods during which the respective groups of data elements were written.
10. The data storage system of claim 9 wherein the processing circuitry is further configured to execute the program instructions out of the memory to identify one or more data sub-streams in the incoming data stream based on one or more of a temporal locality of the data elements, a spatial locality of the data elements, a type of each data element, and a port number through which each data element is received.
11. The data storage system of claim 9 wherein the processing circuitry is further configured to execute the program instructions out of the memory to associate a stream identifier (ID) to each data element in each data sub-stream.
12. The data storage system of claim 11 wherein the processing circuitry is further configured to execute the program instructions out of the memory to write each group of data elements having the same stream ID to the same segment of the data log.
13. The data storage system of claim 9 wherein the processing circuitry is further configured to execute the program instructions out of the memory to:
generate a digest for each data element in each respective data sub-stream,
thereby generating a plurality of digests for a plurality of data elements, respectively, in the respective data sub-stream.
14. The data storage system of claim 13 wherein the processing circuitry is further configured to execute the program instructions out of the memory to:
form a group of digests from the plurality of digests; and
associate a stream identifier (ID) to each digest in the group of digests.
15. The data storage system of claim 14 wherein the processing circuitry is further configured to execute the program instructions out of the memory to write the group of digests as a data stream of log structured data to a segment of the data log.
16. The data storage system of claim 15 wherein the processing circuitry is further configured to execute the program instructions out of the memory to maintain, in persistent data storage, information pertaining to (i) the at least one attribute of the data elements in the respective data sub-stream, and (ii) the respective digests in the data stream, relative to a time period during which each of a respective group of data elements from the respective data sub-stream and the group of digests from the data stream were written to the data log.
17. A computer program product including a set of non-transitory, computer-readable media having instructions that, when executed by control circuitry of a computerized apparatus, cause the control circuitry to perform a method of handling multiple data streams in a stream-aware data storage system, the method comprising:
identifying one or more data sub-streams in an incoming data stream based on at least one attribute of data elements in the respective data sub-streams;
forming one or more groups of data elements from the respective data sub-streams;
writing the one or more groups of data elements as log structured data to one or more segments of a data log; and
maintaining, in persistent data storage, information pertaining to the at least one attribute of the data elements in the respective data sub-streams relative to time periods during which the respective groups of data elements were written.
18. The computer program product of claim 17 wherein the method further comprises:
generating a digest for each data element in each respective data sub-stream, thereby generating a plurality of digests for a plurality of data elements, respectively, in the respective data sub-stream;
forming a group of digests from the plurality of digests;
associating a stream identifier (ID) to each digest in the group of digests; and
writing the group of digests as a data stream of log structured data to a segment of the data log.