Systems and methods for random to sequential storage mapping
Systems and methods for mapping random data writes to sequential or semi-sequential storage. Data writes may be initially directed to an SSD or other relatively low latency and high IOPS data storage layer. The data may remain in the SSD storage layer until a group of data may be written, together, to a sequential or semi-sequential storage layer. A data organizer may group the data into data buckets based on tags and/or policies associated with the data. In this way, data subject to similar lifetime, priority, data protection, and/or other policies may be stored on a same segment or other portion of the sequential storage layer. Similarly, data having similar access patterns, authors, files, objects, or project affiliations, may be stored together on a same segment of the sequential storage layer.
1. A data storage system comprising:
a first storage layer comprising a first type of storage device;
a second storage layer comprising a second type of storage device different from the first type;
an I/O manager to direct incoming write requests to the first storage layer; and
a data organizer to:
determine whether data corresponding to an incoming write request corresponds to a first existing data bucket on the first storage layer based on a comparison of data tags or meta data describing the data corresponding to the incoming write request;
store the data in the first existing data bucket and a second existing data bucket on the first storage layer when data does correspond to the first existing data bucket;
create a new data bucket when data does not correspond to the existing data buckets, the new data bucket to store data having data tags or meta data similar to the data corresponding to the incoming write request, and store the data in the new data bucket in the first storage layer; and
when a data bucket reaches its corresponding data limit, move the data in the data bucket from the first storage layer to the second storage layer.
2. The data storage system of claim 1 , wherein the first storage layer comprises one or more solid state drives.
3. The data storage system of claim 2 , wherein the first storage layer comprises one or more flash storage devices.
4. The data storage system of claim 3 , wherein the second storage layer comprises one or more sequential or semi-sequential storage devices.
5. The data storage system of claim 4 , wherein the second storage layer comprises one or more shingled magnetic recording devices.
6. The data storage system of claim 1 , wherein the data organizer is to review one or more tags associated with the data in a data bucket to determine the at least one similar attribute.
7. The data storage system of claim 6 , wherein the at least one attribute relates to a data lifetime policy.
8. The data storage system of claim 6 , wherein the at least one attribute relates to a data priority policy.
9. The data storage system of claim 6 , wherein the at least one attribute relates to a data protection policy.
10. The data storage system of claim 6 , wherein the at least one attribute relates to temporal locality.
11. The data storage system of claim 1 , wherein the I/O manager additionally directs incoming read requests to either of the first storage layer and the second storage layer, depending on where the relevant data is stored.
12. The data storage system of claim 1 , wherein the data organizer additionally moves data from the second storage layer to the first storage layer based on one or more policies.
13. The data storage system of claim 1 , further comprising a segment manager to search the second storage layer for available storage space.
14. The data storage system of claim 8 , wherein the segment manager to maintain a record of available space on the second storage layer.
15. A method for storing data, the method comprising:
receiving a write request for data;
determining whether data corresponding to the mite request corresponds to a first existing data bucket on the first storage layer based on a comparison of data tags or meta data describing the data corresponding to the incoming write request;
storing the data in the first existing data bucket and a second existing data bucket on the first storage layer when data does correspond to the first existing data bucket;
creating a new data bucket when data does not correspond to the existing data buckets, the new data bucket to store data having data tags or meta data similar to the data corresponding to the incoming write request, and writing the data to the new data bucket in the first storage layer comprising a first type of storage device;
grouping the data with other data having at least one similar attribute into a set of grouped data; and
when the grouped data reaches a predetermined data limit, moving the grouped data to a second storage layer comprising a second type of storage device.
16. The method of claim 15 , further comprising performing a data policy operation with respect to the grouped data.
17. The method of claim 15 , wherein grouping the data with other data having at least one similar attribute into a set of grouped data comprises placing the data in a data bucket based on one or more tags associated with the data.
18. The method of claim 17 , wherein the one or more tags relate to at least one of a data lifetime policy, a data priority policy, a data protection policy, and temporal locality.
19. A data handling system comprising:
an I/O manager to direct incoming write requests to a first storage layer, the first storage layer comprising a first type of data storage device; and
a data organizer to:
determine whether data corresponding to an incoming write request corresponds to a first existing data bucket on the first storage layer based on a comparison of data tags or meta data describing the data corresponding to the incoming write request;
store the data in the first existing data bucket and a second existing data bucket on the first storage layer when data does correspond to the first existing data bucket;
create a new data bucket when data does not correspond to the existing data buckets, the new data bucket to store data having data tags or meta data similar to the data corresponding to the incoming write request, and store the data in the new data bucket in the first storage layer; and
when a data bucket reaches a data limit, moving the data in the bucket from the first storage layer to a second storage layer, the second storage layer comprising a second type of data storage device different from the first type;
wherein the second type of data storage device includes segments with a segment having a size that substantially corresponds to the data limit, and wherein moving the data in the bucket comprises selecting the segment from the second storage layer with the size that substantially corresponds to the data limit, and moving the data from the data bucket to the segment.
20. The data handling system of claim 19 , wherein to determine whether data corresponds to the existing data bucks comprises reviewing one or more tags associated with the data.
21. The data storage system of claim 1 , wherein the data organizer is to:
delete the data from the second existing data bucket when the data in the first existing data bucket is moved from the first storage layer to the second storage layer.