Hybrid tiered storage for cloud platforms
According to examples, a cloud platform implementing a tiered storage that decouples Request Units from storage is disclosed. Data received from partitions that are characterized by corresponding partition keys is stored to the local storage medium of a virtual machine (VM) as hot data. The received data is formatted as immutable segments and stored to a disaggregated storage medium shared by a plurality of VMs. The immutable segments are grouped into sorted runs within the disaggregated storage medium and are further assigned to different levels. The hot data is deleted from a local storage medium when time-to-cold (TTC) criteria are met. The cold data can be accessed for document retrieval via a search key that includes a partition key and a documentID. Additional Request Units can be allocated or de-allocated for a portion of the cold data without moving the cold data.
1 . An apparatus, comprising:
a processor; and
a memory on which are stored machine-readable instructions that when executed by the processor, cause the processor to:
store, as cold data in a disaggregated storage medium, immutable segments from Virtual Machines (VMs), wherein the immutable segments include documents from partitions that are characterized by corresponding partition keys;
format the cold data into sorted runs, wherein a sorted run includes at least one of the immutable segments that is arranged in non-overlapping search key ranges and wherein search keys correspond to the documents and a search key of a document includes corresponding partition key and a documentID of the document;
assign the sorted runs to corresponding levels of multiple levels based on timestamps of the sorted runs; and
generate metadata maps that store metadata identifying the immutable segments and the sorted runs including the immutable segments.
2 . The apparatus of claim 1 , wherein the machine-readable instructions further cause the processor to:
enable searching for corresponding documents in the cold data via the search keys.
3 . The apparatus of claim 2 , wherein to enable searching for corresponding documents in the cold data via the search keys, the machine-readable instructions further cause the processor to:
receive a search key associated with a document of the corresponding documents to be retrieved from the disaggregated storage medium, wherein the received search key includes one or more of a partition key and a document ID.
4 . The apparatus of claim 3 , wherein to enable searching for corresponding documents in the cold data via the search keys, the machine-readable instructions further cause the processor to:
identify a corresponding level of the multiple levels and a sorted run in the disaggregated storage medium with a search key range including the received search key when the received search key includes the partition key and the document ID.
5 . The apparatus of claim 4 , wherein the machine-readable instructions further cause the processor to:
retrieve the document based on a bloom filter output that determines that the received search key is present in a segment of the sorted run in the corresponding level.
6 . The apparatus of claim 3 , wherein to enable searching for corresponding documents in the cold data via the search keys, the machine-readable instructions further cause the processor to:
output documents retrieved from a corresponding partition based on the search key including only a partition key.
7 . The apparatus of claim 1 , wherein the machine-readable instructions further cause the processor to:
receive a request for allocation of additional Request Units for at least a portion of the cold data;
instantiate an additional VM in response to receipt of the request for allocation of additional Request Units for at least a portion of the cold data; and
enable the additional instantiated VM to access the at least a portion of the cold data by passing metadata of a subset of the immutable segments in the at least a portion of the cold data.
8 . The apparatus of claim 7 , wherein to allocate the additional Request Units for at least the portion of the cold data, the machine-readable instructions further cause the processor to:
identify the sorted runs that include a subset of the immutable segments from the portion of the cold data.
9 . The apparatus of claim 8 , wherein to allocate the additional Request Units for at least the portion of the cold data the machine-readable instructions further cause the processor to:
provide access to the identified sorted runs to the additional VM.
10 . A processor-executable method comprising:
storing, by a processor, as cold data in a disaggregated storage medium, received data including documents from partitions that are characterized by corresponding partition keys;
formatting, by the processor, the cold data into sorted runs assigned to multiple levels, wherein each sorted run includes at least one segment and the sorted runs are identified by corresponding search key ranges, and wherein search keys correspond to the documents and a search key of a document includes corresponding partition key and a documentID;
receiving, by the processor, a scale-up request for allocation of additional Request Units for a first portion of cold data stored on a disaggregated storage medium;
instantiating, by the processor, an additional virtual machine (VM) in response to a request for allocation of additional Request Units; and
enabling, by the processor, the additional VM to access the first portion of the cold data by passing metadata including addresses of segments in the first portion of the cold data.
11 . The method of claim 10 , wherein the additional VM is enabled for point reads of documents in the cold data.
12 . The method of claim 10 , further comprising:
receiving, by the processor, a scale-down request for a second portion of the cold data; and
deleting, by the processor, a VM accessing the second portion while keeping the cold data unchanged.
13 . The method of claim 10 , wherein enabling the additional VM to access the first portion further comprises:
enabling, by the processor, the additional VM to access the first portion of the cold data while keeping the cold data unchanged.
14 . A non-transitory computer readable storage medium having instructions stored thereon that, when executed by a processing device, cause the processing device to:
store, as cold data in a disaggregated storage medium, immutable segments from Virtual Machines (VMs), wherein the immutable segments include documents from partitions that are characterized by corresponding partition keys;
format the cold data into sorted runs, wherein a sorted run includes at least one of the immutable segments that is arranged in non-overlapping search key ranges and wherein search keys correspond to the documents and a search key of a document includes corresponding partition key and a documentID of the document;
assign the sorted runs to corresponding levels of multiple levels based on timestamps of the sorted runs; and
generate metadata maps that store metadata identifying the immutable segments and the sorted runs including the immutable segments.
15 . The non-transitory computer readable storage medium of claim 14 , wherein the machine-readable instructions further cause the processor to:
enable searching for corresponding documents in the cold data via the search keys.
16 . The non-transitory computer readable storage medium of claim 15 , wherein to enable searching for corresponding documents in the cold data via the search keys, the machine-readable instructions further cause the processor to:
receive a search key associated with a document of the corresponding documents to be retrieved from the disaggregated storage medium, wherein the received search key includes one or more of a partition key and a document ID.
17 . The non-transitory computer readable storage medium of claim 16 , wherein to enable searching for corresponding documents in the cold data via the search keys, the machine-readable instructions further cause the processor to:
identify a corresponding level of the multiple levels and a sorted run in the disaggregated storage medium with a search key range including the received search key when the received search key includes the partition key and the document ID.
18 . The non-transitory computer readable storage medium of claim 17 , wherein the machine-readable instructions further cause the processor to:
retrieve the document based on a bloom filter output that determines that the received search key is present in a segment of the sorted run in the corresponding level.
19 . The non-transitory computer readable storage medium of claim 16 , wherein to enable searching for corresponding documents in the cold data via the search keys, the machine-readable instructions further cause the processor to:
output documents retrieved from a corresponding partition based on the search key including only a partition key.
20 . The non-transitory computer readable storage medium of claim 14 , wherein the machine-readable instructions further cause the processor to:
receive a request for allocation of additional Request Units for at least a portion of the cold data;
instantiate an additional VM in response to receipt of the request for allocation of additional Request Units for at least a portion of the cold data; and
enable the additional instantiated VM to access the at least a portion of the cold data by passing metadata of a subset of the immutable segments in the at least a portion of the cold data.