PROVIDING BLOCK-BASED DATASET STORAGE USING LAYERED VIRTUAL AND BACKEND DEVICES
A method of providing block-based storage is described. The method comprises: creating a virtual storage system for storing a dataset, the virtual storage system including a virtual drive layer that includes a plurality of virtual drives for storing at least a portion of the dataset as block data; and a block storage backend layer for storing at least the portion of the dataset as block data, the block storage backend layer including one or more block-based storage devices; receiving an I/O operation with respect to the dataset; and based on the received I/O operation, storing, in the block storage backend layer, an update to the dataset.
1 . A method of providing block-based storage, the method comprising:
creating a virtual storage system including:
a virtual drive layer comprising a plurality of virtual drives, wherein at least one virtual drive includes a local storage component for storing data from one or more I/O operations associated with a dataset as block data; and
a backend storage layer comprising one or more block-based storage devices for storing at least a portion of the dataset as block data;
receiving an I/O operation associated with the dataset, including storing data for the I/O operation in one or more virtual drives of the virtual drive layer; and
based on the I/O operation, storing, in the backend storage layer, an update to the dataset as block data, wherein the update is generated at the virtual drive layer.
2 . The method of claim 1 , wherein the local storage component comprises local instance storage associated with a cloud computing instance hosting the at least one virtual drive.
3 . The method of claim 1 , wherein storing data for the I/O operation in the one or more virtual drives includes caching the data in the local storage component prior to transferring the data to the backend storage layer.
4 . The method of claim 1 , wherein storing, in the backend storage layer, the update to the dataset further comprises transferring, from the virtual drive layer to the backend storage layer, the data for the received I/O operation.
5 . The method of claim 1 , wherein the backend storage layer includes a plurality of block-based storage devices comprising at least two different types of cloud-based solid state drives.
6 . The method of claim 5 , wherein storing the update to the dataset in the backend storage layer comprises selecting one of the block-based storage devices based on I/O performance capacity or cost constraints.
7 . The method of claim 1 , further comprising configuring the virtual drive layer to continue processing I/O operations during a modification of the backend storage layer, including replacement or upgrade of a block-based storage device.
8 . An apparatus comprising a virtual storage system, the virtual storage system comprising:
a virtual drive layer comprising a plurality of virtual drives, wherein at least one virtual drive includes a local storage component for storing data from one or more I/O operations associated with a dataset as block data; and
a backend storage layer comprising one or more block-based storage devices for storing at least a portion of the dataset as block data,
wherein the apparatus is configured to:
receive an I/O operation associated with the dataset, including storing data for the I/O operation in one or more virtual drives of the virtual drive layer; and
based on the I/O operation, store, in the backend storage layer, an update to the dataset as block data, wherein the update is generated at the virtual drive layer.
9 . The apparatus of claim 8 , wherein the local storage component comprises local instance storage associated with a cloud computing instance hosting the at least one virtual drive.
10 . The apparatus of claim 8 , wherein storing data for the I/O operation in the one or more virtual drives includes caching the data in the local storage component prior to transferring the data to the backend storage layer.
11 . The apparatus of claim 8 , wherein storing, in the backend storage layer, the update to the dataset further comprises transferring, from the virtual drive layer to the backend storage layer, the data for the received I/O operation.
12 . The apparatus of claim 8 , wherein the backend storage layer includes a plurality of block-based storage devices comprising at least two different types of cloud-based solid state drives.
13 . The apparatus of claim 12 , wherein storing the update to the dataset in the backend storage layer comprises selecting one of the block-based storage devices based on I/O performance capacity or cost constraints.
14 . The apparatus of claim 8 , wherein the virtual drive layer is further configured to continue processing I/O operations during a modification of the backend storage layer, including replacement or upgrade of a block-based storage device.
15 . A non-transitory computer-readable storage medium storing instructions that, when executed by a processing device, cause the processing device to:
create a virtual storage system including:
a virtual drive layer comprising a plurality of virtual drives, wherein at least one virtual drive includes a local storage component for storing data from one or more I/O operations associated with a dataset as block data; and
a backend storage layer comprising one or more block-based storage devices for storing at least a portion of the dataset as block data;
receive an I/O operation associated with the dataset, including storing data for the I/O operation in one or more virtual drives of the virtual drive layer; and
based on the I/O operation, store, in the backend storage layer, an update to the dataset as block data, wherein the update is generated at the virtual drive layer.
16 . The non-transitory computer-readable storage medium of claim 15 , wherein the local storage component comprises local instance storage associated with a cloud computing instance hosting the virtual drive.
17 . The non-transitory computer-readable storage medium of claim 15 , wherein storing data for the I/O operation in the one or more virtual drives includes caching the data in the local storage component prior to transferring the data to the backend storage layer.
18 . The non-transitory computer-readable storage medium of claim 15 , wherein storing, in the backend storage layer, the update to the dataset further comprises transferring, from the virtual drive layer to the backend storage layer, the data for the received I/O operation.
19 . The non-transitory computer-readable storage medium of claim 15 , wherein the backend storage layer includes a plurality of block-based storage devices comprising at least two different types of cloud-based solid state drives.
20 . The non-transitory computer-readable storage medium of claim 19 , wherein storing the update to the dataset in the backend storage layer comprises selecting one of the block-based storage devices based on I/O performance capacity or cost constraints.