Managing cloud storage of block-based and file-based data
A technique for managing cloud storage provides a unified framework that supports both block-type data, such as LUNs (Logical UNits) and volumes, and file-type data, such as files, and expresses both types of data in the form of objects, which form a common base for both types of data. The framework includes a single API (application program interface) for managing storage and retrieval of block-derived objects and file-derived objects from the cloud, e.g., from a cloud-based data store.
1. A method of managing cloud-based data storage, the method comprising:
operating an FTO (file-to-object) component on a computing machine, the FTO component configured to convert between files hosted by a data storage system and cloud-based objects that store contents of the files, the cloud-based objects that store contents of the files being accessible to the data storage system via a cloud-based API (application program interface) component running on the computing machine;
operating a VTO (volume-to-object) component on the computing machine, the VTO component configured to convert between volumes hosted by the data storage system and cloud-based objects that store contents of the volumes, the cloud-based objects that store contents of the volumes being accessible to the data storage system via the cloud-based API component;
in response to a request from the data storage system to access a file, (i) transforming, by the FTO component, the request to access the file into a corresponding set of requests to access a first set of cloud-based objects and (ii) accessing, by the cloud-based API component, the first set of cloud-based objects to obtain data of the file from a set of cloud-based data stores; and
in response to a request from the data storage system to access a volume, (i) transforming, by the VTO component, the request to access the volume into a corresponding set of requests to access a second set of cloud-based objects and (ii) accessing, by the cloud-based API component, the second set of cloud-based objects to obtain data of the volume from the set of cloud-based data stores.
2. The method of claim 1 , further comprising operating the FTO component, the VTO component, and the cloud-based API component together within a CMA (cloud management appliance), the CMA operating within a virtualized processing instance on the computing machine, the virtualized processing instance realized as one of (i) a virtual machine or (ii) a container that provides an isolated userspace instance.
3. The method of claim 2 , wherein the computing machine is one of multiple hardware computing nodes that operate within the data storage system.
4. The method of claim 2 , further comprising operating an orchestration component within the CMA, including:
operating decision logic configured to select between the FTO component and the VTO component when responding to requests to perform cloud-based data storage operations;
implementing a set of policies for performing the cloud-based storage operations on a set of data elements in the data storage system; and
operating a scheduler configured to perform at least one of the cloud-based storage operations on a predetermined schedule.
5. The method of claim 4 , wherein the decision logic is configured to (i) select the FTO component when responding to requests to perform file archiving to the cloud and (ii) select the VTO component when responding to requests to perform at least one of (a) snapshot-shipping of file systems to the cloud and (b) archiving of LUNs (Logical UNits) to the cloud.
6. The method of claim 4 , further comprising:
obtaining, by the CMA, an identifier of a data element in the data storage system upon which a cloud-based storage activity is to be performed; and
directing, by the CMA and in accordance with the predetermined schedule, performance of the cloud-based storage activity on the identified data element in accordance with the set of policies.
7. The method of claim 6 , wherein the CMA has a REST interface to a component of the data storage system, and wherein obtaining the identifier of the data element includes querying, by the CMA, the component of the data storage system over the REST interface.
8. The method of claim 6 , further comprising operating a user-interface component of the CMA, the user-interface component configured to receive commands from a user to identify data elements on which to perform cloud-based storage activities.
9. The method of claim 6 , further comprising operating a user-interface component of the data storage system, the user-interface component configured to (i) receive commands from a user to configure the data storage system and (ii) receive commands from the user to identify data elements on which to perform cloud-based storage activities.
10. The method of claim 2 , wherein the cloud-based objects that store contents of the files and the cloud-based objects that store contents of the volumes reside in the same cloud-based data store, the method thereby using objects in the cloud-based data store as a common base for storing both files and volumes.
11. A computerized apparatus, comprising control circuitry constructed and arranged to:
operate an FTO (file-to-object) component on a computing machine, the FTO component configured to convert between files hosted by a data storage system and cloud-based objects that store contents of the files, the cloud-based objects that store contents of the files being accessible to the data storage system via a cloud-based API (application program interface) component running on the computing machine;
operate a VTO (volume-to-object) component on the computing machine, the VTO component configured to convert between volumes hosted by the data storage system and cloud-based objects that store contents of the volumes, the cloud-based objects that store contents of the volumes being accessible to the data storage system via the cloud-based API component;
in response to a request from the data storage system to access a file, (i) transform, by the FTO component, the request to access the file into a corresponding set of requests to access a first set of cloud-based objects and (ii) access, by the cloud-based API component, the first set of cloud-based objects to obtain data of the file from a set of cloud-based data stores; and
in response to a request from the data storage system to access a volume, (i) transform, by the VTO component, the request to access the volume into a corresponding set of requests to access a second set of cloud-based objects and (ii) access, by the cloud-based API component, the second set of cloud-based objects to obtain data of the volume from the set of cloud-based data stores.
12. A computer program product including a set of non-transitory, computer-readable media having instructions which, when executed by control circuitry of a computerized apparatus, cause the computerized apparatus to perform a method of managing cloud-based storage, the method comprising:
operating an FTO (file-to-object) component on a computing machine, the FTO component configured to convert between files hosted by a data storage system and cloud-based objects that store contents of the files, the cloud-based objects that store contents of the files being accessible to the data storage system via a cloud-based API (application program interface) component running on the computing machine;
operating a VTO (volume-to-object) component on the computing machine, the VTO component configured to convert between volumes hosted by the data storage system and cloud-based objects that store contents of the volumes, the cloud-based objects that store contents of the volumes being accessible to the data storage system via the cloud-based API component;
in response to a request from the data storage system to access a file, (i) transforming, by the FTO component, the request to access the file into a corresponding set of requests to access a first set of cloud-based objects and (ii) accessing, by the cloud-based API component, the first set of cloud-based objects to obtain data of the file from a set of cloud-based data stores; and
in response to a request from the data storage system to access a volume, (i) transforming, by the VTO component, the request to access the volume into a corresponding set of requests to access a second set of cloud-based objects and (ii) accessing, by the cloud-based API component, the second set of cloud-based objects to obtain data of the volume from the set of cloud-based data stores.
13. The computer program product of claim 12 , wherein the method further comprises operating the FTO component, the VTO component, and the cloud-based API component together within a CMA (cloud management appliance), the CMA operating within a virtualized processing instance on the computing machine, the virtualized processing instance realized as one of (i) a virtual machine or (ii) a container that provides an isolated userspace instance.
14. The computer program product of claim 13 , wherein the method further comprises operating an orchestration component within the CMA, including:
operating decision logic configured to select between the FTO component and the VTO component when responding to requests to perform cloud-based data storage operations;
implementing a set of policies for performing the cloud-based storage operations on a set of data elements in the data storage system; and
operating a scheduler configured to perform at least one of the cloud-based storage operations on a predetermined schedule.
15. The computer program product of claim 14 , wherein the decision logic is configured to (i) select the FTO component when responding to requests to perform file archiving to the cloud and (ii) select the VTO component when responding to requests to perform at least one of (a) snapshot-shipping of file systems to the cloud and (b) archiving of LUNs (Logical UNits) to the cloud.
16. The computer program product of claim 14 , wherein the method further comprises:
obtaining, by the CMA, an identifier of a data element in the data storage system upon which a cloud-based storage activity is to be performed; and
directing, by the CMA and in accordance with the predetermined schedule, performance of the cloud-based storage activity on the identified data element in accordance with the set of policies.
17. The computer program product of claim 16 , wherein the CMA has a REST interface to a component of the data storage system, and wherein obtaining the identifier of the data element includes querying, by the CMA, the component of the data storage system over the REST interface.
18. The computer program product of claim 16 , wherein the method further comprises operating a user-interface component of the CMA, the user-interface component configured to receive commands from a user to identify data elements on which to perform cloud-based storage activities.
19. The computer program product of claim 16 , wherein the method further comprises operating a user-interface component of the data storage system, the user-interface component configured to (i) receive commands from a user to configure the data storage system and (ii) receive commands from the user to identify data elements on which to perform cloud-based storage activities.
20. The computer program product of claim 13 , wherein the cloud-based objects that store contents of the files and the cloud-based objects that store contents of the volumes reside in the same cloud-based data store, the method thereby using objects in the cloud-based data store as a common base for storing both files and volumes.
21. The method of claim 1 , wherein the set of cloud-based data stores are object-based data stores, wherein the first set of cloud-based objects are object-based data, and wherein the second set of cloud-based objects are object-based data.