Storage system comprising cluster file system storage nodes and software-defined storage pool in cloud infrastructure
An apparatus in one embodiment comprises a multi-tier storage system having at least a front-end storage tier and a back-end storage tier. The multi-tier storage system is implemented at least in part utilizing a plurality of virtual machines of cloud infrastructure. The front-end storage tier comprises a plurality of storage nodes of a cluster file system, with the storage nodes being implemented on respective ones of the virtual machines. The front-end storage tier further comprises a software-defined storage pool accessible to the storage nodes and implemented utilizing local disk resources of respective ones of the virtual machines. The back-end storage tier of the multi-tier storage system comprises at least one object store. At least a subset of the virtual machines may further comprise respective compute nodes configured to access the multi-tier storage system. Other illustrative embodiments include systems, methods and processor-readable storage media.
1. An apparatus comprising:
a multi-tier storage system comprising at least a front-end storage tier and a back-end storage tier;
the multi-tier storage system being implemented at least in part utilizing a plurality of virtual machines of cloud infrastructure;
the front-end storage tier comprising:
a plurality of storage nodes of a cluster file system with the storage nodes being implemented on respective ones of the virtual machines; and
a software-defined storage pool accessible to the storage nodes and implemented utilizing local disk resources of respective ones of the virtual machines;
the back-end storage tier being separate from the software-defined storage pool and comprising at least one object store;
wherein the multi-tier storage system is configured to recover from a failure in the local disk resources of a given one of the virtual machines by obtaining corresponding data from the local disk resources of one or more other ones of the virtual machines via the software-defined storage pool;
wherein a capacity of the software-defined storage pool is automatically adjustable by adjusting sizes of at least a subset of the virtual machines; and
wherein the cloud infrastructure is implemented by a processing platform comprising a plurality of processing devices.
2. The apparatus of claim 1 wherein at least a subset of the virtual machines further comprise respective compute nodes that are configured to access the multi-tier storage system.
3. The apparatus of claim 2 wherein at least one of the virtual machines implements a storage node but not a compute node.
4. The apparatus of claim 2 wherein the object store is configured to store data in the form of objects having respective object identifiers and wherein a specified set of the objects is made accessible to an application running on a given one of the compute nodes via a corresponding dynamically loadable namespace identifier.
5. The apparatus of claim 1 wherein at least one of the storage nodes comprises a file system client and a file system server.
6. The apparatus of claim 1 wherein at least one of the storage nodes implements at least one of a metadata server and a data server with a given such server being stored in the software-defined storage pool.
7. The apparatus of claim 1 wherein the cluster file system comprises at least one of a parallel file system and a distributed file system.
8. The apparatus of claim 1 further comprising a software-defined storage controller configured to implement the software-defined storage pool utilizing the local disk resources of the respective ones of the virtual machines.
9. The apparatus of claim 8 wherein the software-defined storage controller is implemented in a distributed manner over the respective ones of the virtual machines.
10. The apparatus of claim 8 wherein the software-defined storage controller is configured to mirror an entire virtual disk of a given one of the virtual machines such that access to the entire virtual disk of the given virtual machine can be provided to another virtual machine responsive to a failure in the given virtual machine.
11. The apparatus of claim 1 wherein the storage nodes are implemented on respective ones of the virtual machines at least in part utilizing at least one of flash memories and dynamic random access memories of those virtual machines.
12. An information processing system comprising the apparatus of claim 1 .
13. A method comprising:
providing a plurality of virtual machines of cloud infrastructure for implementing at least portions of a multi-tier storage system;
configuring a front-end storage tier of the multi-tier storage system to include:
a plurality of storage nodes of a cluster file system with the storage nodes being implemented on respective ones of the virtual machines; and
a software-defined storage pool accessible to the storage nodes and comprising local disk resources of respective ones of the virtual machines;
configuring a back-end storage tier of the multi-tier storage system to include at least one object store, the back-end storage tier being separate from the software-defined storage pool; and
recovering from a failure in the local disk resources of a given one of the virtual machines by obtaining corresponding data from the local disk resources of one or more other ones of the virtual machines via the software-defined storage pool;
wherein a capacity of the software-defined storage pool is automatically adjustable by adjusting sizes of at least a subset of the virtual machines; and
wherein the method is performed by a processing platform comprising a plurality of processing devices.
14. The method of claim 13 wherein at least a subset of the virtual machines further comprise respective compute nodes that access the multi-tier storage system.
15. The method of claim 13 wherein configuring the front-end storage tier to include a software-defined storage pool further comprises controlling the software-defined storage pool utilizing a software-defined storage controller implemented in a distributed manner over the respective ones of the virtual machines.
16. A non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by a processing platform comprising a plurality of processing devices causes said processing platform:
to provide a plurality of virtual machines of cloud infrastructure for implementing at least portions of a multi-tier storage system;
to configure a front-end storage tier of the multi-tier storage system to include:
a plurality of storage nodes of a cluster file system with the storage nodes being implemented on respective ones of the virtual machines; and
a software-defined storage pool accessible to the storage nodes and comprising local disk resources of respective ones of the virtual machines;
to configure a back-end storage tier of the multi-tier storage system to include at least one object store, the back-end storage tier being separate from the software-defined storage pool;
wherein the multi-tier storage system is further configured to recover from a failure in the local disk resources of a given one of the virtual machines by obtaining corresponding data from the local disk resources of one or more other ones of the virtual machines via the software-defined storage pool; and
wherein a capacity of the software-defined storage pool is automatically adjustable by adjusting sizes of at least a subset of the virtual machines.
17. The processor-readable storage medium of claim 16 wherein configuring the front-end storage tier to include a software-defined storage pool further comprises controlling the software-defined storage pool using a software-defined storage controller implemented in a distributed manner over the respective ones of the virtual machines.