Multi-tier storage system with controllable relocation of files from file system tier to cloud-based object storage tier

1. An apparatus comprising:

a storage system comprising at least first and second storage tiers each comprising a plurality of storage devices, the storage system being implemented at least in part by at least one processing device comprising a hardware processor coupled to a memory;

the first storage tier implementing a file system having a user visible name space and a corresponding hidden data store;

the second storage tier implementing a cloud-based object store;

the storage system being configured to controllably relocate files from the hidden data store of the file system of the first storage tier to the cloud-based object store of the second storage tier;

wherein in conjunction with the relocation of a given one of the files, the given file is replaced in the hidden data store with a corresponding stub providing user access via the user visible name space to an object comprising the given file in the cloud-based object store of the second storage tier;

wherein the user visible name space comprises a plurality of additional stubs;

a first subset of the plurality of additional stubs of the user visible name space pointing to respective files stored in the hidden data store; and

a second subset of the plurality of additional stubs of the user visible name space pointing to the corresponding stub in the hidden data store that points to the object comprising the given file stored in the cloud-based object store of the second storage tier;

wherein the first subset of the plurality of additional stubs of the user visible name space includes first and second additional stubs, the first and second additional stubs pointing to respective first and second variants of the given file associated with first and second writes by one or more clients to the given file subsequent to relocation of the given file to the cloud-based object store of the second storage tier, the first and second variants being cached locally in the hidden data store;

wherein the first and second variants of the given file cached locally in the hidden data store each point to the corresponding stub in the hidden data store that points to the object comprising the given file stored in the cloud-based object stored of the second storage tier.

2. The apparatus of claim 1 wherein controllable relocation of the files from the hidden data store of the file system of the first storage tier to the cloud-based object store of the second storage tier comprises relocation performed in accordance with one or more configured policies.

3. The apparatus of claim 1 wherein the first storage tier comprises a network-attached storage tier implemented at least in part utilizing software-defined storage.

4. The apparatus of claim 1 wherein the file system of the first storage tier is implemented in one or more containers of a processing platform.

5. The apparatus of claim 1 wherein the first storage tier is coupled to the second storage tier via at least one cloud tiering appliance configured to generate the corresponding stub that replaces the given file in the hidden data store.

6. The apparatus of claim 5 wherein the cloud tiering appliance comprises a policy engine configured to determine which of a plurality of files of the hidden data store will be controllably relocated to the cloud-based object store.

7. The apparatus of claim 5 wherein the cloud tiering appliance is implemented in one or more containers of a processing platform.

8. The apparatus of claim 1 wherein the file system having the user visible name space and the corresponding hidden data store comprises a UFS64 file system implemented in combination with one or more higher-layer file systems in a network-attached storage library of a processing platform.

9. The apparatus of claim 1 wherein the files that are controllably relocated from the hidden data store of the file system of the first storage tier to the cloud-based object store of the second storage tier comprise respective compressed and deduplicated files of the hidden data store of the file system that also have corresponding stubs in the user visible name space.

10. The apparatus of claim 1 wherein write operations against files that have been controllably relocated from the hidden data store of the file system of the first storage tier to the cloud-based object store of the second storage tier are cached locally in the first storage tier and subsequently pushed to the cloud-based object store so as to enhance write performance.

11. The apparatus of claim 1 wherein the first storage tier comprises a distributed storage tier comprising a plurality of file system nodes each having a file system with a user visible name space and a corresponding hidden data store and wherein the file system nodes collectively provide a single name space.

12. The apparatus of claim 1 wherein the first storage tier comprises a distributed storage tier comprising a plurality of file system nodes each having a file system with a user visible name space and a corresponding hidden data store and wherein a given file relocated from multiple ones of the hidden data stores into the cloud-based object store of the second storage tier has only a single corresponding object instance in the cloud-based object store but multiple stubs in respective ones of the multiple hidden data stores from which the given file was relocated.

13. The apparatus of claim 1 wherein the storage system is further configured to store a transaction log of changes to the user visible name space and the hidden data store of the file system as at least one object in the cloud-based object store.

14. The apparatus of claim 1 wherein the object store of the second storage tier comprises a plurality of object storage nodes and wherein stored objects of the object store have respective unique object identifiers.

15. A method performed in a storage system with at least first and second storage tiers each comprising a plurality of storage devices, the storage system being implemented at least in part by at least one processing device comprising a hardware processor coupled to a memory, the first storage tier implementing a file system having a user visible name space and a corresponding hidden data store, the second storage tier implementing a cloud-based object store, the method comprising:

controllably relocating files from the hidden data store of the file system of the first storage tier to the cloud-based object store of the second storage tier; and

in conjunction with the relocation of a given one of the files, replacing the given file in the hidden data store with a corresponding stub providing user access via the user visible name space to an object comprising the given file in the cloud-based object store of the second storage tier;

wherein the user visible name space comprises a plurality of additional stubs;

a first subset of the plurality of additional stubs of the user visible name space pointing to respective files stored in the hidden data store; and

a second subset of the plurality of additional stubs of the user visible name space pointing to the corresponding stub in the hidden data store that points to the object comprising the given file stored in the cloud-based object store of the second storage tier;

wherein the first subset of the plurality of additional stubs of the user visible name space includes first and second additional stubs, the first and second additional stubs pointing to respective first and second variants of the given file associated with first and second writes by one or more clients to the given file subsequent to relocation of the given file to the cloud-based object store of the second storage tier, the first and second variants being cached locally in the hidden data store;

wherein the first and second variants of the given file cached locally in the hidden data store each point to the corresponding stub in the hidden data store that points to the object comprising the given file stored in the cloud-based object stored of the second storage tier.

16. The method of claim 15 wherein the files that are controllably relocated from the hidden data store of the file system of the first storage tier to the cloud-based object store of the second storage tier comprise respective compressed and deduplicated files of the hidden data store of the file system that also have corresponding stubs in the user visible name space.

17. The method of claim 15 wherein write operations against files that have been controllably relocated from the hidden data store of the file system of the first storage tier to the cloud-based object store of the second storage tier are cached locally in the first storage tier and subsequently pushed to the cloud-based object store so as to enhance write performance.

18. A computer program product comprising a non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing device of a storage system, the storage system comprising at least first and second storage tiers each comprising a plurality of storage devices, the first storage tier implementing a file system having a user visible name space and a corresponding hidden data store, and the second storage tier implementing a cloud-based object store, causes said storage system:

to controllably relocate files from the hidden data store of the file system of the first storage tier to the cloud-based object store of the second storage tier; and

in conjunction with the relocation of a given one of the files, to replace the given file in the hidden data store with a corresponding stub providing user access via the user visible name space to an object comprising the given file in the cloud-based object store of the second storage tier;

wherein the user visible name space comprises a plurality of additional stubs;

a first subset of the plurality of additional stubs of the user visible name space pointing to respective files stored in the hidden data store; and

a second subset of the plurality of additional stubs of the user visible name space pointing to the corresponding stub in the hidden data store that points to the object comprising the given file stored in the cloud-based object store of the second storage tier;

wherein the first subset of the plurality of additional stubs of the user visible name space includes first and second additional stubs, the first and second additional stubs pointing to respective first and second variants of the given file associated with first and second writes by one or more clients to the given file subsequent to relocation of the given file to the cloud-based object store of the second storage tier, the first and second variants being cached locally in the hidden data store;

wherein the first and second variants of the given file cached locally in the hidden data store each point to the corresponding stub in the hidden data store that points to the object comprising the given file stored in the cloud-based object stored of the second storage tier; and

wherein the first and second additional stubs and the corresponding stub are associated with the given file and the first and second variants of the given file.

19. The computer program product of claim 18 wherein the files that are controllably relocated from the hidden data store of the file system of the first storage tier to the cloud-based object store of the second storage tier comprise respective compressed and deduplicated files of the hidden data store of the file system that also have corresponding stubs in the user visible name space.

20. The computer program product of claim 18 wherein write operations against files that have been controllably relocated from the hidden data store of the file system of the first storage tier to the cloud-based object store of the second storage tier are cached locally in the first storage tier and subsequently pushed to the cloud-based object store so as to enhance write performance.

21. The apparatus of claim 1 wherein the storage system is further configured:

to monitor the first and second writes from the one or more clients to determine whether a total number of newly-written blocks in the first and second variants of the given file exceeds a designated threshold of newly-written blocks; and

to relocate at least one of the first and second variants of the given file from the hidden data store to the cloud-based object store responsive to determining that the total number of newly-written blocks of the given file exceeds the designated threshold of newly-written blocks.