Transparent data movement between a private cloud and storage ecosystem and another storage system
The disclosure provides methods and systems for transparent data movement between a private cloud storage ecosystem and another storage system. The other storage system may be local storage or external storage separate and apart from the private cloud storage ecosystem. Originally, metadata and the data contents of a file are both stored on local storage in the private cloud storage ecosystem. The method separates the metadata from the data contents of a file such that the metadata and data contents are independently operable. After separation and based on policy, the data content is transparently moved between the private cloud storage system and the other storage system. The data is managed and tracked such that a user, e.g., a client or external program/entity, may access the data content using the original metadata stored on the private cloud storage ecosystem, despite the movement of the data contents to the other storage system.
1. A computer-implemented method for providing transparent movement of data between a private cloud storage ecosystem and another storage system, the method comprising:
separating original metadata from data contents of one or more files stored in the private cloud storage ecosystem, the separating including storing the original metadata and the data contents for each of the one or more files separately such that the original metadata and the data contents become separate objects that are configured to be operated on independently from one another, the original metadata being persistently stored in an original location in the private cloud storage ecosystem and identification information being stored separately from the original metadata, the original metadata being separate from the identification information;
after separation and based on a tier policy, selectively moving the data contents of the one or more files transparently between the private cloud storage ecosystem and the another storage system, the data contents being moved from a first tier to a selected tier that is indicated by the tier policy; and
providing user access to the moved data contents of the one or more files, in response to a user request for the one or more files, the user request being associated with the original metadata for the moved data contents, the providing user access further comprising:
obtaining a destination of the moved data contents of the one or more files from a database storing the identification information residing in the private cloud storage ecosystem; and
based on the destination, providing the user access to the moved data contents of the one or more files such that workflow or client mount points appear unchanged from the user's perspective to account for the movement of the data contents of the one or more files from the first tier to the selected tier.
2. The method of claim 1 , wherein the respective original metadata for each of the one or more files remains stored on the private cloud storage ecosystem after the movement of corresponding data contents to the another storage system.
3. The method of claim 1 , wherein the another storage system is a slower local storage and the private cloud storage ecosystem is a faster local storage.
4. The method of claim 1 , wherein the another storage system is external storage that is coupled via one or more communication networks to the private cloud storage ecosystem.
5. The method of claim 4 , wherein the external storage is a public cloud storage system, a different private cloud storage system, or disk storage on an external server.
6. The method of claim 1 , further comprising managing and tracking the movement of the selectively moved data contents of each file of the one or more files.
7. The method of claim 6 , wherein the managing and tracking comprises: storing in the database the identification information of each of the one or more files for which the respective data contents are being or have been selectively moved and a new destination for the respective data contents that have been selectively moved; and tracking the movement of the data contents to the another storage system, corresponding original metadata remaining stored in its original location on the private cloud storage ecosystem.
8. The method of claim 6 , wherein the destination is part of the identification information that is stored in the database, and the identification information is utilized in tracking movement of the moved data contents of each of the one or more files.
9. The method of claim 1 , wherein the user request is a read or write request from a client or an external program/entity to the private cloud storage ecosystem, upon which the user access is provided for the data contents associated with the one or more files.
10. The method of claim 1 , wherein the database is configured to provide a list of files to be moved and a current destination of the moved data contents of the one or more files.
11. The method of claim 1 , wherein the user access to the moved data contents is provided to a client or an external program/entity via a network file sharing protocol, such that the client or the external program/entity accesses the requested data contents by accessing the respective original metadata stored on the private cloud storage ecosystem using the network file sharing protocol.
12. The method of claim 11 , wherein the network file sharing protocol includes Service Message Block (SMB), Network File System (NFS), or Apple File Protocol (AFP) such as utilized for a Network Attached Storage system (NAS).
13. The method of claim 11 , wherein the data contents are accessible to the client or the external program/entity via the network file sharing protocol using the respective original metadata such that workflow or client mount points are unchanged as a result of the movement of the data contents of the one or more files.
14. The method of claim 1 , wherein the tier policy further comprises a policy specifying Hierarchical Storage Management (HSM).
15. The method of claim 1 , wherein the tier policy further comprises a user-defined policy or combination of user-defined policies.
16. The method of claim 15 , where the user-defined policy for selecting files to move may specify one or more of:
selecting a folder and all its contents;
selecting files that have not been accessed in a threshold period of time;
setting a file size with a directory structure that should be moved;
selecting particular file types; and
using a metadata tag on files to override any other user-defined policy.
17. The method of claim 16 , wherein the metadata tag is based on a geographical restriction so as to not permit transparent data movement out of a geographic location due to regulations of a geographic area.
18. The method of claim 1 , wherein the selectively moving the data contents of the one or more files transparently between the private cloud storage ecosystem and the another storage system further comprises selectively moving the data contents in a same manner as before the data contents were moved, such that the workflow or the client mount points are unchanged as a result of the movement of the data contents of the one or more files and the data contents are accessible to the user by using the original metadata.
19. The method of claim 1 , further comprising processing, by a transparent data movement server, the movement of the data contents of the one or more files based on the tier policy.
20. The method of claim 1 , wherein the selectively moving the data contents of the one or more files transparently between the private cloud storage ecosystem and the another storage system is transparent to the user such that the user can access the moved data contents of the one or more files in a same way the user can access the data contents on a network attached storage (NAS) system.
21. The method of claim 1 , wherein after the moving the data contents of the one or more files transparently from the private cloud storage ecosystem to the another storage system, the original metadata for each of the one or more files remains stored in the original location on the private cloud storage ecosystem.
22. The method of claim 1 , wherein the selectively moving the data contents of the one or more files transparently between the private cloud storage ecosystem and the another storage system results in a reduction in capacity for storage consumption.
23. A system for providing transparent movement of data, the system comprising at least one processor and a memory communicatively coupled to the processor, the memory storing instructions executable by the processor to perform a method for providing transparent movement of data between a private cloud storage ecosystem and another storage system, the method comprising: separating original metadata from data contents of one or more files stored in the private cloud storage ecosystem, the separating including storing the original metadata and the data contents for each of the one or more files separately such that the original metadata and the data contents become separate objects that are configured to be operated on independently from one another, the original metadata being persistently stored in an original location in the private cloud storage ecosystem and identification information being stored separately from the original metadata, the original metadata being separate from the identification information; after separation and based on a tier policy, selectively moving the data contents of the one or more files transparently between the private cloud storage ecosystem and the another storage system, the data contents being moved from a first tier to a selected tier that is indicated by the tier policy; and providing user access to the moved data contents of the one or more files, in response to a user request for the one or more files, the user request being associated with the original metadata for the moved data contents, the providing user access further comprising: obtaining a destination of the moved data contents of the one or more files from a database storing the identification information residing in the private cloud storage ecosystem; and based on the destination, providing the user access to the moved data contents of the one or more files such that workflow or client mount points appear unchanged from the user's perspective to account for the movement of the data contents of the one or more files from the first tier to the selected tier.
24. A non-transitory computer readable medium having instructions stored thereon, which when executed by one or more processors, cause the one or more processors to implement a method for providing transparent movement of data between a private cloud storage ecosystem and another storage system, the method comprising: separating original metadata from data contents of one or more files stored in the private cloud storage ecosystem, the separating including storing the original metadata and the data contents for each of the one or more files separately such that the original metadata and the data contents become separate objects that are configured to be operated on independently from one another, the original metadata being persistently stored in an original location in the private cloud storage ecosystem and identification information being stored separately from the original metadata, the original metadata being separate from the identification information; after separation and based on a tier policy, selectively moving the data contents of the one or more files transparently between the private cloud storage ecosystem and the another storage system, the data contents being moved from a first tier to a selected tier that is indicated by the tier policy; and providing user access to the moved data contents of the one or more files, in response to a user request for the one or more files, the user request being associated with the original metadata for the moved data contents, the providing user access further comprising: obtaining a destination of the moved data contents of the one or more files from a database storing the identification information residing in the private cloud storage ecosystem; and based on the destination, providing the user access to the moved data contents of the one or more files such that workflow or client mount points appear unchanged from the user's perspective to account for the movement of the data contents of the one or more files from the first tier to the selected tier.