Cluster file system comprising data mover module arranged between front-end and back-end file systems
A cluster file system includes a front-end file system comprising a hierarchical storage management application programming interface (API), a back-end file system, and a data mover module arranged between the front-end and back-end file systems and adapted for communication with the hierarchical storage management API of the front-end file system. The data mover module is configured to control movement of data between the front-end file system and the back-end file system responsive to information received via the hierarchical storage management API. For example, in some embodiments, the data mover module splits a given file received from the front-end file system into multiple smaller files for storage in the back-end file system and associates with each of the smaller files a corresponding file extension from a designated namespace to permit reconstruction of the given file upon retrieval of the smaller files from the back-end file system.
1. An apparatus comprising:
a front-end file system comprising a hierarchical storage management application programming interface;
a back-end file system comprising a scale-out network attached storage cluster comprising a plurality of nodes; and
a data mover module arranged between the front-end and back-end file systems and adapted for communication with the hierarchical storage management application programming interface of the front-end file system;
wherein the data mover module is configured to control movement of data between the front-end file system and the back-end file system responsive to information received via the hierarchical storage management application programming interface;
wherein the data mover module in conjunction with controlling the movement of data between the front-end file system and the back-end file system is configured:
to determine whether a given file received from the front-end file system is larger than a maximum file size permitted by one or more nodes of the scale-out network attached storage cluster;
to split the given file received from the front-end file system into multiple smaller files for storage in the back-end file system responsive to determining that the given file is larger than the maximum file size permitted by one or more nodes of the scale-out network attached storage cluster; and
to associate with each of the smaller files a corresponding file extension from a designated namespace so as to permit the given file to be reconstructed upon retrieval of the smaller files from the back-end file system;
wherein the designated namespace comprises a first portion identifying a type of file system utilized by the front-end file system and a second portion identifying at least one of a segment of the given file and a sub-segment of the given file; and
wherein the data mover module is implemented using at least one processing device comprising a processor coupled to a memory.
2. The apparatus of claim 1 wherein the back-end file system is of a different type than the front-end file system.
3. The apparatus of claim 1 wherein the front-end file system comprises a Lustre file system.
4. The apparatus of claim 1 further comprising a plurality of back-end file systems with the data mover module being configured to control movement of data between the front-end file system and each of the back-end file systems responsive to information received via the hierarchical storage management application programming interface.
5. The apparatus of claim 4 wherein the plurality of back-end file systems comprise at least one object store and at least one scale-out network-attached storage cluster.
6. The apparatus of claim 1 wherein the front-end file system and back-end file system are associated with respective different sets of one or more hierarchical storage tiers for use in hierarchical storage management.
7. The apparatus of claim 1 further comprising a plurality of compute nodes coupled to the front-end file system.
8. The apparatus of claim/wherein the data mover module is implemented at least in part utilizing software running on the compute nodes.
9. The apparatus of claim 1 wherein the data mover module comprises a plurality of server nodes that are synchronized with one another utilizing a message passing interface (MPI) protocol.
10. The apparatus of claim 1 wherein the data mover module is implemented at least in part utilizing a virtual layer of a parallel log-structured file system (PLFS).
11. The apparatus of claim 1 wherein the virtual layer comprises PLFS middleware configured to migrate files between the front-end and back-end file systems.
12. A processing platform comprising the apparatus of claim 1 .
13. A method comprising:
configuring a data mover module for arrangement between a front-end file system and a back-end file system;
adapting the data mover module for communication with a hierarchical storage management application programming interface of the front-end file system; and
controlling movement of data between the front-end file system and the back-end file system responsive to information received in the data mover module via the hierarchical storage management application programming interface;
wherein the back-end file system comprises a scale-out network attached storage cluster comprising a plurality of nodes;
wherein controlling movement of data between the front-end file system and the back-end file system further comprises:
determining whether a given file received from the front-end file system is larger than a maximum file size permitted by one or more nodes of the scale-out network attached storage cluster;
splitting the given file received from the front-end file system into multiple smaller files for storage in the back-end file system responsive to determining that the given file is larger than the maximum file size permitted by one or more nodes of the scale-out network attached storage cluster; and
associating with each of the smaller files a corresponding file extension from a designated namespace so as to permit the given file to be reconstructed upon retrieval of the smaller files from the back-end file system; and
wherein the designated namespace comprises a first portion identifying a type of file system utilized by the front-end file system and a second portion identifying at least one of a segment of the given file and a sub-segment of the given file;
wherein the configuring, adapting and controlling are implemented using at least one processing device comprising a processor coupled to a memory.
14. The method of claim 13 wherein configuring the data mover module comprising implementing the data mover module at least in part utilizing a virtual layer of a parallel log-structured file system (PLFS).
15. An article of manufacture comprising a non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing device causes said at least one processing device:
to configure a data mover module for arrangement between a front-end file system and a back-end file system;
to adapt the data mover module for communication with a hierarchical storage management application programming interface of the front-end file system; and
to control movement of data between the front-end file system and the back-end file system responsive to information received in the data mover module via the hierarchical storage management application programming interface;
wherein the back-end file system comprises a scale-out network attached storage cluster comprising a plurality of nodes;
wherein controlling movement of data between the front-end file system and the back-end file system further comprises:
determining whether a given file received from the front-end file system is larger than a maximum file size permitted by one or more nodes of the scale-out network attached storage cluster;
splitting the given file received from the front-end file system into multiple smaller files for storage in the back-end file system responsive to determining that the given file is larger than the maximum file size permitted by one or more nodes of the scale-out network attached storage cluster; and
associating with each of the smaller files a corresponding file extension from a designated namespace so as to permit the given file to be reconstructed upon retrieval of the smaller files from the back-end file system; and
wherein the designated namespace comprises a first portion identifying a type of file system utilized by the front-end file system and a second portion identifying at least one of a segment of the given file and a sub-segment of the given file.
16. An information processing system comprising:
a computer system comprising a plurality of compute nodes; and
a cluster file system coupled to the computer system, the cluster file system comprising:
a front-end file system comprising a hierarchical storage management application programming interface;
a back-end file system comprising a scale-out network attached storage cluster comprising a plurality of nodes; and
a data mover module arranged between the front-end and back-end file systems and adapted for communication with the hierarchical storage management application programming interface of the front-end file system;
wherein the data mover module is configured to control movement of data between the front-end file system and the back-end file system responsive to information received via the hierarchical storage management application programming interface of the front-end file system;
wherein the data mover module in conjunction with controlling the movement of data between the front-end file system and the back-end file system is configured:
to determine whether a given file received from the front-end file system is larger than a maximum file size permitted by one or more nodes of the scale-out network attached storage cluster;
to split the given file received from the front-end file system into multiple smaller files for storage in the back-end file system responsive to determining that the given file is larger than the maximum file size permitted by one or more nodes of the scale-out network attached storage cluster; and
to associate with each of the smaller files a corresponding file extension from a designated namespace so as to permit the given file to be reconstructed upon retrieval of the smaller files from the back-end file system;
wherein the designated namespace comprises a first portion identifying a type of file system utilized by the front-end file system and a second portion identifying at least one of a segment of the given file and a sub-segment of the given file; and
wherein the information processing system is implemented by at least one processing device comprising a processor coupled to a memory.
17. The apparatus of claim 1 wherein the data mover module is further configured to generate one or more metadata files in conjunction with the split of the given file, and to store the one or more metadata files in the back-end file system, wherein the one or more metadata files indicate a manner in which the smaller files are distributed over multiple nodes of the back-end file system.
18. The method of claim 13 wherein the data mover module is further configured to generate one or more metadata files in conjunction with the split of the given file, and storing the one or more metadata files in the back-end file system, wherein the one or more metadata files indicate a manner in which the smaller files are distributed over multiple nodes of the back-end file system.
19. The system of claim 16 wherein the data mover module is further configured to generate one or more metadata files in conjunction with the split of the given file, and store the one or more metadata files in the back-end file system, wherein the one or more metadata files indicate a manner in which the smaller files are distributed over multiple nodes of the back-end file system.
20. The apparatus of claim 1 , wherein the front-end data system comprises one or more object storage targets, the one or more object storage targets of the front-end data system having a maximum file size greater than the maximum file size of permitted by one or more of the nodes of the scale-out network attached storage cluster of the back-end data system.