Cluster file system comprising data mover modules having associated quota manager for managing back-end user quotas
A cluster file system comprises a front-end file system, a back-end file system, data mover modules arranged between the front-end and back-end file systems, and a quota manager associated with at least a given one of data mover modules. The data mover modules are configured to control archiving of data between the front-end file system and the back-end file system for respective users based at least in part on respective user quotas established by the quota manager and identifying corresponding portions of the back-end file system available to the users. The front-end file system may comprise archive directories configured for respective ones of the users, with the data mover modules being configured to archive a given file from the front-end file system in the back-end file system responsive to a link to that file being stored in a corresponding one of the archive directories of the front-end file system.
1. An apparatus comprising:
a storage system comprising at least first and second storage tiers associated with respective first and second file systems;
the first file system comprising a plurality of archive directories configured for respective ones of a plurality of users;
the storage system being configured:
to establish user quotas identifying corresponding portions of the second file system available to the respective users;
to control archiving of data between the first and second file systems based at least in part on the user quotas and the archive directories;
wherein a given file is archived from the first file system to the second file system responsive to a link to that file being stored in a corresponding one of the archive directories of the first file system;
wherein controlling archiving of data between the first and second file systems based at least in part on the user quotas and the archive directories comprises preventing archiving of temporary files from the first file system to the second file system by preventing links to the temporary files in the archive directories;
wherein the storage system is further configured to establish archive directory quotas for respective ones of the archive directories, said archive directories including the respective archive directory quotas; and
wherein the storage system is implemented using at least one processing device comprising a processor coupled to a memory.
2. The apparatus of claim 1 wherein the storage system is further configured to proactively recommend changes to the archive directory quotas based on current sizes of the archive directories relative to the established archive directory quotas.
3. The apparatus of claim 1 wherein the storage system is further configured to utilize one or more policies in establishing the user quotas identifying corresponding portions of the second file system available to the respective users.
4. The apparatus of claim 1 wherein the storage system further comprises a plurality of second file systems, and wherein the storage system is further configured to control archiving of data between the first file system and each of the second file systems responsive to user quotas established for each of the second file systems.
5. The apparatus of claim 1 further comprising a plurality of compute nodes coupled to the first storage tier of the storage system.
6. The apparatus of claim 1 wherein the storage system comprises a plurality of server nodes that are synchronized with one another utilizing a message passing interface (MPI) protocol.
7. The apparatus of claim 1 wherein the storage system comprises a virtual layer of a parallel log-structured file system (PLFS) and wherein the virtual layer comprises PLFS middleware configured to migrate files between the first and second file systems.
8. The apparatus of claim 1 wherein the first file system comprises a hierarchical storage management application programming interface and wherein the storage system is further configured to control movement of data between the first file system and the second file system responsive to information received via the hierarchical storage management application programming interface.
9. A method comprising:
configuring a storage system to include at least first and second storage tiers associated with respective first and second file systems, the first file system comprising a plurality of archive directories configured for respective ones of a plurality of users;
establishing user quotas identifying corresponding portions of the second file system available to the respective users; and
controlling archiving of data between the first and second file systems based at least in part on the user quotas and the archive directories;
wherein a given file is archived from the first file system to the second file system responsive to a link to that file being stored in a corresponding one of the archive directories of the first file system;
wherein controlling archiving of data between the first and second file systems based at least in part on the user quotas and the archive directories comprises preventing archiving of temporary files from the first file system to the second file system by preventing links to the temporary files in the archive directories;
wherein the storage system is further configured to establish archive directory quotas for respective ones of the archive directories, said archive directories including the respective archive directory quotas; and
wherein the method is implemented using at least one processing device comprising a processor coupled to a memory.
10. The method of claim 9 further comprising:
proactively recommending changes to the archive directory quotas based on current sizes of the archive directories relative to the established archive directory quotas.
11. The method of claim 9 wherein the storage system further comprises a plurality of second file systems, and wherein the method further comprises controlling archiving of data between the first file system and each of the second file systems responsive to user quotas established for each of the second file systems.
12. A computer program product comprising a non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing device comprising a processor coupled to a memory causes said at least one processing device:
to configure a storage system to include at least first and second storage tiers associated with respective first and second file systems, the first file system comprising a plurality of archive directories configured for respective ones of a plurality of users;
to establish user quotas identifying corresponding portions of the second file system available to the respective users; and
to control archiving of data between the first and second file systems based at least in part on the user quotas and the archive directories;
wherein a given file is archived from the first file system to the second file system responsive to a link to that file being stored in a corresponding one of the archive directories of the first file system;
wherein controlling archiving of data between the first and second file systems based at least in part on the user quotas and the archive directories comprises preventing archiving of temporary files from the first file system to the second file system by preventing links to the temporary files in the archive directories; and
wherein the storage system is further configured to establish archive directory quotas for respective ones of the archive directories, said archive directories including the respective archive directory quotas.
13. The computer program product of claim 12 wherein the program code when executed by said at least one processing device further causes said at least one processing device:
to proactively recommend changes to the archive directory quotas based on current sizes of the archive directories relative to the established archive directory quotas.
14. The computer program product of claim 12 wherein the storage system further comprises a plurality of second file systems, and wherein the program code when executed by said at least one processing device further causes said at least one processing device to control archiving of data between the first file system and each of the second file systems responsive to user quotas established for each of the second file systems.
15. The computer program product of claim 12 wherein the storage system comprises a plurality of server nodes that are synchronized with one another utilizing a message passing interface (MPI) protocol.
16. The computer program product of claim 12 wherein the storage system comprises a virtual layer of a parallel log-structured file system (PLFS) and wherein the virtual layer comprises PLFS middleware configured to migrate files between the first and second file systems.
17. The computer program product of claim 12 wherein the first file system comprises a hierarchical storage management application programming interface and wherein the storage system is further configured to control movement of data between the first file system and the second file system responsive to information received via the hierarchical storage management application programming interface.
18. The method of claim 9 wherein the storage system comprises a plurality of server nodes that are synchronized with one another utilizing a message passing interface (MPI) protocol.
19. The method of claim 9 wherein the storage system comprises a virtual layer of a parallel log-structured file system (PLFS) and wherein the virtual layer comprises PLFS middleware configured to migrate files between the first and second file systems.
20. The method of claim 9 wherein the first file system comprises a hierarchical storage management application programming interface and wherein the storage system is further configured to control movement of data between the first file system and the second file system responsive to information received via the hierarchical storage management application programming interface.