Efficient search for migration and purge candidates
A computer system configured to store a plurality of files, and a data file, which includes inode and migration or purge information associated with the files. The computer system searches the data file to identify files that are candidates for migration or purge. A plurality of pathnames associated with the files is stored in the data file or a separate data file, and the computer system obtains the pathnames of candidate files from the data file (or separate data file). When candidate files are identified, their inodes are used to look up their pathnames in the plurality of pathnames. The data file may include information about whether a file has been modified and its time of last access. To determine whether a file is a candidate for migration, its entry in the data file is checked to determine whether it has been modified. If it has, it is identified as a migration candidate. A file is determined to be a purge candidate if it has not been modified and its time of last access is older than a predetermined value.
1. A computer system comprising:
a storage configured to store a plurality of files and a first data file, the first data file including inode and migration information associated with a plurality of partitions of one or more files in the plurality of files; and
a processor coupled to the storage, configured to:
search the first data file to identify among the plurality of partitions a candidate partition for migration;
determine a version information associated with the candidate partition based at least in part on previous version information associated with the candidate partition;
migrate the candidate partition, including by storing on a back end storage system as a migrated partition a copy of data comprising the candidate partition;
associate the version information with the migrated partition as stored on the back end storage system, including by updating a second data file stored on the back end storage system to include the version information; and
use the version information to determine, in response to a request to retrieve a requested version of the candidate partition, that the migrated partition as stored on the back end storage system corresponds to the requested version;
wherein the computer system is further configured to store each version of each partition as a separate file on the back end storage system and to include in a respective filename of each stored version of each partition stored on the back end storage system at least a portion of a partition identifier associated with the partition to which the file corresponds.
2. The computer system as recited in claim 1 , wherein the first data file further includes a plurality of pathnames associated with the files, and the computer system is further configured to obtain the pathnames of candidate files from the plurality of pathnames.
3. The computer system as recited in claim 2 , wherein the computer system is configured to obtain the pathnames of candidate files by using inodes associated with the candidate files to obtain the pathnames from the plurality of pathnames.
4. The computer system as recited in claim 3 , wherein the computer system is configured to use the inodes to index into the plurality of pathnames to find pathnames associated with the inodes.
5. The computer system as recited in claim 4 , wherein the first data file includes information about whether any of the plurality of partitions in the computer system has been modified, and the computer system is configured to select migration candidates by identifying among the plurality of partitions one or more partitions that have been modified.
6. The computer system as recited in claim 4 , wherein the first data file includes information about whether any of the plurality of partitions in the computer system has been modified, and time of last access for the plurality of partitions; and wherein the computer system is configured to select purge candidates by identifying among the plurality of partitions at least one partition having a time of access older than a predetermined value.
7. The computer system as recited in claim 1 , wherein the computer system is further configured to store a third data file including a plurality of pathnames associated with the files, and to obtain the pathnames of candidate files from the plurality of pathnames.
8. The computer system as recited in claim 7 , wherein the computer system is configured to obtain the pathnames of candidate files by using inodes associated with the candidate files to index into the plurality of pathnames.
9. The computer system as recited in claim 7 , wherein the third data file includes inodes associated with the pathnames, and the computer system is configured to obtain a pathname by looking up its corresponding inode in the second data file.
10. The computer system as recited in claim 9 , wherein the first data file includes information about whether any of the plurality of partitions in the computer system has been modified and times of last access for the plurality of partitions; and wherein the computer system is configured to select candidates by identifying one or more partitions of the plurality of partitions that have times of access older than a predetermined value.
11. A method for identifying a candidate for migration in a computer system, comprising:
storing on the computer system a plurality of files and a first data file including inode and migration information associated with a plurality of partitions of one or more files in the plurality of files;
searching the first data file to identify among the plurality of partitions the candidate partition for migration;
determining a version information associated with the candidate partition based at least in part on previous version information, associated with the candidate partition;
migrating the candidate partition, including by storing on a back end storage system as a migrated partition a copy of data comprising the candidate partition;
associating the version information with the migrated partition as stored on the back end storage system, including by updating a second data file stored on the back end storage system to include the version information; and
using the version information to determine, in response to a request to retrieve a requested version of the candidate partition, that the migrated partition as stored on the back end storage system corresponds to the requested version;
wherein the computer system is configured to store each version of each partition as a separate file on the back end storage system and to include in a respective filename of each stored version of each partition stored on the back end storage system at least a portion of a partition identifier associated with the partition to which the file corresponds.
12. The method as recited in claim 11 , wherein the first data file includes information about whether any of the plurality of partitions in the computer system has been modified and times of last access, and searching the data file includes finding one or more of the plurality of partitions having times of last access older than a predetermined value.
13. The method as recited in claim 12 , wherein the first data file further includes a plurality of pathnames associated with the files, and comprising obtaining the pathnames of candidate files from the plurality of pathnames.
14. A computer program product for identifying a candidate for migration in a computer system, the computer program product being embodied in a non-transitory computer readable medium and comprising computer instructions for:
storing on the computer system a plurality of files and a first data file including inode and migration information associated with a plurality of partitions of one or more files in the plurality of files;
searching the first data file to identify among the plurality of partitions the candidate partition for migration;
determining a version information associated with the candidate partition based at least in part on previous version information associated with the candidate partition;
migrating the candidate partition, including by storing on a back end storage system as a migrated partition a copy of data comprising the candidate partition;
associating the version information with the migrated partition as stored on the back end storage system, including by updating a second data file stored on the back end storage system to include the version information; and
using the version information to determine, in response to a request to retrieve a requested version of the candidate partition, that the migrated partition as stored on the back end storage system corresponds to the requested version;
wherein the computer program product further comprises computer instructions to store each version of each partition as a separate file on the back end storage system and to include in a respective filename of each stored version of each partition stored on the back end storage system at least a portion of a partition identifier associated with the partition to which the file corresponds.
15. The computer program as recited in claim 14 , wherein the first data file further includes a plurality of pathnames associated with the files, further comprising obtaining the pathnames of candidate files from the plurality of pathnames.
16. The computer system as recited in claim 1 , wherein the computer system is further configured to include in a respective filename of each stored version of each partition stored on the back end storage system at least a portion of the version information associated with the migrated partition to which the file corresponds.
17. The computer system as recited in claim 1 , wherein the computer system comprises a first computer system included in a plurality of computer systems, each of which has access to the back end storage system and each of which uses a corresponding data file stored on the back end storage system to store version information for partitions that have been migrated from that computer system to the back end storage system.
18. The computer system as recited in claim 1 , wherein the each computer system in the plurality of computer systems has at least read access to the respective data files stored on the back end storage system by each of the other computer systems in the plurality.