Data storage system with transparent presentation of file attributes during file system migration
A target data storage system performing a migration operation also responds to client requests by creating and maintaining a metadata cache for caching attributes of a user file as obtained from a source data storage system. A read-type client request is processed by (1) determining whether the metadata cache stores current attribute values, (2) if so, then satisfying the client request using the current values in the metadata cache, and (3) otherwise obtaining current attribute values from the source data storage system and satisfying the client request using the obtained current attribute values, and storing the current values in the metadata cache to be available for use in processing subsequent read-type client requests. A write-type client request is processed forwarding modified attribute values from the client request to the source data storage system for storage therein.
1. A computer-implemented method of operating a data storage system operating as a target data storage system performing a migration operation to migrate a file system to the target data storage system from a separate source data storage system, the method comprising the steps, performed by the target data storage system during the migration operation when a user file is accessible via the target data storage system but not yet migrated to the target data storage system, of:
creating and maintaining a metadata cache for caching attributes of the user file as obtained from the source data storage system;
processing a client request for reading the attributes by (1) first determining whether the metadata cache stores current values for the attributes, (2) if so, then satisfying the client request by using the current values stored in the metadata cache, and (3) otherwise obtaining current values for the attributes from the source data storage system and satisfying the client request by using the current values obtained from the source data storage system, and storing the obtained current values in the metadata cache configured to be available in processing subsequent client requests for reading the attributes; and
processing a client request for modifying the attributes by forwarding modified attribute values from the client request to the source data storage system for storage in the source data storage system,
wherein processing a client request for modifying the attributes includes:
receiving a completion response from the source storage system;
determining whether updated attributes are piggybacked on the completion response; and
if the updated attributes are piggybacked on the completion response, then extracting the updated attributes from the response and storing them in the metadata cache as stored attributes where they are available for satisfying future client requests for reading the attributes, and otherwise invalidating the metadata cache to reflect that currently stored attributes have become outdated and should not be used.
2. The method of claim 1 , wherein the user file is realized by an inode in the target data storage system, and wherein the target data storage system further maintains a vnode being an in-memory copy of the inode and also including a pointer to the metadata cache, the vnode being identifiable based on an inode ID identifying the inode, and wherein processing the client request includes, based on the request identifying a file and the inode, identifying the vnode and then the metadata cache by the pointer in the vnode.
3. The method of claim 1 , wherein memory usage of the metadata cache is controlled by vnode memory management.
4. The method of claim 3 , wherein, when vnode memory usage exceeds a threshold, the vnode is recycled and the associated metadata cache is released.
5. The method of claim 1 , wherein the migration processing includes two interactions with the metadata cache, a first interaction including file importing during the migration and performed in a background fashion, the second interaction being in connection with client request processing, the file importing including importing files from the source storage system and storing or updating file attributes in the metadata cache, the client request processing including the obtaining of the attributes from the source storage system as an initiator forwarding a client request to the source storage system and receiving and processing the response.
6. The method of claim 5 , wherein the file importing further includes creating the metadata cache when commencing the importing of the file, and eventually removing the metadata cache when the file importing is complete.
7. The method of claim 1 , wherein the metadata cache is realized as a migration data store storing all migration-related metadata during the migration operation.
8. The method of claim 7 , wherein the migration data store is realized as a sparse file indexed by inode number and augmented to include file attribute caching.
9. The method of claim 1 , wherein the migration data store is realized using a hidden directory on the target storage system with a single file per migration entity containing cached attributes for a corresponding entity on the source storage system, the hidden directory providing for memory space to be reclaimed as the migration operation progresses.
10. A data storage system comprising secondary storage devices and interface and processing circuitry configured and operative to execute computer program instructions to operate the data storage system as a target data storage system performing a migration operation to migrate a file system to the target data storage system from a separate source data storage system, the operation of the target data storage system including the steps, performed during the migration operation when a user file is accessible via the target data storage system but not yet migrated to the target data storage system, of:
creating and maintaining a metadata cache for caching attributes of the user file as obtained from the source data storage system;
processing a client request for reading the attributes by (1) first determining whether the metadata cache stores current values for the attributes, (2) if so, then satisfying the client request by using the current values stored in the metadata cache, and (3) otherwise obtaining current values for the attributes from the source data storage system and satisfying the client request by using the current values obtained from the source data storage system, and storing the obtained current values in the metadata cache configured to be available in processing subsequent client requests for reading the attributes; and
processing a client request for modifying the attributes by forwarding modified attribute values from the client request to the source data storage system for storage therein,
wherein processing a client request for modifying the attributes includes:
receiving a completion response from the source storage system;
determining whether updated attributes are piggybacked on the completion response; and
if the updated attributes are piggybacked on the completion response, then extracting the updated attributes from the response and storing them in the metadata cache as stored attributes where they are available for satisfying future client requests for reading the attributes, and otherwise invalidating the metadata cache to reflect that currently stored attributes have become outdated and should not be used.
11. The data storage system of claim 10 , wherein the user file is realized by an inode in the target data storage system, and wherein the target data storage system further maintains a vnode being an in-memory copy of the inode and also including a pointer to the metadata cache, the vnode being identifiable based on an mode ID identifying the inode, and wherein processing the client request includes, based on the request identifying a file and the inode, identifying the vnode and then the metadata cache by the pointer in the vnode.
12. The data storage system of claim 10 , wherein memory usage of the metadata cache is controlled by vnode memory management.
13. The data storage system of claim 12 , wherein, when vnode memory usage exceeds a threshold, the vnode is recycled and the associated metadata cache is released.
14. The data storage system of claim 10 , wherein the migration processing includes two interactions with the metadata cache, a first interaction including file importing during the migration and performed in a background fashion, the second interaction being in connection with client request processing, the file importing including importing files from the source storage system and storing or updating file attributes in the metadata cache, the client request processing including the obtaining of the attributes from the source storage system as an initiator forwarding a client request to the source storage system and receiving and processing the response.
15. The data storage system of claim 14 , wherein the file importing further includes creating the metadata cache when commencing the importing of the file, and eventually removing the metadata cache when the file importing is complete.
16. The data storage system of claim 10 , wherein the metadata cache is realized as a migration data store storing all migration-related metadata during the migration operation.
17. The data storage system of claim 16 , wherein the migration data store is realized as a sparse file indexed by Mode number and augmented to include file attribute caching.
18. The data storage system of claim 10 , wherein the migration data store is realized using a hidden directory on the target storage system with a single file per migration entity containing cached attributes for a corresponding entity on the source storage system, the hidden directory providing for memory space to be reclaimed as the migration operation progresses.