Dynamic migration of a cloud based distributed file system metadata server
Systems and methods for migrating a metadata server are provided. A source metadata server is transferred to a new metadata server. During the migration process, only one of the metadata servers is active. Over time, data is transferred to the new metadata server until the new metadata server is fully synchronized with the source metadata server. Then metadata service process is terminated on the source metadata server and started on the new metadata server. The new metadata server transitions to read/write and normal file system operations are resumed at the new metadata server.
1. A method for migrating a metadata server associated with a distributed file system, the method comprising:
copying metadata maintained by a source metadata server to a new metadata server, wherein the source metadata server is configured to provide clients with locations of data associated with the metadata in response to requests from the clients, wherein the data is not stored with the metadata;
transitioning the source metadata server to read only after the metadata maintained by the new metadata server is synchronized with the metadata maintained by the source metadata server;
stopping the source metadata server such that the source metadata server no longer responds to the requests from the clients;
transitioning the new metadata server to read/write; and
resuming file system operations with the new metadata server, wherein the new metadata server responds to the requests from the clients.
2. The method of claim 1 , wherein copying metadata maintained by a source metadata server to a new metadata server includes creating a snapshot of the metadata and copying the snapshot to the new metadata server.
3. The method of claim 1 , wherein copying metadata maintained by a source metadata server to a new metadata server includes restoring a backup of the metadata server to the new metadata server.
4. The method of claim 1 , further comprising adding new metadata to the metadata stored by the new metadata server before the source metadata server is stopped.
5. The method of claim 4 , further comprising applying a series of incremental transfers to the new metadata server, wherein the series of incremental transfers each include at least some of the new metadata.
6. The method of claim 1 , further comprising transitioning the source metadata server to read only.
7. The method of claim 1 , further comprising transitioning the source metadata server to read only prior to stopping the source metadata server.
8. The method of claim 1 , further comprising enabling writes on the new metadata server after the new metadata server is synchronized with the source metadata server.
9. The method of claim 1 , wherein stopping the source metadata server includes releasing a global lock associated with the source metadata sever.
10. The method of claim 9 , further comprising acquiring the global lock for the new metadata server such that the source metadata server is no longer available for clients.
11. The method of claim 1 , further comprising discovering the new metadata server by the clients.
12. A method for migrating a metadata server associated with a distributed file system, the method comprising:
copying metadata of a source metadata server to a new metadata server, wherein the source metadata server remains live in the distributed file system, wherein the source metadata server is configured to provide clients with locations of data associated with the metadata in response to requests from the clients, wherein the data is not stored with the metadata;
as new metadata is added to the source metadata server, applying a series of incremental transfers to copy the new metadata to the new metadata server, wherein each incremental transfer includes at least a portion of the new metadata;
flushing a memory of the source metadata server;
synchronizing the new metadata server with the source metadata server by performing a final transfer;
starting the new metadata server with writes enabled and stopping the source metadata server, wherein the new metadata server pervades the clients with the locations of the data stored in the distributed file system in response to the requests from the clients;
acquiring a global lock for the new metadata server and resuming normal file operations with the new metadata server, wherein the source metadata server is no longer available to clients.
13. The method of claim 12 , wherein copying metadata of a source metadata server to a new metadata server includes creating a snapshot of the metadata and copying the snapshot to the new metadata server.
14. The method of claim 12 , wherein copying metadata of a source metadata server to a new metadata server includes restoring a backup of the sou-e metadata server to the new metadata server.
15. The method of claim 12 , further comprising transitioning the source metadata server to read only after applying the series of incremental transfers.
16. The method of claim 15 , further comprising transitioning the source metadata server to read only prior to stopping the source metadata server.
17. The method of claim 12 , further comprising enabling writes on the new metadata server after the new metadata server is synchronized with the source metadata server.
18. The method of claim 12 , further comprising releasing a global lock associated with the source metadata sever.