Pro-active self-healing in a distributed file system
A first storage server identifies a second storage server connecting to a cluster of storage servers in a file system. The first storage server locates a file in a replication directory hierarchy in the second storage server. The file has not been accessed by a client application. The first storage server determines, prior to the file being accessed by the client application, that the file on the second storage server has stale data and overwrites the stale data in the file on the second storage server with current data.
1. A method comprising:
requesting, by a processing device on a first server, a link status for a network port related to a second server;
determining, by the processing device, that the second server after a failure is reconnecting to a cluster of servers in a file system in view of the link status for the network port;
determining, in response to determining that the second server is reconnecting to the cluster of servers in the file system, that an image file on the second server comprises stale data;
updating the stale data in the image file on the second server with current data; and
responsive to detecting a request to access the image file, providing an indication during the updating that the image file is partially locked at a portion comprising the stale data.
2. The method of claim 1 , wherein determining that the second server is reconnecting to the cluster of servers in the file system comprises determining that the second server is rebooted.
3. The method of claim 1 , wherein determining that the second server is reconnecting to the cluster of servers in the file system comprises determining that the second server is rejoining the cluster of servers in the file system.
4. The method of claim 1 , wherein determining that the image file on the second server comprises stale data comprises examining individual files in a replication directory hierarchy on the second server.
5. The method of claim 1 , wherein determining that the image file on the second server comprises stale data comprises comparing content of the image file on the second server with content of a corresponding image file on the first server.
6. The method of claim 1 , wherein updating the stale data in the image file comprises:
acquiring a partial lock on the image file comprising the stale data, wherein the partial lock is a lock on a region of the image file;
overwriting the stale data in the locked region with current data;
identifying a write request to modify another region of the image file during the overwriting of the locked region of the image file; and
providing write access to the requested region during the overwriting of the locked region of the image file.
7. The method of claim 1 , wherein updating stale data in the image file comprises:
acquiring a file lock on the image file comprising the stale data; and
overwriting the stale data in the locked image file with current data.
8. A non-transitory computer-readable medium having instructions encoded thereon that, when executed by a processing device, cause the processing device to:
request, by the processing device, a link status for a network port related to a server;
determine that the server after a failure is reconnecting to a cluster of servers in a file system in view of the link status for the network port;
determine, in response to determining that the second server is reconnecting to the cluster of servers in the file system, that an image file on the server comprises stale data;
update the stale data in the image file on the server with current data; and
responsive to detecting a request to access the image file, provide an indication during the updating that the image file is partially locked at a portion comprising the stale data.
9. The non-transitory computer-readable medium of claim 8 , wherein the processing device is to determine that the server is reconnecting to the cluster of servers in the file system by determining that the server is rebooted.
10. The non-transitory computer-readable medium of claim 8 , wherein to determine that the server is reconnecting to the cluster of servers in the file system the processing device is to determine that the server is rejoining the cluster of servers in the file system.
11. The non-transitory computer-readable medium of claim 8 , wherein to determine that the image file on the server comprises stale data the processing device is to examine individual files in the replication directory hierarchy on the server.
12. The non-transitory computer-readable medium of claim 8 , wherein to determine that the image file on the server comprises stale data the processing device is to compare content of the image file on the server with content of a corresponding file on another server.
13. The non-transitory computer-readable medium of claim 8 , wherein to update the stale data in the image file the processing device is to:
acquire a partial lock on the image file comprising the stale data, wherein the partial lock is a lock on a region of the image file;
overwrite the stale data in the locked region with current data;
identify a write request to modify another region of the image file during the overwriting of the locked region of the image file; and
provide write access to the requested region during the overwriting of the locked region of the image file.
14. The non-transitory computer-readable medium of claim 8 , wherein to update the stale data in the image file the processing device is to:
acquire a file lock on the image file comprising the stale data; and
overwrite the stale data in the locked file with current data.
15. A system comprising:
a memory; and
a processing device operatively coupled to the memory, the processing device to:
requesting a link status for a network port related to a server;
determine that the server after a failure is reconnecting to a cluster of servers in a file system in view of the link status for the network port;
determine, in response to determining that the server is reconnecting to the cluster of servers in the file system, that an image file on the server comprises stale data; and
update the stale data in the image file on the server with current data; and
responsive to detecting a request to access the image file, provide an indication during the updating that the image file is partially locked at a portion comprising the stale data.
16. The system of claim 15 , wherein to determine that the server is reconnecting to the cluster of servers in the file system the processing device is to determine at least one of that the server is rejoining the cluster of server in the file system or is rebooted.
17. The system of claim 15 , wherein to determine that the image file on the server comprises stale data the processing device is to examine individual files in the replication directory hierarchy on the server.
18. The system of claim 15 , wherein to determine that the image file on the server comprises stale data the processing device is to compare content of the image file on the server with content of a corresponding file on another server.
19. The system of claim 15 , wherein to update the stale data in the image file the processing device is to:
acquire a partial lock on the image file comprising the stale data, wherein the partial lock is a lock on a region of the image file;
overwrite the stale data in the locked region with current data;
identify a write request to modify another region of the image file during the overwriting of the locked region of the image file; and
provide write access to the requested region during the overwriting of the locked region of the image file.
20. The system of claim 15 , wherein to update the stale data in the image file the processing device is to:
acquire a file lock on the image file comprising the stale data; and
overwrite the stale data in the locked image file using current data.