Distributed file system consistency check
Exemplary methods, apparatuses, and systems include a controller node receiving a request to perform a consistency check of a distributed file system. The controller node transmits, to each of a plurality of nodes, a request for the node to use logical metadata of the distributed file system owned by the node to construct an expected copy of physical metadata mapped to the logical metadata, determine which of the plurality of nodes own actual portions of the physical metadata, transmit corresponding portions of the expected copy of the physical metadata to each of the nodes determined to own actual portions of the physical metadata, and compare expected copies of the physical metadata received from other nodes to the actual physical metadata owned by the node. The controller node receives a result of the comparison from each of the nodes, aggregates the received results, and generates an error report.
1. A computer-implemented method, comprising:
receiving, by a controller node of a plurality of nodes implementing a distributed file system, a request to perform a consistency check of the distributed file system;
transmitting, by the controller node to each of the plurality of nodes other than the controller node, a request for the node to
use logical metadata of the distributed file system owned by the node to construct an expected copy of physical metadata mapped to the logical metadata,
determine which of the plurality of nodes own actual portions of the physical metadata,
transmit corresponding portions of the expected copy of the physical metadata to each of the nodes determined to own actual portions of the physical metadata, and
compare expected copies of the physical metadata received from other nodes to the actual physical metadata owned by the node;
receiving, by the controller node from each of the other nodes, a result of the comparison of the expected copy of the physical metadata to the actual physical metadata;
aggregating, by the controller node, received results to represent a comparison of all portions of the expected copy of the physical metadata to all portions of the actual physical metadata; and
generating, by the controller node, a display reporting any errors represented by the aggregated results.
2. The computer-implemented method of claim 1 , further comprising:
determining, by the controller node, each of the plurality of nodes that own logical metadata of the distributed file system.
3. The computer-implemented method of claim 2 , wherein determining each of the plurality of nodes that own logical metadata of the distributed file system includes submitting a request to a directory service that tracks owners of the logical metadata.
4. The computer-implemented method of claim 1 , further comprising:
receiving, by the controller node from each of the plurality of nodes other than the controller node, an indication that the node has constructed the expected copy of physical metadata from the logical metadata and transmitted corresponding portions of the expected copy of the physical metadata to each of the nodes determined to own actual portions of the physical metadata.
5. The computer-implemented method of claim 4 , wherein each node receiving portions of the expected copy of the physical metadata sums reference counts of the expected copy of the physical metadata for each a physical address of the portion of the expected copy of the physical metadata received from a first node matches the physical address of the expected copy of physical metadata constructed by a second node, and wherein the request for each node to compare accumulated expected copies of the physical metadata received from other nodes to the actual physical metadata owned by the node is transmitted to each node in response to the controller node receiving the indication that the node has constructed the expected copy of physical metadata and transmitted the corresponding portions.
6. The computer-implemented method of claim 1 , further comprising:
determining, by the controller node, that an indication of construction of an expected copy of physical metadata has not been received from a first node;
in response to the determination that the indication has not been received, determining, by the controller node, a second node that now owns logical metadata of the distributed file system that was previously owned by the first node; and
transmitting, by the controller node to the second node, a request to use the logical metadata that was previously owned by the first node and now owned by the second node to construct the expected copy of physical metadata mapped to the logical metadata.
7. The computer-implemented method of claim 1 , wherein each node receiving portions of the expected copy of the physical metadata sums reference counts of the expected copy of the physical metadata for each a physical address of the portion of the expected copy of the physical metadata received from a first node matches the physical address of the expected copy of physical metadata constructed by a second node.
8. A non-transitory computer-readable medium storing instructions, which when executed by a processing device, cause the processing device to perform a method comprising:
receiving, by a controller node of a plurality of nodes implementing a distributed file system, a request to perform a consistency check of the distributed file system;
transmitting, by the controller node to each of the plurality of nodes other than the controller node, a request for the node to
use logical metadata of the distributed file system owned by the node to construct an expected copy of physical metadata mapped to the logical metadata,
determine which of the plurality of nodes own actual portions of the physical metadata,
transmit corresponding portions of the expected copy of the physical metadata to each of the nodes determined to own actual portions of the physical metadata, and
compare expected copies of the physical metadata received from other nodes to the actual physical metadata owned by the node;
receiving, by the controller node from each of the other nodes, a result of the comparison of the expected copy of the physical metadata to the actual physical metadata;
aggregating, by the controller node, received results to represent a comparison of all portions of the expected copy of the physical metadata to all portions of the actual physical metadata; and
generating, by the controller node, a display reporting any errors represented by the aggregated results.
9. The non-transitory computer-readable medium of claim 8 , the method further comprising:
determining, by the controller node, each of the plurality of nodes that own logical metadata of the distributed file system.
10. The non-transitory computer-readable medium of claim 9 , wherein determining each of the plurality of nodes that own logical metadata of the distributed file system includes submitting a request to a directory service that tracks owners of the logical metadata.
11. The non-transitory computer-readable medium of claim 8 , the method further comprising:
receiving, by the controller node from each of the plurality of nodes other than the controller node, an indication that the node has constructed the expected copy of physical metadata from the logical metadata and transmitted corresponding portions of the expected copy of the physical metadata to each of the nodes determined to own actual portions of the physical metadata.
12. The non-transitory computer-readable medium of claim 11 , wherein each node receiving portions of the expected copy of the physical metadata sums reference counts of the expected copy of the physical metadata for each a physical address of the portion of the expected copy of the physical metadata received from a first node matches the physical address of the expected copy of physical metadata constructed by a second node, and wherein the request for each node to compare accumulated expected copies of the physical metadata received from other nodes to the actual physical metadata owned by the node is transmitted to each node in response to the controller node receiving the indication that the node has constructed the expected copy of physical metadata and transmitted the corresponding portions.
13. The non-transitory computer-readable medium of claim 8 , the method further comprising:
determining, by the controller node, that an indication of construction of an expected copy of physical metadata has not been received from a first node;
in response to the determination that the indication has not been received, determining, by the controller node, a second node that now owns logical metadata of the distributed file system that was previously owned by the first node; and
transmitting, by the controller node to the second node, a request to use the logical metadata that was previously owned by the first node and now owned by the second node to construct the expected copy of physical metadata mapped to the logical metadata.
14. The non-transitory computer-readable medium of claim 8 , wherein each node receiving portions of the expected copy of the physical metadata sums reference counts of the expected copy of the physical metadata for each a physical address of the portion of the expected copy of the physical metadata received from a first node matches the physical address of the expected copy of physical metadata constructed by a second node.
15. An controller node comprising:
a processing device; and
a memory coupled to the processing device, the memory storing instructions which, when executed by the processing device, cause the controller node to:
receive a request to perform a consistency check of a distributed file system;
transmitting, to each of a plurality of nodes other than the controller node implementing the distributed file system, a request for the node to
use logical metadata of the distributed file system owned by the node to construct an expected copy of physical metadata mapped to the logical metadata,
determine which of the plurality of nodes own actual portions of the physical metadata,
transmit corresponding portions of the expected copy of the physical metadata to each of the nodes determined to own actual portions of the physical metadata, and
compare expected copies of the physical metadata received from other nodes to the actual physical metadata owned by the node;
receive, from each of the other nodes, a result of the comparison of the expected copy of the physical metadata to the actual physical metadata;
aggregate received results to represent a comparison of all portions of the expected copy of the physical metadata to all portions of the actual physical metadata; and
generate a display reporting any errors represented by the aggregated results.
16. The controller node of claim 15 , wherein the instructions further cause the controller node to:
determine each of the plurality of nodes that own logical metadata of the distributed file system.
17. The controller node of claim 16 , wherein determining each of the plurality of nodes that own logical metadata of the distributed file system includes submitting a request to a directory service that tracks owners of the logical metadata.
18. The controller node of claim 15 , wherein the instructions further cause the controller node to:
receive, from each of the plurality of nodes other than the controller node, an indication that the node has constructed the expected copy of physical metadata mapped to the logical metadata and transmitted corresponding portions of the expected copy of the physical metadata to each of the nodes determined to own actual portions of the physical metadata.
19. The controller node of claim 18 , wherein each node receiving portions of the expected copy of the physical metadata sums reference counts of the expected copy of the physical metadata for each a physical address of the portion of the expected copy of the physical metadata received from a first node matches the physical address of the expected copy of physical metadata constructed by a second node, and wherein the request for each node to compare accumulated expected copies of the physical metadata received from other nodes to the actual physical metadata owned by the node is transmitted to each node in response to the controller node receiving the indication that the node has constructed the expected copy of physical metadata and transmitted the corresponding portions.
20. The controller node of claim 15 , wherein the instructions further cause the controller node to:
determine that an indication of construction of an expected copy of physical metadata has not been received from a first node;
in response to the determination that the indication has not been received, determine a second node that now owns logical metadata of the distributed file system that was previously owned by the first node; and
transmit, to the second node, a request to use the logical metadata that was previously owned by the first node and now owned by the second node to construct the expected copy of physical metadata mapped to the logical metadata.