Protecting large distributed file systems
Responsibility for protecting a distributed file system is split across different client hosts. Each client host is assigned a subset of the distributed file system to backup. An index host is identified to create a common namespace under which different client file indexes generated by the different client hosts during backups of respective subsets of the file system will be maintained. During backups, different client file indexes are received from the different client hosts. The client file indexes include metadata about files in the file system included in the backups. Each of the different client file indexes are associated to the same common namespace to provide a unified view of all files across all subsets of the file system that have been backed up by the different client hosts.
1. A method for protecting a distributed file system comprising:
splitting responsibility for protecting the distributed file system across a plurality of client hosts, each client host being assigned a subset of the distributed file system to backup;
identifying an index host to create a common namespace under which client file indexes generated by the client hosts during backups of respective subsets of the distributed file system by the client hosts will be maintained;
during the backups, receiving, from the client hosts, the client file indexes, wherein the client file indexes comprise metadata about files in the distributed file system included in the backups;
associating each of the client file indexes to the same common namespace; and
allowing the client file indexes to be accessed using a name of the index host to provide a unified view of all files across all subsets of the distributed file system that have been backed up by the client hosts.
2. The method of claim 1 wherein the identifying an index host further comprises:
prompting an administrator to specify a name of the index host.
3. The method of claim 1 wherein the associating each of the client file indexes to the same common namespace further comprises:
storing all the client file indexes under a name of the index host, the name of the index host being different from a name of a client host that generated a client file index.
4. The method of claim 1 further comprising:
receiving, during a backup of a first subset of the distributed file system, a first client file index from a first client host;
receiving, during a backup of a second subset of the distributed file system, a second client file index from a second client host; and
storing the first client file index and the second client file index in a directory corresponding to the name of the index host, wherein the name of the index host is different from a name of the first client host, a name of the second client host, or both the name of the first client host and the name of the second client host.
5. The method of claim 1 further comprising:
maintaining each of the client file indexes under the name of the index host.
6. The method of claim 1 further comprising:
instructing each of the client hosts to index backups of the distributed file system under the index host.
7. A system for protecting a distributed file system, the system comprising: a processor; and memory configured to store one or more sequences of instructions which, when executed by the processor, cause the processor to carry out the steps of:
splitting responsibility for protecting the distributed file system across a plurality of client hosts, each client host being assigned a subset of the distributed file system to backup;
identifying an index host to create a common namespace under which client file indexes generated by the client hosts during backups of respective subsets of the distributed file system by the client hosts will be maintained;
during the backups, receiving, from the client hosts, the client file indexes, wherein the client file indexes comprise metadata about files in the distributed file system included in the backups;
associating each of the client file indexes to the same common namespace; and
allowing the client file indexes to be accessed using a name of the index host to provide a unified view of all files across all subsets of the distributed file system that have been backed up by the client hosts.
8. The system of claim 7 wherein the identifying an index host further comprises:
prompting an administrator to specify a name of the index host.
9. The system of claim 7 wherein the associating each of the client file indexes to the same common namespace further comprises:
storing all the client file indexes under a name of the index host, the name of the index host being different from a name of a client host that generated a client file index.
10. The system of claim 7 wherein the processor further carries out the steps of:
receiving, during a backup of a first subset of the distributed file system, a first client file index from a first client host;
receiving, during a backup of a second subset of the distributed file system, a second client file index from a second client host; and
storing the first client file index and the second client file index in a directory corresponding to the name of the index host, wherein the name of the index host is different from a name of the first client host, a name of the second client host, or both the name of the first client host and the name of the second client host.
11. The system of claim 7 wherein the processor further carries out the steps of:
maintaining each of the client file indexes under the name of the index host.
12. The system of claim 7 wherein the processor further carries out the steps of:
instructing each of the client hosts to index backups of the distributed file system under the index host.
13. A computer program product, comprising a non-transitory computer-readable medium having a computer-readable program code embodied therein, the computer-readable program code adapted to be executed by one or more processors to implement a method for protecting a distributed file system, the method comprising:
splitting responsibility for protecting the distributed file system across a plurality of client hosts, each client host being assigned a subset of the distributed file system to backup;
identifying an index host to create a common namespace under which client file indexes generated by the client hosts during backups of respective subsets of the distributed file system by the client hosts will be maintained;
during the backups, receiving, from the client hosts, the client file indexes, wherein the client file indexes comprise metadata about files in the distributed file system included in the backups;
associating each of the client file indexes to the same common namespace; and
allowing the client file indexes to be accessed using a name of the index host to provide a unified view of all files across all subsets of the distributed file system that have been backed up by the client hosts.
14. The computer program product of claim 13 wherein the identifying an index host further comprises:
prompting an administrator to specify a name of the index host.
15. The computer program product of claim 13 wherein the associating each of the client file indexes to the same common namespace further comprises:
storing all the client file indexes under a name of the index host, the name of the index host being different from a name of a client host that generated a client file index.
16. The computer program product of claim 13 wherein the method further comprises:
receiving, during a backup of a first subset of the distributed file system, a first client file index from a first client host;
receiving, during a backup of a second subset of the distributed file system, a second client file index from a second client host; and
storing the first client file index and the second client file index in a directory corresponding to the name of the index host, wherein the name of the index host is different from a name of the first client host, a name of the second client host, or both the name of the first client host and the name of the second client host.
17. The computer program product of claim 13 wherein the method further comprises:
maintaining each of the client file indexes under the name of the index host.
18. The computer program product of claim 13 wherein the method further comprises:
instructing each of the client hosts to index backups of the distributed file system under the index host.