Using multiple clients for data backup
Various systems and methods for using multiple clients for data backup are described. For example, one method involves accessing a namespace, where the namespace indicates locations of files. At least a portion of the namespace is divided into sets, where the sets correspond to a subset of the files. The files of the subset are accessed by plurality of nodes, and the accessing is based on the sets. Each of the nodes accesses the files of the subset. The nodes are used to store, at data storage, the files of the subset.
1. A method comprising:
accessing a namespace, wherein
the namespace indicates locations of a plurality of files;
dividing, using at least one processor, at least a portion of the namespace into a plurality of sets, wherein
the plurality of sets corresponds to a subset of the plurality of files;
accessing each of the files of the subset, wherein
the accessing is performed based on the plurality of sets,
the accessing comprises accessing the files of the subset using a plurality of streams,
the accessing is performed by a plurality of nodes, and
each of the plurality of streams is configured to access one or more of the plurality of sets;
and
storing the files of the subset, wherein
the storing is performed by the plurality of nodes, and
the storing stores the files of the subset in data storage.
2. The method of claim 1 , wherein
the data storage comprises two or more storage devices, and
during the storing, each of the plurality of nodes stores the files of the subset to the two or more storage devices.
3. The method of claim 2 , further comprising:
restoring the files of the subset, wherein
a first of the two or more storage devices stores files corresponding to a first of the plurality of sets, and
a second of the two or more storage devices stores files corresponding to a second of the plurality of sets.
4. The method of claim 1 , further comprising:
creating the plurality of streams, wherein
a first node of the plurality of nodes is configured to use the first stream to access files of a first set of the plurality of sets, and
a second node of the plurality of nodes is configured to use a second stream of the plurality of streams to access files of a second set of the plurality of sets.
5. The method of claim 4 , wherein
the creating the plurality of streams is based on attributes of the plurality of nodes.
6. The method of claim 5 , further comprising:
if one of the attributes changes, determining whether to discontinue using a node of the plurality of nodes,
in response to a determination to discontinue using a node,
discontinuing using the second node to use the second stream, and
determining whether to modify use of the plurality of streams by the plurality of nodes,
wherein
the modifying is performed during prior to completion of the storing.
7. The method of claim 4 , wherein
the accessing further comprises modifying a first stream of the plurality of streams based on a change in an attribute of one of the nodes.
8. The method of claim 1 , wherein
the dividing generates set information,
the set information indicates that the each of the plurality of sets corresponds to one or more of the plurality of files, and
the accessing is based on the set information.
9. The method of claim 1 , wherein
the dividing is based on file attributes of the files of the subset, and
the file attributes include locality of the files of the subset.
10. The method of claim 9 , wherein
the dividing divides the at least a portion of the namespace such that files in each of the plurality of sets have similar file attributes.
11. The method of claim 1 , wherein
the storing comprises
a first of the plurality of nodes storing first files to a first of two or more storage devices, wherein
the first files correspond to a first of the plurality of sets,
a second of the plurality of nodes storing second files to a second of the two or more storage devices, wherein
the second files correspond to a second of the plurality of sets, and
determining whether the first the plurality of nodes is capable of performing a next storage operation, and
in response to determining that the first of the plurality of nodes is capable of performing the next storage operation, the first of the plurality of nodes storing third files to the first of the two or more storage devices, wherein the third files correspond to a third of the plurality of sets.
12. A system comprising:
one or more processors,
a data cluster, comprising
a plurality of files,
a namespace, wherein
the namespace indicates locations of the plurality of files;
a coordinator module, wherein
the coordinator module is configured to
access the namespace,
divide at least a portion of the namespace into a plurality of sets, wherein
the plurality of sets corresponds to a subset of the plurality of files, and
access the subset of the plurality of files on the data cluster, wherein
the accessing is performed using the plurality of sets,
the accessing comprises accessing the files of the subset using a plurality of streams, wherein
the accessing is performed using a plurality of nodes, and
each of the plurality of streams is configured to access one or more of the plurality of sets,
and
the coordinator module is configured to be executed using the one or more processors; and
plurality of nodes, configured to
store the subset of the plurality of files, wherein
the subset of the plurality of files is stored at data storage.
13. The system of claim 12 , wherein
the data storage comprises two or more storage devices, and
each of the plurality of nodes stores the files of the subset to the two or more storage devices.
14. The system of claim 12 , wherein
the coordinator module is further configured to
create the plurality of streams, wherein
a first node of the plurality of nodes is configured to use the first stream to access files of a first set of the plurality of sets, and
a second node of the plurality of nodes is configured to use a second stream of the plurality of streams to access files of a second set of the plurality of sets.
15. The system of claim 14 , wherein
the coordinator node is configured to create the plurality of streams based on attributes of the plurality of nodes.
16. The system of claim 12 , wherein
the coordinator module is further configured to
if one of the attributes changes, determine whether to discontinue using a node of the plurality of nodes,
in response to a determination to discontinue using a node,
discontinue using the second node to use the second stream, and
determining whether to modify use of the plurality of streams by the plurality of nodes,
wherein the modifying is performed prior to completion of the storing.
17. A computer program product comprising:
a plurality of instructions, comprising
a first set of instructions, executable on a computer system, configured to access a namespace, wherein
the namespace indicates locations of a plurality of files,
a second set of instructions, executable on the computer system, configured to divide at least a portion of the namespace into a plurality of sets, wherein
the plurality of sets corresponds to a subset of the plurality of files,
a third set of instructions, executable on the computer system, configured to access files of the subset, wherein
the accessing is performed based on the plurality of sets,
the accessing comprises accessing the files of the subset using a plurality of streams,
the accessing is performed by a plurality of nodes, and
each of the plurality of streams is configured to access one or more of the plurality of sets,
and
a fourth set of instructions, executable on the computer system, configured to store the files of the subset, wherein
the storing is performed using the plurality of nodes, and
the storing stores the files of the subset in data storage; and
a non-transitory computer-readable storage medium, wherein the instructions are encoded in the non-transitory computer-readable storage medium.
18. The computer program product of claim 17 , wherein the instructions further comprise:
a sixth set of instructions, executable on the computer system, configured to create the plurality of streams, wherein
a first node of the plurality of nodes is configured to use the first stream to access files of a first set of the plurality of sets, and
a second node of the plurality of nodes is configured to use a second stream of the plurality of streams to access files of a second set of the plurality of sets.
19. The computer program product of claim 18 , wherein
the creating the plurality of streams is based on attributes of the plurality of nodes.
20. The computer program product of claim 17 , wherein
the dividing is based on file attributes of the files of the subset, and
the file attributes include locality of the files of the subset.