Using multiple clients for data backup
Various systems and methods for using multiple clients for data backup are described. For example, one method involves accessing a namespace, where the namespace indicates locations of files. At least a portion of the namespace is divided into sets, where the sets correspond to a subset of the files. The files of the subset are accessed by plurality of nodes, and the accessing is based on the sets. Each of the nodes accesses the files of the subset. The nodes are used to store, at data storage, the files of the subset.
1. A method comprising:
accessing a portion of a namespace, wherein
the namespace indicates locations of a plurality of files;
determining size of each file corresponding to the portion of the namespace;
determining file structure of files corresponding to the portion;
determining a storing location of the each file, wherein
the determining the storing location is based on one or more of the size of the each file, or
the file structure of files corresponding to the portion; and
based on a determination of the storing location, dividing, using at least one processor, at least a portion of the namespace into a plurality of sets, wherein
the plurality of sets corresponds to a subset of the plurality of files.
2. The method of claim 1 , wherein
each of the sets is configured to be stored in a container of a plurality of containers, and
the determining the storing location comprises determining one of the plurality of containers where the each file is to be stored.
3. The method of claim 2 , wherein
each of the plurality of containers is configured to store a substantially similar amount of data,
each of the plurality of containers comprises used space and unused space, and
the determining the storing location is further based on an amount of unused space of the each of the containers.
4. The method of claim 3 , wherein
the determining the location comprises reserving a portion of the unused space in each of the plurality of containers.
5. The method of claim 2 , further comprising:
determining that a size of a first portion of the subset has increased, wherein the first portion of the subset corresponds to one of the sets; and
in response to a determination that the size of the first portion of the subset has increased, determining whether to
continue using a first of the plurality of containers to store the first portion of the subset, wherein
the first container stores the first portion of the subset prior to the determination that the size of the first portion of the subset has increased, or
add a new container to the plurality of containers to store the first portion of the subset.
6. The method of claim 5 , wherein
determining that the size of the first portion of the subset has increased comprises one or more of
determining that another file is added to the subset, or
determining that a size of at least one of the files in the subset is increased.
7. The method of claim 2 , further comprising:
determining that a size of a first portion of the subset has decreased, wherein the first portion of the subset corresponds to one of the sets; and
in response to a determination that the size of the first portion of the subset has decreased, determining whether to
continue using a first of the plurality of containers to store the first portion of the subset, wherein
the first container stores the first portion of the subset prior to the determination that the size of the first portion of the subset has decreased, or
store the first portion of the subset in a second of the plurality of containers.
8. The method of claim 7 , wherein
determining that the size of the first portion of the subset has decreased comprises one or more of
determining that a file is deleted from the subset, or
determining that a size of at least one of the files in the subset is decreased.
9. The method of claim 1 , wherein
the determining the location is further based on locality of the files of the subset.
10. The method of claim 1 , wherein
at least one of the accessing, the determining the size, the determining the file structure, the determining the storing location, or dividing is performed prior to receiving a command for replicating the subset of the plurality of files.
11. A system comprising:
one or more processors;
a data cluster, comprising
a plurality of files,
a namespace, wherein
the namespace indicates locations of the plurality of files;
a coordinator module, wherein
the coordinator module is configured to
determine size of each file of a subset of the plurality of files,
determine file structure of the subset,
determine a storing location of the each file, wherein
the determination of the storing location is based on one or more of
the size of the each file, or
the file structure of the subset, and
based on determination of the storing location, divide at least a portion of the namespace into a plurality of sets, wherein
the plurality of sets corresponds to the subset of the plurality of files, and
the coordinator module is configured to be executed using the one or more processors; and
data storage, wherein
the subset of the plurality of files is stored at the data storage.
12. The system of claim 11 , wherein
the coordinator module is configured to store each of the sets in a container of a plurality of containers,
the coordinator module is further configured to determine one of the plurality of containers where the each file is to be stored, and
the data storage comprises the plurality of containers.
13. The system of claim 11 , wherein
each of the plurality of containers is configured to store a substantially similar amount of data,
each of the plurality of containers comprises used space and unused space, and
the coordinator module is further configured to
perform the determination of the storing location further based on an amount of unused space of the each of the containers.
14. The system of claim 13 , wherein
the coordinator module is further configured to determine that a size of a first portion of the subset has increased, wherein
the first portion of the subset corresponds to one of the sets, and
in response to the determination that the size of the first portion of the subset has increased, the coordinator module is further configured to
continue using a first of the plurality of containers to store the first portion of the subset, wherein
the first container stores the first portion of the subset prior to the determination that the size of the first portion of the subset has increased, or
add a new container to the plurality of containers to store the first portion of the subset.
15. The system of claim 13 , wherein the coordinator module is further configured to:
determine that a size of a first portion of the subset has decreased, wherein
the first portion of the subset corresponds to one of the sets, and
in response to the determination that the size of the first portion of the subset has decreased, the coordinator module is further configured to
continue using a first of the plurality of containers to store the first portion of the subset, wherein
the first container stores the first portion of the subset prior to the determination that the size of the first portion of the subset has decreased, or
store the first portion of the subset in a second of the plurality of containers.
16. A computer program product comprising:
a plurality of instructions, comprising
a first set of instructions, executable on a computer system, configured to access a portion of a namespace, wherein
the namespace indicates locations of a plurality of files,
a second set of instructions, executable on the computer system, configured to determine size of each file corresponding to the portion of the namespace,
a third set of instructions, executable on the computer system, configured to determine file structure of files corresponding to the portion,
a fourth set of instructions, executable on the computer system, configured to determine a storing location of the each file, wherein
the determining the storing location is based on one or more of the size of the each file, or
the file structure of files corresponding to the portion, and
a fifth set of instructions, executable on the computer system, configured to, based on determination of the storing location, divide at least a portion of the namespace into a plurality of sets, wherein
the plurality of sets corresponds to a subset of the plurality of files; and
a computer-readable storage medium, wherein the instructions are encoded in the computer-readable storage medium.
17. The computer program product of claim 16 , wherein
each of the sets is configured to be stored in a container of a plurality of containers, and
the determining the storing location comprises determining one of the plurality of containers where the each file is to be stored.
18. The computer program product of claim 17 , wherein the instructions further comprise:
a sixth set of instructions, executable on the computer system, configured to determine that a size of a first portion of the subset has increased, wherein the first portion of the subset corresponds to one of the sets, and
a seventh set of instructions, executable on the computer system, configured to, in response to the determination that the size of the first portion of the subset has increased, determine whether to
continue using a first of the plurality of containers to store the first portion of the subset, wherein
the first container stores the first portion of the subset prior to the determination that the size of the first portion of the subset has increased, or
add a new container to the plurality of containers to store the first portion of the subset.
19. The computer program product of claim 17 , wherein the instructions further comprise:
a sixth set of instructions, executable on the computer system, configured to determine that a size of a first portion of the subset has decreased, wherein the first portion of the subset corresponds to one of the sets, and
a seventh set of instructions, executable on the computer system, configured to, in response to the determination that the size of the first portion of the subset has decreased, determine whether to
continue using a first of the plurality of containers to store the first portion of the subset, wherein
the first container stores the first portion of the subset prior to the determination that the size of the first portion of the subset has decreased, or
store the first portion of the subset in a second of the plurality of containers.
20. The computer program product of claim 16 , wherein
each of the plurality of containers is configured to store a substantially similar amount of data,
each of the plurality of containers comprises used space and unused space, and
the determining the storing location is further based on an amount of unused space of the each of the containers.