Data transfer to a distributed storage environment
In certain systems disclosed herein, a distributed data monitoring and management system is provided that can replicate a distributed storage environment. The distributed data monitoring and management system can intelligently and automatically configure data access nodes to form a structure that matches the distributed storage environment. By matching the structure of the distributed storage environment, the distributed structure of the data may be maintained, enabling the data to be backed up from and/or restored to the distributed storage environment and/or migrated to another distributed storage environment without altering the distribution of the data. Further, embodiments herein enable the transfer of data from a non-distributed environment to a distributed storage environment. Thus, in some cases, an entity can migrate data from a local storage structure to a network-based distributed storage structure.
1. A distributed system for restoring backed up data to a distributed computing environment, the distributed system comprising:
a plurality of data access nodes implemented, respectively, on a plurality of computing systems,
wherein each of the plurality of data access nodes is a software module configured to restore data from a backup repository to a plurality of storage nodes within a distributed storage environment,
wherein the plurality of computing systems are separate and distinct from the distributed storage environment; and
a control agent implemented on a control computing system,
wherein the control agent is programmed to:
access configuration information for the distributed storage environment from a configuration node of the distributed storage environment, wherein the configuration information includes information about how data is distributed throughout the distributed storage environment;
determine a distributed file system type for the distributed storage environment based at least in part on the configuration information;
determine a total number of storage nodes within the distributed storage environment based at least in part on the configuration information,
wherein the distributed storage environment comprises a plurality of storage nodes;
assign the plurality of data access nodes among the plurality of storage nodes based at least in part on the distributed file system type, and the total number of storage nodes,
wherein the assigning the plurality of data access nodes comprises assigning to improve efficiency of data transfer by reducing number of storage nodes that each of the plurality of data access nodes accesses; and
restore data from the backup repository to the plurality of storage nodes.
2. The distributed system of claim 1 , wherein the control agent is further programmed to:
access backup configuration information for a backup system comprising the backup repository; and
configure the plurality of data access nodes to restore the data from the backup repository based at least in part on the backup configuration information.
3. The distributed system of claim 2 , wherein the control agent is further programmed to assign the plurality of data access nodes among a plurality of media agents of the backup system based at least in part on the backup configuration information.
4. The distributed system of claim 3 , wherein distribution of the plurality of data access nodes among the plurality of storage nodes does not match distribution of the plurality of data access nodes among the plurality of media agents.
5. The distributed system of claim 1 , wherein an organizational structure of the data stored at the backup repository does not match an organizational structure of the distributed storage environment.
6. The distributed system of claim 1 , wherein, in response to determining that a data access node from the plurality of data access nodes is inaccessible, the control agent is further programmed to re-distribute the plurality of data access nodes among the plurality of storage nodes.
7. The distributed system of claim 1 , wherein a data access node from the plurality of data access nodes is assigned to restore a subset of data from the data to be restored to multiple storage nodes from the plurality of storage nodes.
8. The distributed system of claim 1 , wherein multiple data access nodes from the plurality of data access nodes are assigned to restore a subset of data from the data to be restored.
9. The distributed system of claim 8 , wherein each data access node from the multiple data access nodes is assigned to restore the subset of data from the data to be restored to a different storage node from the plurality of storage nodes.
10. The distributed system of claim 1 , wherein a data access node from the plurality of data access nodes is assigned to restore a subset of data from the data to be restored to a storage node from the plurality of storage nodes and to a mirror node configured to mirror the storage node.
11. A computer-implemented method of monitoring data stored in a distributed computing environment, the computer-implemented method comprising:
as implemented by a control agent comprising one or more hardware processors and configured with specific computer-executable instructions,
accessing configuration information for a distributed storage environment from a configuration node of the distributed storage environment,
wherein the configuration information includes information about how data is distributed throughout the distributed storage environment;
determining a distributed file system type for the distributed storage environment based at least in part on the configuration information;
determining a total number of storage nodes within the distributed storage environment based at least in part on the configuration information, wherein the distributed storage environment comprises a plurality of storage nodes;
assigning a plurality of data access nodes among the plurality of storage nodes based at least in part on the distributed file system type, and the total number of storage nodes,
wherein the assigning the plurality of data access nodes comprises assigning to improve efficiency of data transfer by reducing number of storage nodes that each of the plurality of data access nodes accesses,
wherein the plurality of data access nodes are implemented, respectively, on a plurality of computing systems,
wherein each of the plurality of data access nodes is a software module configured to restore data from a backup repository to the plurality of storage nodes within a distributed storage environment,
wherein the plurality of computing systems are separate and distinct from the distributed storage environment; and
restoring data from the backup repository to the plurality of storage nodes.
12. The computer-implemented method of claim 11 , further comprising:
accessing backup configuration information for a backup system comprising the backup repository; and
configuring the plurality of data access nodes to restore the data from the backup repository based at least in part on the backup configuration information.
13. The computer-implemented method of claim 12 , further comprising assigning the plurality of data access nodes among a plurality of media agents of the backup system based at least in part on the backup configuration information.
14. The computer-implemented method of claim 13 , wherein distribution of the plurality of data access nodes among the plurality of storage nodes does not match distribution of the plurality of data access nodes among the plurality of media agents.
15. The computer-implemented method of claim 11 , wherein an organizational structure of the data stored at the backup repository does not match an organizational structure of the distributed storage environment.
16. The computer-implemented method of claim 11 , wherein, in response to determining that a data access node from the plurality of data access nodes is inaccessible, the computer-implemented method further comprises re-distributing the plurality of data access nodes among the plurality of storage nodes.
17. The computer-implemented method of claim 11 , further comprising assigning a data access node from the plurality of data access nodes to restore a subset of data from the data to be restored to multiple storage nodes from the plurality of storage nodes.
18. The computer-implemented method of claim 11 , further comprising assigning multiple data access nodes from the plurality of data access nodes to restore a subset of data from the data to be restored.
19. The computer-implemented method of claim 18 , wherein each data access node from the multiple data access nodes is assigned to restore the subset of data from the data to be restored to a different storage node from the plurality of storage nodes.
20. The computer-implemented method of claim 11 , further comprising assigning a data access node from the plurality of data access nodes to restore a subset of data from the data to be restored to a storage node from the plurality of storage nodes and to a mirror node configured to mirror the storage node.