Data transfer to a distributed storage environment
In certain systems disclosed herein, a distributed data monitoring and management system is provided that can replicate a distributed storage environment. The distributed data monitoring and management system can intelligently and automatically configure data access nodes to form a structure that matches the distributed storage environment. By matching the structure of the distributed storage environment, the distributed structure of the data may be maintained, enabling the data to be backed up from and/or restored to the distributed storage environment and/or migrated to another distributed storage environment without altering the distribution of the data. Further, embodiments herein enable the transfer of data from a non-distributed environment to a distributed storage environment. Thus, in some cases, an entity can migrate data from a local storage structure to a network-based distributed storage structure.
1. A distributed system for restoring backed up data to a distributed computing environment, the distributed system comprising:
a control agent implemented on a control computing system; and
a plurality of data access nodes implemented, respectively, on a plurality of client computing devices, wherein the plurality of data access nodes are configured to access data stored on a plurality of storage nodes within a distributed storage environment,
wherein the control agent is programmed to:
access configuration information for the distributed storage environment from a configuration node of the distributed storage environment;
determine a distributed file system type for the distributed storage environment based at least in part on the configuration information, wherein the configuration node includes configuration information for each of the storage nodes within the distributed storage environment;
determine a total number of storage nodes within the distributed storage environment based at least in part on the configuration information, wherein the storage nodes store application data accessible to the client computing devices; wherein the data storage nodes are storage nodes that are available to store the data to be restored from a backup repository;
assign the plurality of data access nodes to the plurality of storage nodes in a way that spreads out the data from the backup repository among a plurality of storage nodes, wherein the control agent considers the configuration information and the total number of storage nodes in assigning the plurality of data access nodes to the plurality of storage nodes; and
restore data from the backup repository to the plurality of storage nodes.
2. The distributed system of claim 1 , wherein the control agent is further programmed to:
access backup configuration information for a backup system comprising the backup repository; and
configure the plurality of data access nodes to restore the data from the backup repository based at least in part on the backup configuration information.
3. The distributed system of claim 2 , wherein the control agent is further programmed to assign the plurality of data access nodes to a plurality of media agents of the backup system based at least in part on the backup configuration information.
4. The distributed system of claim 3 , wherein the assigning of the plurality of data access nodes to the plurality of storage nodes does not match the assigning of the plurality of data access nodes to the plurality of media agents.
5. The distributed system of claim 1 , wherein an organizational structure of the data stored at the backup repository does not match an organizational structure of the distributed storage environment.
6. The distributed system of claim 1 , wherein, in response to determining that a data access node from the plurality of data access nodes is inaccessible, the control agent is further programmed to re-assign the plurality of data access nodes to the plurality of storage nodes.
7. The distributed system of claim 1 , wherein a data access node from the plurality of data access nodes is configured to restore a subset of data from the data to be restored to multiple storage nodes from the plurality of storage nodes.
8. The distributed system of claim 1 , wherein multiple data access nodes from the plurality of data access nodes are configured to restore a subset of data from the data to be restored.
9. The distributed system of claim 8 , wherein each data access node from the multiple data access nodes is configured to restore the subset of data from the data to be restored to a different storage node from the plurality of storage nodes.
10. The distributed system of claim 1 , wherein a data access node from the plurality of data access nodes is configured to restore a subset of data from the data to be restored to a storage node from the plurality of storage nodes and to a mirror node configured to mirror the storage node.
11. A computer-implemented method of monitoring data stored in a distributed computing environment, the computer-implemented method comprising:
as implemented by a control agent comprising one or more hardware processors and configured with specific computer-executable instructions,
accessing configuration information for a distributed storage environment from a configuration node of the distributed storage environment;
determining a distributed file system type for the distributed storage environment based at least in part on the configuration information;
determining a total number of storage nodes within the distributed storage environment based at least in part on the configuration information, wherein the distributed storage environment comprises a plurality of storage nodes, wherein the configuration node includes configuration information for each of the storage nodes within the distributed storage environment, wherein the data storage nodes are storage nodes that are available to store the data to be restored from a backup repository;
assigning a plurality of data access nodes to the plurality of storage nodes that are compatible with the distributed file system type, wherein the plurality of data access nodes are implemented, respectively, on a plurality of client computing devices, wherein the plurality of data access nodes have access to the data stored on the storage nodes in a way that spreads out the data from the backup repository among a plurality of storage nodes, wherein the control agent considers the configuration information and the total number of storage nodes in assigning the plurality of data access nodes to the plurality of storage nodes; and
restoring data from flail the backup repository to the plurality of storage nodes.
12. The computer-implemented method of claim 11 , further comprising:
accessing backup configuration information for a backup system comprising the backup repository; and
configuring the plurality of data access nodes to restore the data from the backup repository based at least in part on the backup configuration information.
13. The computer-implemented method of claim 12 , further comprising assigning the plurality of data access nodes to a plurality of media agents of the backup system based at least in part on the backup configuration information.
14. The computer-implemented method of claim 13 , wherein the assigning of the plurality of data access nodes to the plurality of storage nodes does not match the assigning of the plurality of data access nodes to the plurality of media agents.
15. The computer-implemented method of claim 11 , wherein an organizational structure of the data stored at the backup repository does not match an organizational structure of the distributed storage environment.
16. The computer-implemented method of claim 11 , wherein, in response to determining that a data access node from the plurality of data access nodes is inaccessible, the computer-implemented method further comprises re-assigning the plurality of data access nodes to the plurality of storage nodes.
17. The computer-implemented method of claim 11 , further comprising configuring a data access node from the plurality of data access nodes to restore a subset of data from the data to be restored to multiple storage nodes from the plurality of storage nodes.
18. The computer-implemented method of claim 11 , further comprising configuring multiple data access nodes from the plurality of data access nodes to restore a subset of data from the data to be restored.
19. The computer-implemented method of claim 18 , wherein each data access node from the multiple data access nodes is configured to restore the subset of data from the data to be restored to a different storage node from the plurality of storage nodes.
20. The computer-implemented method of claim 11 , further comprising configuring a data access node from the plurality of data access nodes to restore a subset of data from the data to be restored to a storage node from the plurality of storage nodes and to a mirror node configured to mirror the storage node.
21. The distributed system of claim 1 wherein the control agent is programmed to assign the plurality of data access nodes to the plurality of storage nodes based at least in part on the distributed file system type and the total number of storage nodes.
22. The computer-implemented method of claim 11 wherein said assigning the plurality of data access nodes to the plurality of storage nodes is based at least in part on the distributed file system type and the total number of storage nodes.