Data management and backup of distributed storage environment
In certain systems disclosed herein, a distributed data monitoring and management system is provided that can replicate a distributed storage environment. The distributed data monitoring and management system can intelligently and automatically configure data access nodes to form a structure that matches the distributed storage environment. By matching the structure of the distributed storage environment, the distributed structure of the data may be maintained, enabling the data to be backed up from and/or restored to the distributed storage environment and/or migrated to another distributed storage environment without altering the distribution of the data. Further, embodiments herein enable the transfer of data from a non-distributed environment to a distributed storage environment. Thus, in some cases, an entity can migrate data from a local storage structure to a network-based distributed storage structure.
1. A distributed system for monitoring data stored in a distributed computing environment, the distributed system comprising on or more processors:
a control agent implemented on a control computing system; and
a plurality of data access nodes implemented, respectively, on a plurality of computing systems,
wherein the control agent is programmed to:
access configuration information for a distributed storage environment from a configuration node of the distributed storage environment, wherein the distributed storage environment comprises a plurality of storage nodes;
determine a distributed file system type for the distributed storage environment based at least in part on the configuration information;
determine a total number of storage nodes within the distributed storage environment based at least in part on the configuration information;
make a determination on how to assign the plurality of data access nodes to back up the plurality of storage nodes by distributing the plurality of data access nodes among the plurality of storage nodes based at least in part on the distributed file system type and the total number of storage nodes;
assign the plurality of data access nodes to the plurality of storage nodes based on the determination; and
back up the plurality of storage nodes to the plurality of data access nodes.
2. The distributed system of claim 1 , wherein the plurality of data access nodes comprise a plurality of data agents and the plurality of computing systems are part of a primary storage subsystem.
3. The distributed system of claim 1 , wherein the plurality of data access nodes comprise a plurality of media agents and the plurality of computing systems are part of a secondary storage subsystem.
4. The distributed system of claim 1 , wherein, in response to determining that a data access node from the plurality of data access nodes is inaccessible, the control agent is further programmed to re-distribute the plurality of data access nodes among the plurality of storage nodes.
5. The distributed system of claim 1 , wherein, in response to determining that a data access node from the plurality of data access nodes is inaccessible, the control agent is further programmed to:
allocate a new data access node to the plurality of data access nodes; and
re-distribute the plurality of data access nodes with the new data access node among the plurality of storage nodes.
6. The distributed system of claim 1 , wherein, in response to determining that a storage node from the plurality of storage nodes is inaccessible, the control agent is further programmed to reassign data access nodes from the plurality of data access nodes assigned to the storage node to one or more other storage nodes from the plurality of storage nodes that store at least a subset of data blocks stored on the storage node.
7. The distributed system of claim 1 , wherein, in response to determining that a storage node from the plurality of storage nodes is inaccessible, the control agent is further programmed to reassign data access nodes from the plurality of data access nodes assigned to the storage node to a mirror node that mirrors data stored on the storage node.
8. The distributed system of claim 1 , wherein the control agent is further programmed to distribute the plurality of data access nodes among the plurality of storage nodes based at least in part on size of files stored at the plurality of storage nodes.
9. The distributed system of claim 1 , wherein the plurality of data access nodes comprises a plurality of heterogeneous data access nodes with differing backup capabilities.
10. The distributed system of claim 9 , wherein the control agent is further programmed to distribute the plurality of data access nodes among the plurality of storage nodes by distributing the plurality of heterogeneous data access nodes based at least in part on the differing backup capabilities of the plurality of heterogeneous data access nodes.
11. A computer-implemented method of monitoring data stored in a distributed computing environment, the computer-implemented method comprising:
as implemented by a control agent comprising one or more hardware processors and configured with specific computer-executable instructions, accessing configuration information for a distributed storage environment from a configuration node of the distributed storage environment, wherein the distributed storage environment comprises a plurality of storage nodes;
determining a distributed file system type for the distributed storage environment based at least in part on the configuration information;
determining a total number of storage nodes within the distributed storage environment based at least in part on the configuration information;
determining how to assign a plurality of data access nodes to back up the plurality of storage nodes by distributing the plurality of data access nodes among the plurality of storage nodes based at least in part on the distributed file system type and the total number of storage nodes, wherein the plurality of data access nodes are implemented, respectively, on a plurality of computing systems;
assigning the plurality of data access nodes to the plurality of storage nodes based on the determining how to assign the plurality of data access nodes; and
backing up the plurality of storage nodes to the plurality of data access nodes.
12. The computer-implemented method of claim 11 , wherein the plurality of data access nodes comprises a plurality of data agents and the plurality of computing systems are part of a primary storage subsystem.
13. The computer-implemented method of claim 11 , wherein the plurality of data access nodes comprises a plurality of media agents and the plurality of computing systems are part of a secondary storage subsystem.
14. The computer-implemented method of claim 11 , wherein, in response to determining that a data access node from the plurality of data access nodes is inaccessible, the computer-implemented method further comprises re-distributing the plurality of data access nodes among the plurality of storage nodes.
15. The computer-implemented method of claim 11 , wherein, in response to determining that a data access node from the plurality of data access nodes is inaccessible, the computer-implemented method further comprises:
allocating a new data access node to the plurality of data access nodes; and
re-distributing the plurality of data access nodes with the new data access node among the plurality of storage nodes.
16. The computer-implemented method of claim 11 , wherein, in response to determining that a storage node from the plurality of storage nodes is inaccessible, the computer-implemented method further comprises reassigning data access nodes from the plurality of data access nodes assigned to the storage node to one or more other storage nodes from the plurality of storage nodes that store at least a subset of data blocks stored on the storage node.
17. The computer-implemented method of claim 11 , wherein, in response to determining that a storage node from the plurality of storage nodes is inaccessible, the computer-implemented method further comprises reassigning data access nodes from the plurality of data access nodes assigned to the storage node to a mirror node that mirrors data stored on the storage node.
18. The computer-implemented method of claim 11 , further comprising distributing the plurality of data access nodes among the plurality of storage nodes based at least in part on size of files stored at the plurality of storage nodes.
19. The computer-implemented method of claim 11 , wherein the plurality of data access nodes comprises a plurality of heterogeneous data access nodes with differing backup capabilities.
20. The computer-implemented method of claim 19 , further comprising distributing the plurality of data access nodes among the plurality of storage nodes by distributing the plurality of heterogeneous data access nodes based at least in part on the differing backup capabilities of the plurality of heterogeneous data access nodes.