Automated development of recovery plans
An automated system monitors network traffic to determine dependencies between different machines. These dependencies can be used to automatically develop a recovery plan for the machines, for example restoring servers in a certain order. This approach can also automatically adjust the recovery plan for changes in system configuration, for example as different servers come online or are taken offline or change their roles.
1. A method for developing a recovery plan for machines in a compute infrastructure, the method comprising:
collecting, by one or more processors, statistics on network connections between the machines in the compute infrastructure, the collected statistics being representative of respective incoming network connections and respective outgoing network connections for the machines in the compute infrastructure;
based on the respective incoming network connections and respective outgoing network connections for the machines in the compute infrastructure as represented by the collected statistics, determining dependencies between the machines in the compute infrastructure;
constructing, by the one or more processors, a dependency graph representing the dependencies between the machines in the compute infrastructure, wherein an edge in the dependency graph represents a strength of a dependency between two machines in the compute infrastructure;
based on the dependency graph, by the one or more processors, developing the recovery plan for the machines in the compute infrastructure, the recovery plan including an order of recovery for the machines;
validating the recovery plan by identifying inconsistencies between the dependency graph and the recovery plan; and
outputting an alert based at least on an identified inconsistency between the dependency graph and the recovery plan, the identified inconsistency being between the recovery plan and a dependency order as represented by the dependency graph and corresponding to one or more of the respective incoming network connections for the machines, one or more of the respective outgoing network connections for the machines, or any combination thereof.
2. The method of claim 1 , wherein collecting the statistics on the network connections between the machines in the compute infrastructure includes:
sampling statistics on incoming and outgoing network connections in the compute infrastructure at periodic intervals; and
accessing a list of established and listening ports to produce raw network flow data at a machine level.
3. The method of claim 2 , wherein the raw network flow data is collected in an identified format, the identified format including:
for listening ports [local port] [process name]; and
for established connections [source IP address] [source port] [local port] [process name].
4. The method of claim 1 , wherein nodes in the dependency graph represent individual machines in the compute infrastructure.
5. The method of claim 1 , wherein the edge in the dependency graph represents a confidence level that the edge is an actual dependency between two of the machines in the compute infrastructure.
6. A system for recovering machines in a compute infrastructure, the system comprising:
processors; and
a memory storing instructions that, when executed by at least one processor among the processors, cause the system to perform operations comprising, at least:
collecting statistics on network connections between the machines in the compute infrastructure, the collected statistics being representative of respective incoming network connections and respective outgoing network connections for the machines in the compute infrastructure;
based on the respective incoming network connections and respective outgoing network connections for the machines in the compute infrastructure as represented by the collected statistics, determining dependencies between the machines in the compute infrastructure;
constructing a dependency graph representing the dependencies between the machines in the compute infrastructure, wherein an edge in the dependency graph represents a strength of a dependency between two machines in the compute infrastructure;
based on the dependency graph, by the at least one processor, developing a recovery plan for the machines in the compute infrastructure, the recovery plan including an order of recovery for the machines;
validating the recovery plan by identifying inconsistencies between the dependency graph and the recovery plan; and
outputting an alert based at least on an identified inconsistency between the dependency graph and the recovery plan, the identified inconsistency being between the recovery plan and a dependency order as represented by the dependency graph and corresponding to one or more of the respective incoming network connections for the machines, one or more of the respective outgoing network connections for the machines, or any combination thereof.
7. The system of claim 6 , wherein collecting the statistics on the network connections between the machines in the compute infrastructure includes:
sampling statistics on incoming and outgoing network connections in the compute infrastructure at periodic intervals; and
accessing a list of established and listening ports to produce raw network flow data at a machine level.
8. The system of claim 7 , wherein the operations further comprise collecting the raw network flow data in an identified format, the identified format including:
for listening ports [local port] [process name]; and
for established connections [source IP address] [source port] [local port] [process name].
9. The system of claim 6 , wherein nodes in the dependency graph represent individual machines in the compute infrastructure.
10. The system of claim 6 , wherein the edge in the dependency graph represents a confidence level that the edge is an actual dependency between two of the machines in the compute infrastructure.
11. A non-transitory machine-readable medium including instructions which, when read by a machine, cause the machine to perform operations including, at least:
collecting statistics on network connections between machines in a compute infrastructure, the collected statistics being representative of respective incoming network connections and respective outgoing network connections for the machines in the compute infrastructure;
based on the respective incoming network connections and respective outgoing network connections for the machines in the compute infrastructure as represented by the collected statistics, determining dependencies between the machines in the compute infrastructure;
constructing a dependency graph representing the dependencies between the machines in the compute infrastructure, wherein an edge in the dependency graph represents a strength of a dependency between two machines in the compute infrastructure;
based on the dependency graph, developing a recovery plan for the machines in the compute infrastructure, the recovery plan including an order of recovery for the machines;
validating the recovery plan by identifying inconsistencies between the dependency graph and the recovery plan; and
outputting an alert based at least on an identified inconsistency between the dependency graph and the recovery plan, the identified inconsistency being between the recovery plan and a dependency order as represented by the dependency graph and corresponding to one or more of the respective incoming network connections for the machines, one or more of the respective outgoing network connections for the machines, or any combination thereof.
12. The non-transitory machine-readable medium of claim 11 , wherein collecting the statistics on the network connections between the machines in the compute infrastructure includes:
sampling statistics on incoming and outgoing network connections in the compute infrastructure at periodic intervals; and
accessing a list of established and listening ports to produce raw network flow data at a machine level.
13. The non-transitory machine-readable medium of claim 12 , wherein the operations further comprise collecting the raw network flow data in an identified format, the identified format including:
for listening ports [local port] [process name]; and
for established connections [source IP address] [source port] [local port] [process name].
14. The non-transitory machine-readable medium of claim 11 , wherein nodes in the dependency graph represent individual machines in the compute infrastructure.
15. The non-transitory machine-readable medium of claim 11 , wherein the edge in the dependency graph represents a confidence level that the edge is an actual dependency between two of the machines in the compute infrastructure.