Automated development of recovery plans
An automated system monitors network traffic to determine dependencies between different machines. These dependencies can be used to automatically develop a recovery plan for the machines, for example restoring servers in a certain order. This approach can also automatically adjust the recovery plan for changes in system configuration, for example as different servers come online or are taken offline or change their roles.
1. A method for developing a recovery plan for machines in a compute infrastructure, the method comprising:
collecting, by one or more processors, statistics on network connections between machines in the compute infrastructure, wherein collecting statistics on network connections between machines in the compute infrastructure includes sampling virtual machines executing on the machines in the compute infrastructure at periodic intervals, and accessing a list of established and listening ports of the compute infrastructure to produce raw network flow data at a machine level, the raw network flow data collected in an identified format, the identified format including:
for listening ports [local port] [process name]; and
for established connections [source IP address] [source port] [local port] [process name]:
based on the collected statistics, determining, by the one or more processors, dependencies between the machines in the compute infrastructure;
constructing, by the one or more processors, a dependency graph of the dependencies of the machines based on the raw network flow data, wherein a directed edge in the dependency graph represents a dependency between two machines in the compute infrastructure, and wherein the directed edge in the dependency graph represents a strength of the dependency and a confidence level that the directed edge is an actual dependency between two machines in the compute infrastructure; and
based on the dependencies, by the one or more processors, developing a recovery plan for the machines in the compute infrastructure, the recovery plan including an order of recovery for the machines.
2. The method of claim 1 , wherein nodes in the dependency graph represent individual machines in the compute infrastructure.
3. A system for recovering machines in a compute infrastructure, the system comprising:
processors; and
a memory storing instructions that, when executed by at least one processor among the processors, cause the system to perform operations comprising, at least:
collecting statistics on network connections between machines in the compute infrastructure, wherein collecting statistics on network connections between machines in the compute infrastructure includes sampling virtual machines executing on the machines in the compute infrastructure at periodic intervals, and accessing a list of established and listening ports in the compute infrastructure to produce raw network flow data at a machine level the raw network flow data collected in an identified format, the identified format including:
for listening ports [local port] [process name]; and
for established connections [source IP address] [source port] [local port] [process name];
based on the collected statistics, determining dependencies between the machines in the compute infrastructure;
constructing, by the one or more processors, a dependency graph of the dependencies of the machines based on the raw network flow data, wherein a directed edge in the dependency graph represents a dependency between two machines in the compute infrastructure, and wherein the directed edge in the dependency graph represents a strength of the dependency and a confidence level that the directed edge is an actual dependency between two machines in the compute infrastructure; and
based on the dependencies, developing a recovery plan for the machines in the compute infrastructure, the recovery plan including an order of recovery for the machines.
4. The system of claim 3 , wherein nodes in the dependency graph represent individual machines in the compute infrastructure.
5. A non-transitory machine-readable medium including instructions which, when read by a machine, cause the machine to perform operations including, at least:
collecting statistics on network connections between machines in the compute infrastructure, wherein collecting statistics on network connections between machines in the compute infrastructure includes sampling virtual machines executing on the machines in the compute infrastructure at periodic intervals, and accessing a list of established and listening ports in the compute infrastructure to produce raw network flow data at a machine level, the raw network flow data collected in an identified format, the identified format including:
for listening ports [local port] [process name]; and
for established connections [source IP address] [source port] [local port] [process name];
based on the collected statistics, determining dependencies between the machines in the compute infrastructure;
constructing, by the one or more processors, a dependency graph of the dependencies of the machines based on the raw network flow data, wherein a directed edge in the dependency graph represents a dependency between two machines in the compute infrastructure, and wherein the directed edge in the dependency graph represents a strength of the dependency and a confidence level that the directed edge is an actual dependency between two machines in the compute infrastructure and
based on the dependencies, developing a recovery plan for the machines in the compute infrastructure, the recovery plan including an order of recovery for the machines.
6. The medium of claim 5 wherein nodes in the dependency graph represent individual machines in the compute infrastructure.