Methods to automatically correct and improve system recovery and replication processes
A backup recovery and testing method is disclosed. The recovery method restores backed-up systems according to a stored configuration, and collects logs as the restored systems start up. The collected logs are compared with stored baseline logs, by a process which includes parsing the logs into templates and parameters, and identifying anomalies automatically for example by principal component analysis. Anomalies identified may be used to modify the stored configuration.
1. A method of recovering a backed-up computer system using a stored configuration, wherein the stored configuration includes a hardware or virtual hardware resource parameter and an instruction or set of instructions for modification of the backed-up computer system, the method comprising:
creating a virtual environment in which to restore the backed-up computer system, the virtual environment being created according to the stored configuration;
restoring the backed-up computer system from backup or replicated data to the virtual environment to produce a restored system including modifying the backed-up computer system, wherein modifying the backed-up computer system is done in accordance with the instruction or set of instructions for modifying the backed-up system of the stored configuration or the hardware or virtual hardware resource parameter of the stored configuration, the instructions or set of instructions being configured to enable the virtual environment to be compatible with hardware or virtual hardware present in the restored system; then
starting up the restored system in the virtual environment, and collecting logs from the restored system;
automatically comparing the collected logs to stored baseline logs to identify anomalous events corresponding to error conditions;
automatically making changes to the stored configuration according to the identified anomalous events to create a new stored configuration;
repeating the above steps according to the new stored configuration, until no anomalous events corresponding to error conditions are identified.
2. The method as claimed in claim 1 , in which the instructions or set of instructions enable modification of the system including installing.
3. The method as claimed in claim 1 , in which modifying the system includes adding a logging driver.
4. The method as claimed in claim 1 , in which creating the virtual environment includes providing connectivity between multiple systems being restored.
5. The method as claimed in claim 1 , in which creating the virtual environment includes providing connectivity to external networks.
6. The method as claimed in claim 5 , in which connectivity to external networks is limited according to the stored configuration.
7. The method as claimed in claim 1 , in which creating the virtual environment includes creating virtual machines and/or containers into which the system(s) is restored.
8. The method as claimed in claim 1 , including the step of retrieving logs from the backup, and in which the stored baseline logs include the logs retrieved from the backup.
9. The method as claimed in claim 1 , in which the collected logs and baseline logs are parsed to convert the logs into lists of templates and parameters.
10. The method as claimed in claim 9 , in which the DRAIN algorithm is used for parsing.
11. The method as claimed in claim 9 , in which the templates and parameters are grouped into contiguous sets of related events.
12. The method as claimed in claim 11 , in which principal component analysis is used on the sets of events to identify anomalous log entries.
13. A non-transitory computer-readable medium containing instructions which, when executed on a computer, carry out the method of claim 1 .
14. A method of recovering multiple backed-up computer systems from a stored configuration, wherein the stored configuration includes a dependency graph reflecting the dependency of each backed-up computer system of the multiple backed-up computer system, a hardware or virtual hardware resource parameter for each system, and an instruction or set of instructions for modifying the backed-up computer system, the method comprising:
creating a virtual environment in which to restore the backed-up computer systems, the virtual environment being created according to the stored configuration;
for each system, restoring the backed-up computer system from backup files to the virtual environment to produce a restored system including modifying the backed-up computer system, wherein modifying the backed-up computer system is done in accordance with the instruction or set of instructions for modifying the backed-up system of the stored configuration or the hardware or virtual hardware resource parameter of the stored configuration, the instructions or set of instructions being configured to enable the virtual environment to be compatible with hardware or virtual hardware present in the restored system; then
starting up the restored systems in the virtual environment according to the dependency graph forming part of the stored configuration, the dependency graph linking the systems in order of startup, wherein each system is started if and only if any required systems are already started and healthy, and for each system collecting logs from the restored system;
for each restored system, automatically comparing the collected logs to stored baseline logs to identify anomalous events corresponding to error conditions;
for each restored system, making automatic changes to the stored configuration according to the identified anomalous events to create a new stored configuration;
repeating the above steps according to the new stored configuration, until no anomalous events corresponding to error conditions are identified, and when no anomalous events are identified, issuing a notification that the restored system is healthy to trigger startup of any dependent systems.
15. A non-transient computer-readable medium containing instructions which, when executed on a computer, carry out the method of claim 14 .
16. The method as claimed in claim 1 , further comprising the step of searching for a potential solution for the identified anomalous events, wherein the step of automatically making changes to the stored configuration is performed according to the potential solution found for the identified anomalous events.
17. The method as claimed in claim 1 , in which the hardware or virtual hardware resource parameter comprises at least one of memory, CPU resources, virtual devices associated with each machine, drivers and services.
18. The method of claim 14 , in which the hardware or virtual hardware resource parameter comprises at least one of memory, CPU resources, virtual devices associated with each machine, drivers and services.