Virtual machine fault tolerance

1. In a computer system running at least a first virtual machine (VM) and a second VM on virtualization software, a computer implemented method for the second VM to provide quasi-lockstep fault tolerance for the first VM comprises:

executing a workload on the first VM, the workload having at least one output that is an externally visible event;

concurrently executing the workload on the second VM;

enabling operations of the second VM to diverge from being in lockstep with operations with the first VM without resynchronizing the second VM with the first VM, if the output of the second VM is not diverging from output of the first VM when the output is the externally visible event;

comparing the output that is the externally visible event of the second VM with the output that is the externally visible event of the first VM to determine whether the output of the second VM has diverged from the output of the first VM when the output is the externally visible event; and

executing a resynchronization of the second VM in response to a determination that the output of the second VM has diverged from the output of the first VM, the executing the resynchronization is triggered upon detection that the externally visible event that is output from the second VM does not match the externally visible event that is output from the first VM and includes keeping the externally visible event output from being output externally by the first VM until completion of the resynchronization.

2. The method of claim 1 further comprising reducing occurrences of the output of the second VM diverging from the output of the first VM, including: suppressing the output from the second VM; and

allowing I/O operation from external sources to be performed at the second VM, until physical outputs divergence between the first VM and the second VM.

3. The method of claim 1 further comprising reducing occurrences of the output of the second VM diverging from the output of the first VM, including: recording virtual non-determinism of the first VM except memory access by different virtual processors of the first VM to log entries, for replay of the log entries by the second VM.

4. The method of claim 3 further comprising reducing occurrences of the output of the second VM diverging from the output of the first VM, including: allowing the second VM to continue replay of the log entries even after control divergence in network packets within the virtualized computer system, until a physical output packet diverges between the first virtual machine and the second virtual machine.

5. The method of claim 1 further comprising reducing occurrences of the output of the second VM diverging from the output of the first VM, including: suppressing the output at the second VM; and

allowing I/O operation from external sources to be performed at the second VM, until physical output divergence between the first VM and the second VM.

6. The method of claim 1 further comprising reducing occurrences of the output of the second VM diverging from the output of the first VM, including: allowing I/O operations to continue in the second VM even if I/O packets diverge between the first VM and the second VM, until actual payload data in the I/O packets diverge between the first VM and the second VM.

7. The method of claim 1 further comprising reducing occurrences of the output of the second VM diverging from the output of the first VM, including: scheduling virtual processors of the first VM accessing a shared memory region at different times, for replay of log entries, including a scheduling plan of the virtual processors by the second VM.

8. The method of claim 1 further comprising reducing occurrences of the output of the second VM diverging from the output of the first VM, including: tracking access by a virtual processor of the first VM to a memory page not belonging to the virtual processor, for replay of log entries including the tracked access by the second VM.

9. The method of claim 1 further comprising reducing occurrences of the output of the second VM diverging from the output of the first VM, including: adding read values from access to a shared memory region by virtual processors not corresponding to the first VM to the log entries, for replay of the read values at the second VM based on log entries.

10. The method of claim 1 further comprising reducing occurrences of the output of the second VM diverging from the output of the first VM, including: adding an order of lock acquisition by a guest operating system of the first VM to log entries, for replay of the order of lock acquisitions at the second VM based on the log entries.

11. The method of claim 1 further comprising, responsive to a fault in the first VM, the second VM taking over operation for the first VM.

12. Tangible non-transitory computer-readable storage media having instructions embedded thereon, the computer program implementing a method for operating running a secondary virtual machine (VM) in quasi-lockstep with a primary, VM, wherein the primary VM is executing on a primary virtualized computer system (VCS) and the secondary VM is executing on a secondary VCS, the instructions being executable to implement a method comprising:

executing a workload on the first VM, the workload having at least one output that is an externally visible event;

concurrently executing the workload on the second VM;

enabling operations of the second VM to diverge from being in lockstep with operations with the first VM without resynchronizing the second VM with the first VM, if output of the second VM is not diverging from output of the first VM;

comparing the output of the second VM with the output of the first VM to determine whether the output of the second VM has diverged from the output of the first VM when the output is the externally visible event; and

executing a resynchronization of the second VM in response to a determination that the output second VM has diverged from the output of the first VM, wherein the outputs of the first and second VMs are events which are externally visible relative to the first and second VMs and the resynchronization is triggered upon detection that an externally visible event that is output from the second VM does not match an externally visible event that is output from the first VM and keeping the externally visible event output from the first VM pending until completion of the resynchronization.

13. The tangible non-transitory computer-readable storage media of claim 12 further comprising:

reducing occurrences of the output of the second VM diverging the output of the first VM, including: suppressing the output from the second VM; and

allowing I/O operation from external sources to be performed at the second VM, until physical outputs divergence between the first VM and the second VM.

14. The tangible non-transitory computer-readable storage media of claim 12 further comprising:

reducing occurrences of the output of the second VM diverging from the output of the first VM, including: recording virtual non-determinism of the first VM except memory access by different virtual processors of the first VM to log entries, for replay of the log entries by the second VM.

15. The tangible non-transitory computer-readable storage media of claim 14 further comprising:

reducing occurrences of the output of the second VM diverging from the output of the first VM, including: allowing the second VM to continue replay of the log entries even after control divergence in network packets within the virtualized computer system, until a physical output packet diverges between the first virtual machine and the second virtual machine.

16. The tangible non-transitory computer-readable storage media of claim 12 further comprising:

reducing occurrences of the output of the second VM diverging from the output of the first VM, including: suppressing the output at the second VM; and

allowing I/O operation from external sources to be performed at the second VM, until physical output divergence between the first VM and the second VM.

17. The tangible non-transitory computer-readable storage media of claim 12 further comprising:

reducing occurrences of the output of the second VM diverging from the output of the first VM, including: allowing I/O operations to continue in the second VM even if I/O packets diverge between the first VM and the second VM, until actual payload data in the I/O packets diverge between the first VM and the second VM.

18. The tangible non-transitory computer-readable storage media of claim 12 further comprising:

reducing occurrences of the output of the second VM diverging from the output of the first VM, including: scheduling virtual processors of the first VM accessing a shared memory region at different times, for replay of log entries, including a scheduling plan of the virtual processors by the second VM.

19. The tangible non-transitory computer-readable storage media of claim 12 further comprising:

reducing occurrences of the output of the second VM diverging from the output of the first VM, including: tracking access by a virtual processor of the first VM to a memory page not belonging to the virtual processor, for replay of log entries including the tracked access by the second VM.

20. The tangible non-transitory computer-readable storage media of claim 12 further comprising:

reducing occurrences of the output of the second VM diverging from the output of the first VM, including: adding read values from access to a shared memory region by virtual processors not corresponding to the first VM to the log entries, for replay of the read values at the second VM based on log entries.

21. The tangible non-transitory computer-readable storage media of claim 12 further comprising:

reducing occurrences of the output of the second VM diverging from the output of the first VM, including: adding an order of lock acquisition by a guest operating system of the first VM to log entries, for replay of the order of lock acquisitions at the second VM based on the log entries.

22. The tangible non-transitory computer-readable storage media of claim 12 further comprising:

responsive to a fault in the first VM, the second VM taking over operation for the first VM.

23. A computer system running at least a first virtual machine (VM) and a second VM on virtualization software comprising:

a processor; and

a non-transitory computer readable medium having stored thereon program code that, when executed by the processor, causes the processor to implement a method for the second VM to provide quasi-lockstep fault tolerance for the first VM comprising:

executing a workload on the first VM, the workload having at least one output that is an externally visible event;

concurrently executing the workload on the second VM;

enabling operations of the second VM to diverge from being in lockstep with operations with the first VM without resynchronizing the second VM with the first VM, if the output of the second VM is not diverging from output of the first VM when the output is the externally visible event;

comparing the output that is the externally visible event of the second VM with the output that is the externally visible event of the first VM to determine whether the output of the second VM has diverged from the output of the first VM when the output is the externally visible event; and

executing a resynchronization of the second VM in response to a determination that the output of the second VM has diverged from the output of the first VM, the executing the resynchronization is triggered upon detection that the externally visible event that is output from the second VM does not match the externally visible event that is output from the first VM and includes keeping the externally visible event output from being output externally by the first VM until completion of the resynchronization.

24. The computer system of claim 23 further comprising:

reducing occurrences of the output of the second VM diverging from the output of the first VM, including: suppressing the output from the second VM; and

allowing I/O operation from external sources to be performed at the second VM, until physical outputs divergence between the first VM and the second VM.

25. The computer system of claim 23 further comprising:

reducing occurrences of the output of the second VM diverging from the output of the first VM, including: recording virtual non-determinism of the first VM except memory access by different virtual processors of the first VM to log entries, for replay of the log entries by the second VM.

26. The computer system of claim 25 further comprising:

reducing occurrences of the output of the second VM diverging from the output of the first VM, including: allowing the second VM to continue replay of the log entries even after control divergence in network packets within the virtualized computer system, until a physical output packet diverges between the first virtual machine and the second virtual machine.

27. The computer system of claim 23 further comprising:

reducing occurrences of the output of the second VM diverging from the output of the first VM, including: suppressing the output at the second VM; and

allowing I/O operation from external sources to be performed at the second VM, until physical output divergence between the first VM and the second VM.

28. The computer system of claim 23 further comprising:

reducing occurrences of the output of the second VM diverging from the output of the first VM, including: allowing I/O operations to continue in the second VM even if I/O packets diverge between the first VM and the second VM, until actual payload data in the I/O packets diverge between the first VM and the second VM.

29. The computer system of claim 23 further comprising:

reducing occurrences of the output of the second VM diverging from the output of the first VM, including: scheduling virtual processors of the first VM accessing a shared memory region at different times, for replay of log entries, including a scheduling plan of the virtual processors by the second VM.

30. The computer system of claim 23 further comprising:

reducing occurrences of the output of the second VM diverging from the output of the first VM, including: tracking access by a virtual processor of the first VM to a memory page not belonging to the virtual processor, for replay of log entries including the tracked access by the second VM.

31. The computer system of claim 23 further comprising:

reducing occurrences of the output of the second VM diverging from the output of the first VM, including: adding read values from access to a shared memory region by virtual processors not corresponding to the first VM to the log entries, for replay of the read values at the second VM based on log entries.

32. The computer system of claim 23 further comprising:

reducing occurrences of the output of the second VM diverging from the output of the first VM, including: adding an order of lock acquisition by a guest operating system of the first VM to log entries, for replay of the order of lock acquisitions at the second VM based on the log entries.

33. The computer system of claim 23 further comprising:

responsive to a fault in the first VM, the second VM taking over operation for the first VM.