Late load technique for deploying a virtualization layer underneath a running operating system
View Patent ↗A late load technique deploys a virtualization layer underneath an operating system executing on a node of a network environment to enable the virtualization layer to control the operating system. Binary executable files (binaries) for the virtualization layer may be included in a ring 0 driver loaded in memory of the node with the highest privilege level (e.g., host mode ring 0) needed to control the guest operating system. The ring 0 driver may request allocation of physical memory from the guest operating system for the virtualization layer and thereafter suspend the guest operating system and hardware resources of the node in a deterministic manner. The ring 0 driver may capture architectural states of those resources, which are used to create a virtual machine and virtual devices having initial states that are substantially identical to the states of the operating system and hardware resources at the time of suspension. The ring 0 driver may then load the binaries of the virtualization layer in the allocated memory, bootstrap the virtualization layer to create the virtual machine, initialize the virtual devices and restart the devices in the virtual machine environment with the captured architectural states.
1. A method comprising:
acquiring, by a ring 0 driver executing on a node, physical memory from a guest operating system executing on a central processing unit (CPU) of the node, the CPU coupled to a memory and to a plurality of hardware resources;
suspending, by the ring 0 driver executing on the node, the guest operating system and the hardware resources to a first state, the ring 0 driver, residing in the guest operating system, having a highest privilege level of the guest operating system;
capturing and saving states of the hardware resources by the ring 0 driver executing on the node;
loading, by the ring 0 driver executing on the node, binary executables for a virtualization layer into the acquired physical memory; and
bootstrapping, by the ring 0 driver executing on the node, the virtualization layer to create a virtual machine with an initial state that corresponds to a state of the node executing the guest operating system prior to deployment of the virtualization layer, including the captured states of the hardware resources.
2. The method of claim 1 wherein the virtualization layer provides pass-through virtualization for the plurality of hardware resources excluding security-critical devices that include the CPU and a memory management unit (MMU) coupled to the memory, the MMU supporting a guest page table and a nested page table, and wherein the CPU is coupled to the MMU.
3. The method of claim 2 further comprising:
creating a memory map, wherein memory controlled by the virtual machine is mapped to memory controlled by the virtualization layer excluding the physical memory for the virtualization layer, thereby ensuring that the acquired physical memory is not available to the guest operating system for inspection, modification and execution.
4. The method of claim 3 wherein the virtualization layer modifies the nested page table to prevent access by the guest operating system to the acquired physical memory.
5. The method of claim 4 further comprising:
initializing the hardware resources with a configuration specified by the virtualization layer, wherein the security-critical devices include an input/output MMU (IOMMU); and
creating virtual devices for the hardware resources initialized with the captured states of the hardware resources, being captured architectural states of the hardware resources, prior to bootstrap of the virtualization layer, the virtual devices including a virtual CPU.
6. The method of claim 5 further comprising:
bootstrapping the virtual machine; and
restarting the virtual CPU in the virtual machine initialized with the captured architectural states.
7. The method of claim 1 wherein a guest operating system suspend function is used to capture and save the states of the hardware resources.
8. The method of claim 1 wherein a guest operation system resume function is used to restore the initial state of the virtual machine.
9. The method of claim 7 wherein a core of the CPU remains operating after suspending the guest operating system such that control passed to ring 0 driver.
10. A system comprising:
a memory configured to store a guest operating system including a ring 0 driver having a highest privilege level of the guest operating system and a virtualization layer, the ring 0 driver including binary executables for the virtualization layer; and
a plurality of hardware resources coupled to the memory, the hardware resources including a central processing unit (CPU) adapted to execute the ring 0 driver, the ring 0 driver when executed operable to:
acquire physical memory from the guest operating system for the virtualization layer;
suspend the guest operating system and the hardware resources to a quiescent state;
capture and save states of the hardware resources;
load the binary executables for the virtualization layer into the acquired physical memory; and
bootstrap the virtualization layer to create a virtual machine with an initial state that corresponds to a state of the system prior to deployment of the virtualization layer, including the captured states of the hardware resources.
11. The system of claim 10 wherein the hardware resources further include:
a memory management unit (MMU) coupled to the memory, the MMU supporting a guest page table and a nested page table, wherein the CPU is coupled to the MMU, and wherein the virtualization layer provides pass-through virtualization for the plurality of hardware resources excluding the CPU and MMU.
12. The system of claim 11 wherein the CPU is further adapted to execute the virtualization layer, the virtualization layer when executed operable to:
create a memory map wherein memory controlled by the virtual machine is mapped to memory controlled by the virtualization layer excluding the physical memory for the virtualization layer, thereby ensuring that the acquired physical memory is not available to the guest operating system for modification.
13. The system of claim 12 wherein the virtualization layer, when executed to ensure that the acquired physical memory is not available to the guest operating system for modification, is further operable to modify the nested page table to prevent access by the guest operating system to the acquired physical memory.
14. The system of claim 12 wherein the hardware resources further include:
an input/output MMU (IOMMU) and one or more devices, the IOMMU adapted to couple each device to the memory such that accesses to and from the memory by the device occurs directly through the IOMMU, and wherein the virtualization layer excludes the IOMMU from pass-through virtualization.
15. The system of claim 14 wherein the virtualization layer when executed is further operable to:
initialize the hardware resources with a configuration specified by the virtualization layer, wherein the hardware resources include the CPU, MMU and IOMMU; and
create virtual devices for the hardware resources initialized with the captured states of the hardware resources, being captured architectural states of the hardware resources, prior to bootstrap of the virtualization layer, the virtual devices including a virtual CPU.
16. The system of claim 15 wherein the virtualization layer when executed is further operable to:
bootstrap the virtual machine; and
restart the virtual CPU in the virtual machine with the captured architectural state.
17. The system of claim 10 wherein a guest operating system suspend function is used to capture and save the architectural states of the resources.
18. The system of claim 10 wherein a guest operation system resume function is used to restore the initial state of the virtual machine.
19. The system of claim 17 wherein a core of the CPU remains operating after suspending the guest operating system such that control passed to ring 0 driver.
20. A non-transitory computer readable medium including program instructions for execution within a node including a central processing unit (CPU) coupled to a memory and coupled to a plurality of hardware resources, the program instructions when executed operable to:
acquire, by a ring 0 driver, physical memory from a guest operating system executing on the CPU, the ring 0 driver, residing in the guest operating system, having a highest privilege level of the guest operating system;
suspend, by the ring 0 driver, the guest operating system and the hardware resources to a quiescent state;
capture and save, by the ring 0 driver, states of the hardware resources;
load by the ring 0 driver, binary executables, for a virtualization layer into the acquired physical memory; and
bootstrap, by the ring 0 driver, the virtualization layer to create a virtual machine with an initial state that corresponds to a state of the node executing the guest operating system prior to deployment of the virtualization layer, including the captured states of the hardware resources.