Tracking virtual machine memory modified by a single root I/O virtualization (SR-IOV) device
Techniques for tracking, by a host system, virtual machine (VM) memory modified by a physical input/output (I/O) device that supports I/O virtualization are provided. In one embodiment, a hypervisor of the host system can receive a hardware interrupt from the physical I/O device, where the hardware interrupt indicates that a virtual function (VF) of the physical I/O device has completed a direct memory access (DMA) write to a guest memory space of a VM running on the host system. In response to the hardware interrupt, the hypervisor can invoke a function implemented by a physical function (PF) driver of the physical I/O device, where the function is configured to inspect the VF's state in order to identify memory portions modified by the DMA write. The hypervisor can then mark, in a hypervisor-level page table, one or more memory pages corresponding to the identified memory portions as dirty pages.
1. A method performed by a host system for tracking virtual machine (VM) memory modified by a physical input/output (I/O) device supporting I/O virtualization, the method comprising:
receiving, by a hypervisor of the host system, a hardware interrupt from the physical I/O device, the hardware interrupt indicating that a virtual function (VF) of the physical I/O device has completed a direct memory access (DMA) write to a guest memory space of a VM running on the host system;
invoking, by the hypervisor in response to the hardware interrupt, a function implemented by a physical function (PF) driver of the physical I/O device, the function being configured to inspect the VF's state in order to identify memory portions modified by the DMA write; and
marking, by the hypervisor in a hypervisor-level page table, one or more memory pages corresponding to the identified memory portions as dirty pages.
2. The method of claim 1 wherein the receiving is performed by an interrupt handler of the hypervisor, and wherein the invoking comprises:
activating, by the interrupt handler, an interrupt handler extension of the hypervisor; and
invoking, by the interrupt handler extension, the function implemented by the PF driver.
3. The method of claim 1 wherein the function is a callback function that implements a standard callback interface defined by the hypervisor.
4. The method of claim 1 wherein the function inspects the VF's state by:
accessing a receive (RX) descriptor ring associated with the VF, the RX descriptor ring including a plurality of descriptor entries, each descriptor entry including a buffer address of a memory buffer in the guest memory space of the VM;
traversing the RX descriptor ring to identify descriptor entries corresponding to memory buffers written to as part of the DMA write; and
returning a list of buffer addresses included in the identified descriptor entries.
5. The method of claim 4 further comprising:
converting, by the hypervisor, the list of buffer addresses into memory page numbers for the one or more memory pages prior to performing the marking.
6. The method of claim 4 wherein the RX descriptor ring that is traversed is a shadow copy of another RX descriptor ring maintained in the guest memory space of the VM.
7. The method of claim 1 wherein the invoking and the marking are performed only if the hypervisor is concurrently carrying out a virtualization function that requires tracking of memory pages dirtied by the DMA write.
8. The method of claim 1 wherein the physical I/O device is a Single Root I/O Virtualization (SR-IOV) device.
9. A non-transitory computer readable storage medium having stored thereon program code executable by a host system, the program code embodying a method for tracking virtual machine (VM) memory modified by a physical input/output (I/O) device supporting I/O virtualization, the method comprising:
receiving, by a hypervisor of the host system, a hardware interrupt from the physical I/O device, the hardware interrupt indicating that a virtual function (VF) of the physical I/O device has completed a direct memory access (DMA) write to a guest memory space of a VM running on the host system;
invoking, by the hypervisor in response to the hardware interrupt, a function implemented by a physical function (PF) driver of the physical I/O device, the function being configured to inspect the VF's state in order to identify memory portions modified by the DMA write; and
marking, by the hypervisor in a hypervisor-level page table, one or more memory pages corresponding to the identified memory portions as dirty pages.
10. The non-transitory computer readable storage medium of claim 9 wherein the receiving is performed by an interrupt handler of the hypervisor, and wherein the invoking comprises:
activating, by the interrupt handler, an interrupt handler extension of the hypervisor; and
invoking, by the interrupt handler extension, the function implemented by the PF driver.
11. The non-transitory computer readable storage medium of claim 9 wherein the function is a callback function that implements a standard callback interface defined by the hypervisor.
12. The non-transitory computer readable storage medium of claim 9 wherein the function inspects the VF's state by:
accessing a receive (RX) descriptor ring associated with the VF, the RX descriptor ring including a plurality of descriptor entries, each descriptor entry including a buffer address of a memory buffer in the guest memory space of the VM;
traversing the RX descriptor ring to identify descriptor entries corresponding to memory buffers written to as part of the DMA write; and
returning a list of buffer addresses included in the identified descriptor entries.
13. The non-transitory computer readable storage medium of claim 12 wherein the method further comprises:
converting, by the hypervisor, the list of buffer addresses into memory page numbers for the one or more memory pages prior to performing the marking.
14. The non-transitory computer readable storage medium of claim 12 wherein the RX descriptor ring that is traversed is a shadow copy of another RX descriptor ring maintained in the guest memory space of the VM.
15. The non-transitory computer readable storage medium of claim 9 wherein the invoking and the marking are performed only if the hypervisor is concurrently carrying out a virtualization function that requires tracking of memory pages dirtied by the DMA write.
16. The non-transitory computer readable storage medium of claim 9 wherein the physical I/O device is a Single Root I/O Virtualization (SR-IOV) device.
17. A host system comprising:
a processor;
a hardware virtualization layer running on the processor and supporting execution of a plurality of contexts;
a physical input/output (I/O) device having at least a physical function (PF) and a plurality of virtual functions (VFs) associated with the PF, wherein there is one VF assigned to each of the contexts; and
a non-transitory computer readable medium having stored thereon program code that, when executed by the processor, causes the processor to:
receive, via the hardware virtualization layer, a hardware interrupt from the physical I/O device, the hardware interrupt indicating that a first VF of the physical I/O device has completed a direct memory access (DMA) write to a guest memory space of the context corresponding to the first VF;
invoke, via the hardware virtualization layer in response to the hardware interrupt, a function implemented by a physical function (PF) driver of the physical I/O device, the function being configured to inspect the first VF's state in order to identify memory portions modified by the DMA write; and
mark, via the hardware virtualization layer in a page table maintained by the hardware virtualization layer, one or more memory pages corresponding to the identified memory portions as dirty pages.
18. The host system of claim 17 wherein the hardware virtualization layer includes a hypervisor, and wherein each context in the plurality of context includes a virtual machine (VM).
19. The host system of claim 18 wherein the program code that causes the processor to invoke the function implemented by the PF driver comprises program code that causes the processor to:
activate an interrupt handler extension of the hypervisor; and
invoke the function via the interrupt handler extension.
20. The host system of claim 18 wherein the function is a callback function that implements a standard callback interface defined by the hypervisor.
21. The host system of claim 18 wherein the function inspects the first VF's state by:
accessing a receive (RX) descriptor ring associated with the first VF, the RX descriptor ring including a plurality of descriptor entries, each descriptor entry including a buffer address of a memory buffer in the guest memory space of the context;
traversing the RX descriptor ring to identify descriptor entries corresponding to memory buffers written to as part of the DMA write; and
returning a list of buffer addresses included in the identified descriptor entries.
22. The host system of claim 21 wherein the program code further causes the processor to:
convert, via the hypervisor, the list of buffer addresses into memory page numbers for the one or more memory pages prior to performing the marking.
23. The host system of claim 21 wherein the RX descriptor ring that is traversed is a shadow copy of another RX descriptor ring maintained in the guest memory space of the context.
24. The host system of claim 18 wherein the program code causes the processor to invoke the function and mark the one or more memory pages only if the hypervisor is concurrently carrying out a virtualization function that requires tracking of memory pages in a VM dirtied by the DMA write.
25. The host system of claim 17 wherein the physical I/O device is a Single Root I/O Virtualization (SR-IOV) device.