Methods and systems for using a peripheral device to assist virtual machine IO memory access tracking
SR-IOV (single root IO virtualization) capable PCIe devices can implement virtual functions (VFs) that are assigned to VMs running on a host machine, thereby speeding IO operation by writing directly to the VMs' memory while bypassing the hypervisor managing the VMs. As such, VFs thwart the dirty page tracking that hypervisors use to minimize VM downtime when the VM is migrated between hosts. The SR-IOV PCIe devices can help resolve this problem by maintaining dirty page tracking data for VMs running on the host machine. The SR-IOV PCIe devices bypassing the hypervisor while writing into a memory page of the VM can set the dirty page tracking data to indicate the memory pages that are dirty (i.e., written to by the VF), and can provide access to the dirty page tracking data. The hypervisor can thereby obtain and use the dirty page tracking data.
1. A method comprising:
implementing a first single root IO virtualization (SR-IOV) physical function (PF) that is a network interface card (NIC) PF;
implementing a second SR-IOV PF that is a dirty page tracker PF;
using a virtual function (VF) implemented by the NIC PF to bypass a hypervisor that is used to manage a virtual machine (VM) while writing into a memory page of the VM;
setting dirty page tracking data to indicate that the memory page is dirty in response to the VF writing into the memory page of the VM; and
providing access to the dirty page tracking data via the dirty page tracker PF.
2. The method of claim 1 wherein the NIC PF is configured to set the dirty page tracking data to indicate that the memory page is dirty in response to the NIC PF writing into the memory page.
3. The method of claim 1 wherein the dirty page tracker PF is configured to provide access to the dirty page tracking data via a circular buffer.
4. The method of claim 1 wherein a peripheral component interface express (PCIe) device is configured to implement the NIC PF and the dirty page tracker PF.
5. The method of claim 4 further including issuing a rate limiting command to the VF, wherein
the VF is configured to limit a memory write rate in response to the rate limiting command.
6. The method of claim 1 wherein the dirty page tracker PF is configured to maintain the dirty page tracking data using a bitmap and to provide an atomic read and clear operation for reading the bitmap.
7. The method of claim 1 wherein the dirty page tracking data is maintained in a bitmap, and the dirty page tracker PF provides access to the dirty page tracking data using an atomic read and clear operation.
8. The method of claim 4 wherein a network server implemented by the PCIe device provides access to the dirty page tracking data.
9. The method of claim 1 wherein:
the NIC PF is configured to initiate dirty page tracking for the VM in response to receiving a tracking start command; and
the NIC PF is configured to stop dirty page tracking for the VM in response to receiving a tracking stop command.
10. A system comprising:
a peripheral component interconnect express (PCIe) device configured to:
implement a first single root IO virtualization (SR-IOV) physical function (PF) that is a dirty page tracker PF that provides access to dirty page tracking data;
implement a second SR-IOV physical function that is a network interface card (NIC) PF that includes a virtual function (VF) that is configured to bypass a hypervisor while writing into a memory page of a virtual machine (VM) that is managed by the hypervisor; and
set the dirty page tracking data to indicate that the memory page of the VM is dirty in response to the VF bypassing the hypervisor while writing into the memory page.
11. The system of claim 10 wherein:
the dirty page tracker PF is configured for a host driver running on a host machine to access the dirty page tracking data; and
the VF is configured for a VF driver running on the VM to access the VF.
12. The system of claim 11 wherein the VF is configured to limit a memory write rate in response to receiving a rate limiting command from the PF.
13. The system of claim 11 wherein the dirty page tracker PF is configured to clear the dirty page tracking data in response to a read operation that reads the dirty page tracking data.
14. The system of claim 11 further including a network server implemented by the PCIe device that provides access to the dirty page tracking data.
15. The system of claim 11 wherein:
the NIC PF is configured to initiate dirty page tracking for the VM in response to receiving a tracking start command; and
the NIC PF is configured to stop dirty page tracking for the VM in response to receiving a tracking stop command.
16. A system comprising:
a peripheral component interconnect express (PCIe) device configured to:
implement a first single root IO virtualization (SR-IOV) physical function (PF) configured to implement a virtual function means for bypassing a hypervisor while writing into a memory page of a virtual machine (VM) that uses the hypervisor to manage the VM;
implement a means for indicating the memory page is dirty due to the virtual function means bypassing the hypervisor while writing into the memory page; and
implement a second SR-IOV PF configured to implement an access means for providing access to the means for indicating the memory page is dirty.
17. The system of claim 16 further including a means for throttling the virtual function means based on a state of the means for indicating the memory page is dirty.