IOMMU-based direct memory access (DMA) tracking for enabling live migration of virtual machines (VMS) using passthrough physical devices
Techniques for implementing IOMMU-based DMA tracking for enabling live migration of VMs that use passthrough physical devices are provided. In one set of embodiments, these techniques leverage an IOMMU feature known as dirty bit tracking which is available in most, if not all, modern IOMMU implementations. The use of this feature allows for the tracking of passthrough DMA in a manner that is device/vendor/driver agnostic, resulting in a solution that is universally applicable to all passthrough physical devices.
1. A method comprising, at a time of performing a live migration of a virtual machine (VM) that uses a passthrough physical device from a host system to a destination host system:
prior to initiating a pre-copy phase of the live migration, enabling, by a hypervisor of the host system, dirty bit tracking on an input-output memory management unit (IOMMU) of the host system for a memory domain of the VM, wherein the dirty bit tracking causes the IOMMU to, in response to direct memory access writes from the passthrough physical device, set, in one or more input/output (I/O) page tables associated with the memory domain, a dirty bit for each guest memory page in the memory domain dirtied by the passthrough physical device via direct memory access (DMA); and
upon completion of the pre-copy phase:
identifying, in the I/O page tables, one or more guest memory pages with the dirty bit set; and
copying the one or more guest memory pages to the destination host system;
wherein the VM includes an emulated virtual IOMMU and wherein the identifying comprises, for a guest memory page associated with a host physical address that is determined to be dirty, performing a lookup for a quest physical address of the guest memory page from the host physical address.
2. The method of claim 1 further comprising, subsequently to the identifying:
clearing the dirty bit for each of the one or more guest memory pages.
3. The method of claim 1 further comprising, once the one or more guest memory pages have been copied:
disabling the dirty bit tracking on the IOMMU for the memory domain of the VM.
4. The method of claim 1 further comprising:
performing the identifying and the copying in each of a plurality of pre-copy iterations of the pre-copy phase.
5. The method of claim 4 further comprising:
issuing invalidations for one or more I/O translation lookaside buffers of the passthrough physical device in each of the plurality of pre-copy iterations.
6. The method of claim 4 further comprising, during each pre-copy iteration:
dynamically splitting, in the I/O pages tables, dirty guest memory pages that are larger than a predefined size into one or more smaller pages.
7. A non-transitory computer readable storage medium having stored thereon program code executable by a hypervisor of a host system, the program code embodying a method that is executed at a time of performing a live migration of a virtual machine (VM) that uses a passthrough physical device from the host system to a destination host system, the method comprising:
prior to initiating a pre-copy phase of the live migration, enabling dirty bit tracking on an input-output memory management unit (IOMMU) of the host system for a memory domain of the VM, wherein the dirty bit tracking causes the IOMMU to, in response to direct memory access writes from the passthrough physical device, set, in one or more input/output (I/O) page tables associated with the memory domain, a dirty bit for each guest memory page in the memory domain dirtied by the passthrough physical device via direct memory access (DMA); and
upon completion of the pre-copy phase:
identifying, in the I/O page tables, one or more guest memory pages with the dirty bit set; and
copying the one or more guest memory pages to the destination host system;
wherein the VM includes an emulated virtual IOMMU and wherein the identifying comprises, for a guest memory page associated with a host physical address that is determined to be dirty, performing a lookup for a guest physical address of the guest memory page from the host physical address.
8. The non-transitory computer readable storage medium of claim 7 wherein the method further comprises, subsequently to the identifying:
clearing the dirty bit for each of the one or more guest memory pages.
9. The non-transitory computer readable storage medium of claim 7 wherein the method further comprises, once the one or more guest memory pages have been copied:
disabling the dirty bit tracking on the IOMMU for the memory domain of the VM.
10. The non-transitory computer readable storage medium of claim 7 wherein the method further comprises:
performing the identifying and the copying in each of a plurality of pre-copy iterations of the pre-copy phase.
11. The non-transitory computer readable storage medium of claim 10 wherein the method further comprises:
issuing invalidations for one or more I/O translation lookaside buffers of the passthrough physical device in each of the plurality of pre-copy iterations.
12. The non-transitory computer readable storage medium of claim 10 wherein the method further comprises, during each pre-copy iteration:
dynamically splitting, in the I/O pages tables, dirty guest memory pages that are larger than a predefined size into one or more smaller pages.
13. A host system comprising:
a hypervisor;
an input-output memory management unit (IOMMU);
a passthrough physical device; and
a non-transitory computer readable medium having stored thereon program code that causes the hypervisor to, at a time of performing a live migration of a virtual machine (VM) that uses the passthrough physical device from the host system to a destination host system:
prior to initiating a pre-copy phase of the live migration, enable dirty bit tracking on the IOMMU for a memory domain of the VM, wherein the dirty bit tracking causes the IOMMU to, in response to direct memory access writes from the passthrough physical device, set, in one or more input/output (I/O) page tables associated with the memory domain, a dirty bit for each guest memory page in the memory domain dirtied by the passthrough physical device via direct memory access (DMA); and
upon completion of the pre-copy phase:
identify, in the I/O page tables, one or more guest memory pages with the dirty bit set; and
copy the one or more guest memory pages to the destination host system;
wherein the VM includes an emulated virtual IOMMU and wherein the identifying comprises, for a guest memory page associated with a host physical address that is determined to be dirty, performing a lookup for a guest physical address of the quest memory page from the host physical address.
14. The host system of claim 13 wherein the program code further causes the hypervisor to, subsequently to the identifying:
clear the dirty bit for each of the one or more guest memory pages.
15. The host system of claim 13 wherein the program code further causes the hypervisor to, once the one or more guest memory pages have been copied:
disable the dirty bit tracking on the IOMMU for the memory domain of the VM.
16. The host system of claim 13 wherein the program code further causes the hypervisor to:
perform the identifying and the copying in each of a plurality of pre-copy iterations of the pre-copy phase.
17. The host system of claim 16 wherein the program code further causes the hypervisor to:
issue invalidations for one or more I/O translation lookaside buffers of the passthrough physical device in each of the plurality of pre-copy iterations.
18. The host system of claim 16 wherein the program code further causes the hypervisor to, during each pre-copy iteration:
dynamically split, in the I/O pages tables, dirty guest memory pages that are larger than a predefined size into one or more smaller pages.