Hardware-assisted tracking of remote direct memory access (RDMA) writes to virtual machine memory

1. A method comprising, during a pre-copy phase of live migrating a paravirtual remote direct memory access (PVRDM) virtual machine (VM) from a source host system to a destination host system:

invoking, by a source hypervisor of the source host system, a first application programming interface (API) exposed by a source host channel adapter (HCA) of the source host system for initiating write tracing of remote direct memory access (RDMA) writes or atomic operations comprising writes received from remote endpoints;

retrieving, by the source hypervisor, a write trace generated by the source HCA in response to the invoking, the write trace including a write trace element identifying a memory region of the PVRDMA VM written to as a result of an RDMA write or atomic operation and a write location within the memory region; and

processing, by the source hypervisor, the write trace element by:

identifying, based on the write trace element, one or more dirty guest memory pages of the PVRDMA VM; and

transmitting data contents of the one or more dirty guest memory pages to a destination hypervisor of the destination host system.

2. The method of claim 1 further comprising, during a switchover phase of the live migrating of the PVRDMA VM:

retrieving and processing any remaining write trace elements not previously processed during the pre-copy phase; and

invoking a second API exposed by the source HCA for disabling the write tracing.

3. The method of claim 1 wherein prior to processing the write trace element, the source hypervisor receives, from the PVRDMA VM, a request to deregister the memory region.

4. The method of claim 3 wherein the source hypervisor ensures that it processes the write trace element before deleting a shadow copy of the memory region in response the request.

5. The method of claim 1 wherein prior to processing the write trace element, the source hypervisor detects completion of a work request element on the source HCA that requests invalidation of the memory region.

6. The method of claim 5 wherein the source hypervisor ensures that it processes the write trace element before deleting a shadow copy of the memory region in response to the detecting.

7. The method of claim 1 wherein the remote endpoints include at least one bare-metal RDMA endpoint.

8. A non-transitory computer readable storage medium having stored thereon program code executable by a source hypervisor of a source host system, the program code embodying a method comprising, during a pre-copy phase of live migrating a paravirtual remote direct memory access (PVRDM) virtual machine (VM) from the source host system to a destination host system:

invoking, by the source hypervisor, a first application programming interface (API) exposed by a source host channel adapter (HCA) of the source host system for initiating write tracing of remote direct memory access (RDMA) writes or atomic operations comprising writes received from remote endpoints;

retrieving, by the source hypervisor, a write trace generated by the source HCA in response to the invoking, the write trace including a write trace element identifying a memory region of the PVRDMA VM written to as a result of an RDMA write or atomic operation and a write location within the memory region; and

processing, by the source hypervisor, the write trace element by:

identifying, based on the write trace element, one or more dirty guest memory pages of the PVRDMA VM; and

transmitting data contents of the one or more dirty guest memory pages to a destination hypervisor of the destination host system.

9. The non-transitory computer readable storage medium of claim 8 wherein the method further comprises, during a switchover phase of the live migrating of the PVRDMA VM:

retrieving and processing any remaining write trace elements not previously processed during the pre-copy phase; and

invoking a second API exposed by the source HCA for disabling the write tracing.

10. The non-transitory computer readable storage medium of claim 8 wherein prior to processing the write trace element, the source hypervisor receives, from the PVRDMA VM, a request to deregister the memory region.

11. The non-transitory computer readable storage medium of claim 10 wherein the source hypervisor ensures that it processes the write trace element before deleting a shadow copy of the memory region in response the request.

12. The non-transitory computer readable storage medium of claim 8 wherein prior to processing the write trace element, the source hypervisor detects completion of a work request element on the source HCA that requests invalidation of the memory region.

13. The non-transitory computer readable storage medium of claim 12 wherein the source hypervisor ensures that it processes the write trace element before deleting a shadow copy of the memory region in response to the detecting.

14. The non-transitory computer readable storage medium of claim 8 wherein the remote endpoints include at least one bare-metal RDMA endpoint.

15. A host system comprising:

a hypervisor including a paravirtual remote direct memory access (PVRDMA) device;

a host channel adapter (HCA);

a PVRDMA virtual machine (VM) using the PVRDMA device for remote direct memory access (RDMA) communication; and

a non-transitory computer readable medium having stored thereon program code that causes the hypervisor to, during a pre-copy phase of live migrating the PVRDMA VM to a destination host system:

invoke a first application programming interface (API) exposed by the HCA for initiating write tracing of RDMA writes or atomic operations comprising writes received from remote endpoints;

retrieve a write trace generated by the HCA in response to the invoking, the write trace including a write trace element identifying a memory region of the PVRDMA VM written to as a result of an RDMA write or atomic operation and a write location within the memory region; and

process the write trace element by:

identifying, based on the write trace element, one or more dirty guest memory pages of the PVRDMA VM; and

transmitting data contents of the one or more dirty guest memory pages to a destination hypervisor of the destination host system.

16. The host system of claim 15 wherein the program code further causes the hypervisor to, during a switchover phase of the live migrating of the PVRDMA VM:

retrieve and process any remaining write trace elements not previously processed during the pre-copy phase; and

invoke a second API exposed by the HCA for disabling the write tracing.

17. The host system of claim 15 wherein prior to processing the write trace element, the hypervisor receives, from the PVRDMA VM, a request to deregister the memory region.

18. The host system of claim 17 wherein the hypervisor ensures that it processes the write trace element before deleting a shadow copy of the memory region in response the request.

19. The host system of claim 15 wherein prior to processing the write trace element, the hypervisor detects completion of a work request element on the HCA that requests invalidation of the memory region.

20. The host system of claim 19 wherein the hypervisor ensures that it processes the write trace element before deleting a shadow copy of the memory region in response to the detecting.

21. The host system of claim 15 wherein the remote endpoints include at least one bare-metal RDMA endpoint.