Method for virtual machine migration with artificial intelligence accelerator status validation in virtualization environment
View Patent ↗Systems and methods are disclosed for migrating a virtual machine (VM) having a virtual function that maps resources of an artificial intelligence (AI) accelerator to the VM. A driver for the AI accelerator can generate a checkpoint of VM processes that make calls to the AI accelerator, and can the checkpoint can include a list and configuration of resources mapped to the AI accelerator by the virtual function. The driver can also access the code, data, and memory of the AI accelerator to generate a checkpoint of the AI accelerator status. When the VM is migrated to a new host, then either, or both, of these checkpoint frames can be used to ensure that resuming the VM on a new host having appropriate AI accelerator resources, can be successful resumed on the new host. One or both checkpoint frames can be captured based upon an event, in anticipation of the need to migrate the VM.
1. A computer-implemented method of migrating a source virtual machine (VM-S) that is executing an application that accesses a virtual function of an artificial intelligence (AI) accelerator, the method comprising:
in response to receiving a command to migrate the VM-S and the virtual function, and receiving a selection of a checkpoint of the VM-S and virtual function to use in performing the migration:
recording, then stopping, one or more executing AI tasks of the application;
generating, or selecting, a status of the AI accelerator associated with the virtual function, wherein the status of the AI accelerator comprises an indication of pending instructions to be processed by the AI accelerator and contents of memory within the AI accelerator comprising one or more registers associated with a processor of the AI accelerator; and
transmitting the checkpoint and the status of the AI accelerator to a hypervisor of a target host to generate a target virtual machine (VM-T); and
in response to receiving a notification that the target host validated the checkpoint and AI status, and has generated and configured resources for generating the VM-T, and has loaded the AI accelerator at the target host with data from the AI accelerator status:
migrating the VM-S and the virtual function to the VM-T.
2. The method of claim 1 , further comprising:
in response to receiving a notification that the VM-T has restarted the application and AI tasks, performing a post-migration cleanup of the VM-S and the virtual function, including:
erasing at least a secure memory of the AI accelerator, including any AI inferences, AI models, intermediate results of secure computations, or portions thereof; and
erasing a memory of the VM-S associated with the virtual function, and any calls to the virtual function by the application.
3. The method of claim 1 further comprising:
storing a checkpoint of a state of the VM-S and the virtual function into a storage of a plurality of checkpoints of the VM-S, wherein each checkpoint of the VM-S includes a state of the resources of the VM-S, a state of the application, and a state of the virtual function associated with the resources of AI accelerator.
4. The method of claim 3 , wherein the checkpoint further includes:
a recording of one or more executing AI tasks;
configuration information of resources within the AI accelerator communicatively coupled to the VM-S;
a snapshot of memory of the VM-S, including virtual function scheduling information and communication buffers within the one or more AI accelerators; and
a date and time stamp of the checkpoint.
5. The method of claim 1 , wherein generating a status of the AI accelerator comprises:
storing, in the AI accelerator status, a date and time stamp of the status;
storing, in the AI accelerator status, the contents of the memory within the AI accelerator, including the one or more registers associated with the processor of the AI accelerator, and a cache, queue, or pipeline of the pending instructions to be processed by the AI accelerator; and
generating a hash of the status of the AI accelerator, and digitally signing the status, hash, and date and time stamp.
6. The method of claim 5 , wherein the AI accelerator status further includes one or more register settings indicating one or more other AI accelerators in a cluster of AI accelerators that the AI accelerator is configured to communicate with.
7. The method of claim 1 , wherein validating a signature and freshness of the AI accelerator status comprises:
decrypting the signature of the AI status using a public key of the VM-S;
determining that a date and time stamp of the AI accelerator status is within a threshold date and time range; and
validating a hash of the AI accelerator status.
8. A non-transitory computer readable medium programming with executable instructions that, when executed by a processing system having at least one hardware processor that is communicatively coupled to an artificial intelligence (AI) processor, perform operations of migrating a source virtual machine (VM-S) that is executing an application that accesses a virtual function of an artificial intelligence (AI) accelerator of the system, the operations comprising:
in response to receiving a command to migrate the VM-S and the virtual function, and receiving a selection of a checkpoint of the VM-S and virtual function to use in performing the migration:
recording, then stopping, one or more executing AI tasks of the application;
generating, or selecting, a status of the AI accelerator associated with the virtual function, wherein the status of the AI accelerator comprises an indication of pending instructions to be processed by the AI accelerator and contents of memory within the AI accelerator comprising one or more registers associated with a processor of the AI accelerator; and
transmitting the checkpoint and the status of the AI accelerator to a hypervisor of a target host to generate a target virtual machine (VM-T), and
in response to receiving a notification that the target host validated the checkpoint and AI status, and has generated and configured resources for generating the VM-T, and has loaded the AI accelerator at the target host with data from the AI accelerator status:
migrating the VM-S and the virtual function to the VM-T.
9. The medium of claim 8 , wherein the operations further comprise:
in response to receiving a notification that the VM-T has restarted the application and AI tasks, performing a post-migration cleanup of the VM-S and the virtual function, including:
erasing at least a secure memory of the AI accelerator, including any AI inferences, AI models, intermediate results of secure computations, or portions thereof; and
erasing a memory of the VM-S associated with the virtual function, and any calls to the virtual function by the application.
10. The medium of claim 8 , wherein the operations further comprise:
storing a checkpoint of a state of the VM-S and the virtual function into a storage of a plurality of checkpoints of the VM-S, wherein each checkpoint of the VM-S includes a state of the resources of the VM-S, a state of the application, and a state of the virtual function associated with the resources of AI accelerator.
11. The medium of claim 10 , wherein the checkpoint further includes:
a recording of one or more executing AI tasks;
configuration information of resources within the AI accelerator communicatively coupled to the VM-S;
a snapshot of memory of the VM-S, including virtual function scheduling information and communication buffers within the one or more AI accelerators; and
a date and time stamp of the checkpoint.
12. The medium of claim 8 , wherein generating a status of the AI accelerator comprises:
storing, in the AI accelerator status, a date and time stamp of the status;
storing, in the AI accelerator status, the contents of the memory within the AI accelerator, including the one or more registers associated with the processor of the AI accelerator, and a cache, queue, or pipeline of the pending instructions to be processed by the AI accelerator; and
generating a hash of the status of the AI accelerator, and digitally signing the status, hash, and date and time stamp.
13. The medium of claim 12 , wherein the AI accelerator status further includes one or more register settings indicating one or more other AI accelerators in a cluster of AI accelerators that the AI accelerator is configured to communicate with.
14. The medium of claim 8 , wherein validating a signature and freshness of the AI accelerator status comprises:
decrypting the signature of the AI status using a public key of the VM-S;
determining that a date and time stamp of the AI accelerator status is within a threshold date and time range; and
validating a hash of the AI accelerator status.
15. A system comprising at least one hardware processor coupled to a memory programmed with instructions that, when executed by the at least one hardware processor cause the system to perform operations for migrating a source virtual machine (VM-S) that is executing an application that accesses a virtual function of an artificial intelligence (AI) accelerator, the operations comprising:
in response to receiving a command to migrate the VM-S and the virtual function, and receiving a selection of a checkpoint of the VM-S and virtual function to use in performing the migration:
recording, then stopping, one or more executing AI tasks of the application;
generating, or selecting, a status of the AI accelerator associated with the virtual function, wherein the status of the AI accelerator comprises an indication of pending instructions to be processed by the AI accelerator and contents of memory within the AI accelerator comprising one or more registers associated with a processor of the AI accelerator; and
transmitting the checkpoint and the status of the AI accelerator to a hypervisor of a target host to generate a virtual machine (VM-T); and
in response to receiving a notification that the target host validated the checkpoint and AI status, and has generated and configured resources for generating the VM-T, and has loaded the AI accelerator at the target host with data from the AI accelerator status:
migrating the VM-S and the virtual function to the VM-T.
16. The system of claim 15 , wherein the operations further comprise:
in response to receiving a notification that the VM-T has restarted the application and AI tasks, performing a post-migration cleanup of the VM-S and the virtual function, including:
erasing at least a secure memory of the AI accelerator, including any AI inferences, AI models, intermediate results of secure computations, or portions thereof; and
erasing a memory of the VM-S associated with the virtual function, and any calls to the virtual function by the application.
17. The system of claim 15 , wherein the operations further comprise:
storing a checkpoint of a state of the VM-S and the virtual function into a storage of a plurality of checkpoints of the VM-S, wherein each checkpoint of the VM-S includes a state of the resources of the VM-S, a state of the application, and a state of the virtual function associated with the resources of AI accelerator.
18. The system of claim 17 , wherein the checkpoint further includes:
a recording of one or more executing AI tasks;
configuration information of resources within the AI accelerator communicatively coupled to the VM-S;
a snapshot of memory of the VM-S, including virtual function scheduling information and communication buffers within the one or more AI accelerators; and
a date and time stamp of the checkpoint.
19. The system of claim 15 , wherein generating a status of the AI accelerator comprises:
storing, in the AI accelerator status, a date and time stamp of the status;
storing, in the AI accelerator status, the contents of the memory within the AI accelerator, including the one or more registers associated with the processor of the AI accelerator, and a cache, queue, or pipeline of the pending instructions to be processed by the AI accelerator; and
generating a hash of the status of the AI accelerator, and digitally signing the status, hash, and date and time stamp.
20. The system of claim 19 , wherein the AI accelerator status further includes one or more register settings indicating one or more other AI accelerators in a cluster of AI accelerators that the AI accelerator is configured to communicate with; and
wherein validating a signature and freshness of the AI accelerator status comprises:
decrypting the signature of the AI status using a public key of the VM-S;
determining that a date and time stamp of the AI accelerator status is within a threshold date and time range; and
validating a hash of the AI accelerator status.