IP Library › Granted Patent US 10,956,197
Granted Patent B2
US 10,956,197 · App. 15/840,256 · Granted Mar 23, 2021

Virtual machine with an emulator manager for migration of synchronized streams of state data

Inventor: Jennifer Rachel Herbert (Cambridge, GB)
Assignee: CITRIX SYSTEMS, INC.
G06F9/45558G06F3/0604G06F3/067G06F3/0647G06F3/0664G06F9/4856G06F2009/4557G06F2009/45583
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,956,197
App. No.
15/840,256
Granted
Mar 23, 2021
Kind
B2
Abstract

A server includes a hardware platform, a hypervisor platform, and at least one virtual machine operating as an independent guest computing device. The hypervisor includes a memory facilitator, at least one hardware emulator, and an emulator manager. The memory facilitator provides memory for a virtual machine, with the memory having state data associated therewith at a current location within the virtual machine. The at least one hardware emulator provides at least one set of hardware resources for the virtual machine, with the at least one set of hardware resources having state data associated therewith at the current location within the virtual machine. The emulator manager coordinates transfer of the respective state data from the current location to a different location, and tracks progress of the transfer of the respective state data to the different location.

Claims (74)

1. A server comprising:

a hardware platform comprising hardware resources with at least a portion of said hardware resources to be allocated as virtualized resources;

a hypervisor being provided based on execution of code instructions by said hardware platform; and

at least one virtual machine operating as an independent guest computing device, with at least a portion of the virtualized resources being allocated by said hypervisor to said at least one virtual machine;

said hypervisor comprising:

a memory facilitator to provide memory from the hardware resources within said hardware platform for said at least one virtual machine, with the memory having state data associated therewith at a current location within said at least one virtual machine,

at least two hardware emulators, each hardware emulator to provide at least one set of hardware resources from said hardware platform for said at least one virtual machine, with the at least one set of hardware resources having state data associated therewith at the current location within said at least one virtual machine,

a toolstack to control said hypervisor including generation of a start state data transfer request, with the start state data transfer request including a list of resources having state data to be transferred;

an emulator manager configured to:

receive from said tool stack the start state data transfer request to coordinate transfer of respective state data associated with said memory facilitator and state data associated with at least one of the hardware emulators of the at least two hardware emulators including in the list of resources,

provide the start state data transfer request to said memory facilitator and said at least one of the hardware emulators to start transferring the respective state data from the current location to a different location with respect to the current location,

receive a progress report from said memory facilitator and said at least one of the hardware emulators providing a progress level on transfer of the respective state data from the current location to the different location,

track the progress level on transfer of the respective state data from the current location to the different location,

stop execution of said at least one virtual machine and provide a pause request to said memory facilitator and said at least one of the hardware emulators to stop executing when the progress level on transfer of the respective state data reaches a predetermined progress level,

receive a confirm pause request reply from said memory facilitator and said at least one of the hardware emulators indicating that said memory facilitator and said at least one of the hardware emulators are at a point to stop executing,

provide a suspend virtual machine confirmation to said toolstack indicating that said at least one virtual machine has stopped executing, and

provide a suspended complete command to said memory facilitator and said at least one of the hardware emulators so that remaining respective state data is transferred from the current location to the different location,

receive a state data transfer completed command from said memory facilitator and said at least one of the hardware emulators when the transfer of the respective state data from the current location to the different location is completed,

provide the state data transfer completed command to said toolstack, and

stop coordinating transfer and tracking of the respective state data.

2. The server according to claim 1 wherein the respective state data from said memory facilitator and said at least one of the hardware emulators are transferred in parallel to the different location.

3. The server according to claim 1 wherein a majority of the respective state data are transferred while said at least one virtual machine is executing.

4. The server according to claim 1 wherein the predetermined progress level corresponds to when an amount of the respective state data still needing to be transferred while said at least one virtual machine is executing is not being reduced.

5. The server according to claim 1 wherein the respective state data are iteratively transferred to the different location; and wherein the predetermined progress level is based on a number of iterations having been performed reaching a predetermined number.

6. The server according to claim 5 wherein the iterative transfer of the respective state data comprises iteratively writing pages to the different location, and if any initial pages that were written to the different location were modified after being sent, then the modified pages are re-sent to the different location.

7. The server according to claim 6 wherein said hypervisor further comprises a modified page tracker configured to track the modified pages and coordinate the same with said memory facilitator and said at least one of the hardware emulators.

8. The server according to claim 1 wherein the different location receiving the transfer of the respective state data is at least one of a file and disk within the server.

9. The server according to claim 1 wherein the different location receiving the transfer of the respective state data is a different physical machine.

10. A method for transferring state data of at least one virtual machine operating within a server, wherein the server comprising:

a hardware platform comprising hardware resources with at least a portion of the hardware resources to be allocated as virtualized resources;

a hypervisor being provided based on execution of code instructions by the hardware platform; and

the at least one virtual machine operating as an independent guest computing device, with at least a portion of the virtualized resources being allocated by the hypervisor to the at least one virtual machine;

wherein the hypervisor comprising: a memory facilitator, at least two hardware emulators, an emulator manager and a toolstack, wherein the method comprising:

operating the memory facilitator to provide memory from the hardware resources within the hardware platform for the at least one virtual machine, with the memory having state data associated therewith at a current location within the at least one virtual machine;

operating the at least two hardware emulators, each hardware emulator to provide at least one set of hardware resources from the hardware platform for the at least one virtual machine, with the at least one set of hardware resources having state data associated therewith at the current location within the at least one virtual machine;

operating the toolstack to control the hypervisor including generation of a start state data transfer request, with the start state data transfer request including a list of resources having state data to be transferred; and

operating the emulator manager to perform the following:

receive from the tool stack the start state data transfer request to coordinate transfer of respective state data associated with the memory facilitator and state data associated with at least one of the hardware emulators of the at least two hardware emulators including in the list of resources,

provide the start state data transfer request to the memory facilitator and the at least one of the hardware emulators to start transferring the respective state data from the current location to a different location with respect to the current location,

receive a progress report from the memory facilitator and the at least one of the hardware emulators providing a progress level on transfer of the respective state data from the current location to the different location,

track the progress level on transfer of the respective state data from the current location to the different location,

stop execution of the at least one virtual machine and provide a pause request to the memory facilitator and the at least one of the hardware emulators to stop executing when the progress level on transfer of the respective state data reaches a predetermined progress level,

receive a confirm pause request reply from the memory facilitator and the at least one of the hardware emulators indicating that the memory facilitator and the at least one of the hardware emulators are at a point to stop executing,

provide a suspend virtual machine confirmation to the toolstack indicating that the at least one virtual machine has stopped executing

provide a suspended complete command to the memory facilitator and the at least one of the hardware emulators so that remaining respective state data is transferred from the current location to the different location,

receive a state data transfer completed command from the memory facilitator and the at least one of the hardware emulators when the transfer of the respective state data from the current location to the different location is completed,

provide the state data transfer completed command to the toolstack, and

stop coordinating transfer and tracking of the respective state data.

11. The method according to claim 10 wherein the respective state data from the memory facilitator and the at least one of the hardware emulators are transferred in parallel to the different location.

12. The method according to claim 10 wherein a majority of the respective state data are transferred while the virtual machine is executing.

13. The method according to claim 10 wherein the predetermined progress level corresponds to when an amount of the respective state data still needing to be transferred while the virtual machine is executing is not being significantly reduced.

14. The method according to claim 10 wherein the respective state data are iteratively transferred to the different location; and wherein the predetermined progress level is based on a number of iterations having been performed reaching a predetermined number.

15. A non-transitory computer readable medium storing a plurality of computer executable instructions for at least one virtual machine in a server, wherein the server comprising:

a hardware platform comprising hardware resources with at least a portion of the hardware resources to be allocated as virtualized resources;

a hypervisor being provided based on execution of code instructions by the hardware platform; and,

the at least one virtual machine operating as an independent guest computing device, with at least a portion of the virtualized resources being allocated by the hypervisor to the at least one virtual machine;

wherein the hypervisor comprising: a memory facilitator, at least two hardware emulators, an emulator manager and a toolstack, wherein executing of the plurality of computer executable instructions causing the at least one virtual machine to perform steps comprising:

operating the memory facilitator to provide memory from the hardware resources within the hardware platform for the at least one virtual machine, with the memory having state data associated therewith at a current location within the at least one virtual machine;

operating the at least two hardware emulators, each hardware emulator to provide at least one set of hardware resources from the hardware platform for the at least one virtual machine, with the at least one set of hardware resources having state data associated therewith at the current location within the at least one virtual machine;

operating the toolstack to control the hypervisor including generation of a start state data transfer request, with the start state data transfer request including a list of resources having state data to be transferred; and

operating the emulator manager to perform the following:

receive from the tool stack the start state data transfer request to coordinate transfer of respective state data associated with the memory facilitator and state data associated with at least one of the hardware emulators of the at least two hardware emulators including in the list of resources,

provide the start state data transfer request to the memory facilitator and the at least one of the hardware emulators to start transferring the respective state data from the current location to a different location with respect to the current location,

receive a progress report from the memory facilitator and the at least one of the hardware emulators providing a progress level on transfer of the respective state data from the current location to the different location,

track the progress level on transfer of the respective state data from the current location to the different location,

stop execution of the at least one virtual machine and provide a pause request to the memory facilitator and the at least one of the hardware emulators to stop executing when the progress level on transfer of the respective state data reaches a predetermined progress level,

receive a confirm pause request reply from the memory facilitator and the at least one of the hardware emulators indicating that the memory facilitator and the at least one of the hardware emulators are at a point to stop executing,

provide a suspend virtual machine confirmation to the toolstack indicating that the at least one virtual machine has stopped executing,

provide a suspended complete command to the memory facilitator and the at least one of the hardware emulators so that remaining respective state data is transferred from the current location to the different location,

receive a state data transfer completed command from the memory facilitator and the at least one of the hardware emulators when the transfer of the respective state data from the current location to the different location is completed,

provide the state data transfer completed command to the toolstack, and

stop coordinating transfer and tracking of the respective state data.

16. The non-transitory computer readable medium according to claim 15 wherein the respective state data from the memory facilitator and the at least one of the hardware emulators are transferred in parallel to the different location.

17. The non-transitory computer readable medium according to claim 15 wherein a majority of the respective state data are transferred while the virtual machine is executing.

Assignments (9)
PATENT SECURITY AGREEMENT Recorded Aug 15, 2025
From: CLOUD SOFTWARE GROUP, INC.; CITRIX SYSTEMS, INC.
To: WILMINGTON TRUST, NATIONAL ASSOCIATION, AS NOTES COLLATERAL AGENT
Reel/Frame 072488/0172 →
SECURITY INTEREST Recorded May 24, 2024
From: CLOUD SOFTWARE GROUP, INC. (F/K/A TIBCO SOFTWARE INC.); CITRIX SYSTEMS, INC.
To: WILMINGTON TRUST, NATIONAL ASSOCIATION, AS NOTES COLLATERAL AGENT
Reel/Frame 067662/0568 →
PATENT SECURITY AGREEMENT Recorded Apr 14, 2023
From: CLOUD SOFTWARE GROUP, INC. (F/K/A TIBCO SOFTWARE INC.); CITRIX SYSTEMS, INC.
To: WILMINGTON TRUST, NATIONAL ASSOCIATION, AS NOTES COLLATERAL AGENT
Reel/Frame 063340/0164 →
RELEASE AND REASSIGNMENT OF SECURITY INTEREST IN PATENT (REEL/FRAME 062113/0001) Recorded Apr 14, 2023
From: GOLDMAN SACHS BANK USA, AS COLLATERAL AGENT
To: CITRIX SYSTEMS, INC.; CLOUD SOFTWARE GROUP, INC. (F/K/A TIBCO SOFTWARE INC.)
Reel/Frame 063339/0525 →
PATENT SECURITY AGREEMENT Recorded Oct 7, 2022
From: TIBCO SOFTWARE INC.; CITRIX SYSTEMS, INC.
To: BANK OF AMERICA, N.A., AS COLLATERAL AGENT
Reel/Frame 062112/0262 →
PATENT SECURITY AGREEMENT Recorded Oct 7, 2022
From: TIBCO SOFTWARE INC.; CITRIX SYSTEMS, INC.
To: WILMINGTON TRUST, NATIONAL ASSOCIATION, AS NOTES COLLATERAL AGENT
Reel/Frame 062113/0470 →
SECOND LIEN PATENT SECURITY AGREEMENT Recorded Oct 7, 2022
From: TIBCO SOFTWARE INC.; CITRIX SYSTEMS, INC.
To: GOLDMAN SACHS BANK USA, AS COLLATERAL AGENT
Reel/Frame 062113/0001 →
SECURITY INTEREST Recorded Sep 30, 2022
From: CITRIX SYSTEMS, INC.
To: WILMINGTON TRUST, NATIONAL ASSOCIATION
Reel/Frame 062079/0001 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 14, 2017
From: HERBERT, JENNIFER RACHEL
To: CITRIX SYSTEMS, INC.
Reel/Frame 044396/0593 →
Continuity (1)
Related Publication 20190179659A1 · Jun 13, 2019