IP Library › Granted Patent US 12,282,795
Granted Patent B2
US 12,282,795 · App. 18/667,648 · Granted Apr 22, 2025

Hypervisor hibernation

Inventor: Binny Sher Gill (San Jose, CA)
Assignee: Nutanix, Inc.
G06F9/45558G06F9/4418G06F12/0246G06F2009/45575G06F2009/45583G06F2212/7201
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,282,795
App. No.
18/667,648
Filed
May 17, 2024
Granted
Apr 22, 2025
Kind
B2
Examiner
KIM, DONG U
Art Unit
2197
USPC
718/1
Abstract

Upon receiving a request to hibernate a hypervisor of a virtualization system running on a first computer, acts are carried out to capture a state of the hypervisor, where the state of the hypervisor comprises hypervisor logical resource parameters and an execution state of the hypervisor. After hibernating the hypervisor by quiescing the hypervisor and storing the state of the hypervisor into a data structure, the data structure is moved to a different location. At a later moment in time, the data structure is loaded onto a second computing machine and restored. The restore operation restores the hypervisor and all of its state, including all of the virtual machines of the hypervisor as well as all of the virtual disks and other virtual devices of the virtual machines. Differences between the first computing machine and the second computing machine are reconciled before execution of the hypervisor on the second machine.

Claims (38)

1. A non-transitory computer readable medium having stored thereon a sequence of instructions which, when stored in memory and executed by a processor cause a set of acts for migrating configurations of computing clusters between cloud environments, the set of acts comprising:

identifying a computing cluster to be moved from a first cloud environment to a different cloud environment, the computing cluster comprising a plurality of nodes that each have a hypervisor and at least one virtual machine instance that manages a vDisk structured from any one or more storage devices in a storage pool, wherein the storage pool comprises at least a plurality of storage devices, wherein each node of the plurality of nodes of the computing cluster includes at least one said storage device of the storage pool; and

migrating a configuration of the computing cluster to the different cloud environment at least by, for each node of the plurality of nodes of the computing cluster that is to be migrated:

hibernating a corresponding hypervisor and at least one corresponding virtual machine instance that manages a respective vDisk structured from any one or more of the storage devices in the storage pool,

copying a configuration of the hibernated hypervisor and the at least one corresponding virtual machine instance to a new node at the different cloud environment, and

powering on the at least one virtual machine at the different cloud environment, wherein, after powering on the at least one virtual machine instance on each new node, the computing cluster runs on the different cloud environment.

2. The non-transitory computer readable medium of claim 1 , wherein the first cloud environment comprises a first private cloud or a first public cloud and the different cloud environment comprises a second private cloud or a second public cloud.

3. The non-transitory computer readable medium of claim 2 , wherein the first private cloud or the second private cloud is on-premises.

4. The non-transitory computer readable medium of claim 1 , wherein in the first cloud environment comprises a private cloud and the different cloud environment comprises a public cloud.

5. The non-transitory computer readable medium of claim 1 , wherein the computing cluster comprises a first hypervisor on a first node and a second hypervisor on a second node, each of the first and second hypervisors having one or more user virtualized processes separate from the at least one virtual machine instance that manages a vDisk structured from any one or more storage devices in the storage pool.

6. The non-transitory computer readable medium of claim 1 , wherein hypervisors on the nodes are on physical hardware of a respective node.

7. The non-transitory computer readable medium of claim 1 , wherein a first individual hypervisor of the hypervisors on the nodes in the first cloud environment runs on first physical hardware of a first node, the first individual hypervisor in the different cloud environment runs on different physical hardware of a different node.

8. A method for migrating configurations of computing clusters between cloud environments, the method comprising:

identifying a computing cluster to be moved from a first cloud environment to a different cloud environment, the computing cluster comprising a plurality of nodes that each have a hypervisor and at least one virtual machine instance that manages a vDisk structured from any one or more storage devices in a storage pool, wherein the storage pool comprises at least a plurality of storage devices, wherein each node of the plurality of nodes of the computing cluster includes at least one said storage device of the storage pool; and

migrating a configuration of the computing cluster to the different cloud environment at least by, for each node of the plurality of nodes of the computing cluster that is to be migrated:

hibernating a corresponding hypervisor and at least one corresponding virtual machine instance that manages a respective vDisk structured from any one or more of the storage devices in the storage pool,

copying a configuration of the hibernated hypervisor and the at least one corresponding virtual machine instance to a new node at the different cloud environment, and

powering on the at least one virtual machine at the different cloud environment, wherein, after powering on the at least one virtual machine instance on each new node, the computing cluster runs on the different cloud environment.

9. The method of claim 8 , wherein the first cloud environment comprises a first private cloud or a first public cloud and the different cloud environment comprises a second private cloud or a second public cloud.

10. The method of claim 9 , wherein the first private cloud or the second private cloud is on-premises.

11. The method of claim 8 , wherein in the first cloud environment comprises a private cloud and the different cloud environment comprises a public cloud.

12. The method of claim 8 , wherein the computing cluster comprises a first hypervisor on a first node and a second hypervisor on a second node, each of the first and second hypervisors having one or more user virtualized processes separate from the at least one virtual machine instance that manages a vDisk structured from any one or more storage devices in the storage pool.

13. The method of claim 8 , wherein hypervisors on the nodes are on physical hardware of a respective node.

14. The method of claim 8 , wherein a first individual hypervisor of the hypervisors on the nodes in the first cloud environment runs on first physical hardware of a first node, the first individual hypervisor in the different cloud environment runs on different physical hardware of a different node.

15. A system for migrating configurations of virtualization environments between cloud environments, the system comprising:

a storage medium having stored thereon a sequence of instructions; and

a processor that executes the sequence of instructions to cause a set of acts comprising:

identifying a computing cluster to be moved from a first cloud environment to a different cloud environment, the computing cluster comprising a plurality of nodes that each have a hypervisor and at least one virtual machine instance that manages a vDisk structured from any one or more storage devices in a storage pool, wherein the storage pool comprises at least a plurality of storage devices, wherein each node of the plurality of nodes of the computing cluster includes at least one said storage device of the storage pool; and

migrating a configuration of the computing cluster to the different cloud environment at least by, for each node of the plurality of nodes of the computing cluster that is to be migrated:

hibernating a corresponding hypervisor and at least one corresponding virtual machine instance that manages a respective vDisk structured from any one or more of the storage devices in the storage pool,

copying a configuration of the hibernated hypervisor and the at least one corresponding virtual machine instance to a new node at the different cloud environment, and

powering on the at least one virtual machine at the different cloud environment, wherein, after powering on the at least one virtual machine instance on each new node, the computing cluster runs on the different cloud environment.

16. The system of claim 15 , wherein the first cloud environment comprises a first private cloud or a first public cloud and the different cloud environment comprises a second private cloud or a second public cloud.

17. The system of claim 16 , wherein the first private cloud or the second private cloud is on-premises.

18. The system of claim 15 , wherein in the first cloud environment comprises a private cloud and the different cloud environment comprises a public cloud.

19. The system of claim 15 , wherein the computing cluster comprises a first hypervisor on a first node and a second hypervisor on a second node, each of the first and second hypervisors having one or more user virtualized processes separate from the at least one virtual machine instance that manages a vDisk structured from any one or more storage devices in the storage pool.

20. The system of claim 15 , wherein hypervisors on the nodes are on physical hardware of a respective node.

21. The system of claim 15 , wherein a first individual hypervisor of the hypervisors on the nodes in the first cloud environment runs on first physical hardware of a first node, the first individual hypervisor in the different cloud environment runs on different physical hardware of a different node.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 21, 2025
From: GILL, BINNY SHER
To: NUTANIX, INC.
Reel/Frame 070584/0698 →
SECURITY INTEREST Recorded Feb 13, 2025
From: NUTANIX, INC.
To: BANK OF AMERICA, N.A., AS COLLATERAL AGENT
Reel/Frame 070206/0463 →
Continuity (5)
Continuation 18161091 · Jan 30, 2023
Continuation 16778909 · Jan 31, 2020
Provisional Application 62939573 · Nov 22, 2019
Provisional Application 62894674 · Aug 30, 2019
Related Publication 20240419475A1 · Dec 19, 2024
References Cited (121)
US 6654830B1 · Taylor et al. · 2003 [cited by applicant]
US 6802062B1 · Oyamada et al. · 2004 [cited by applicant]
US 6807630B2 · Lay et al. · 2004 [cited by applicant]
US 6813708B2 · Smith et al. · 2004 [cited by applicant]
US 6882795B1 · Mcmurdie et al. · 2005 [cited by applicant]
US 7693877B1 · Zasman · 2010 [cited by applicant]
US 8166477B1 · Tormasov et al. · 2012 [cited by applicant]
US 8230204B2 · Rostampour · 2012 [cited by applicant]
US 8549518B1 · Aron et al. · 2013 [cited by applicant]
US 8601473B1 · Aron et al. · 2013 [cited by applicant]
US 8607009B2 · Nicholas et al. · 2013 [cited by applicant]
US 8850130B1 · Aron et al. · 2014 [cited by applicant]
US 8959323B2 · Ganti et al. · 2015 [cited by applicant]
US 9430268B2 · Hussain et al. · 2016 [cited by applicant]
US 9552215B2 · Voccio et al. · 2017 [cited by applicant]
US 9652265B1 · Narayanasamy et al. · 2017 [cited by applicant]
US 9772866B1 · Aron et al. · 2017 [cited by applicant]
US 9804876B2 · Tsirkin et al. · 2017 [cited by applicant]
US 9940148B1 · Dannowski · 2018 [cited by examiner]
US 10108517B1 · Efremov et al. · 2018 [cited by applicant]
US 10114565B2 · Nithrakashyap et al. · 2018 [cited by applicant]
US 10860363B1 · Gabrielson · 2020 [cited by applicant]
US 10901768B1 · Mandadi · 2021 [cited by examiner]
US 11169835B1 · Duong et al. · 2021 [cited by applicant]
US 20030208675A1 · Burokas et al. · 2003 [cited by applicant]
US 20040083355A1 · Smith et al. · 2004 [cited by applicant]
US 20050060590A1 · Bradley et al. · 2005 [cited by applicant]
US 20050091354A1 · Lowell et al. · 2005 [cited by applicant]
US 20050160423A1 · Bantz et al. · 2005 [cited by applicant]
US 20070185934A1 · Cannon et al. · 2007 [cited by applicant]
US 20100037096A1 · Bum · 2010 [cited by examiner]
US 20120195187A1 · Ashihara et al. · 2012 [cited by applicant]
US 20120254865A1 · Saeki · 2012 [cited by examiner]
US 20130080814A1 · Cong et al. · 2013 [cited by applicant]
US 20130138995A1 · Sivaramakrishnan · 2013 [cited by examiner]
US 20130139154A1 · Shah · 2013 [cited by examiner]
US 20130191354A1 · Wilf · 2013 [cited by applicant]
US 20130227553A1 · Tsirkin et al. · 2013 [cited by applicant]
US 20130227554A1 · Tsirkin et al. · 2013 [cited by applicant]
US 20130227555A1 · Tsirkin et al. · 2013 [cited by applicant]
US 20130275975A1 · Masuda · 2013 [cited by examiner]
US 20140149635A1 · Bacher et al. · 2014 [cited by applicant]
US 20150058521A1 · Armstrong et al. · 2015 [cited by applicant]
US 20150058522A1 · Armstrong et al. · 2015 [cited by applicant]
US 20150227192A1 · Kruglick · 2015 [cited by examiner]
US 20160277308A1 · Challa et al. · 2016 [cited by applicant]
US 20160283281A1 · Antony · 2016 [cited by applicant]
US 20160306649A1 · Gunti · 2016 [cited by examiner]
US 20160323098A1 · Bathen · 2016 [cited by examiner]
US 20160344821A1 · Khosrowpour · 2016 [cited by examiner]
US 20170039082A1 · Ganesan et al. · 2017 [cited by applicant]
US 20170147409A1 · Chandrakar · 2017 [cited by examiner]
US 20180173513A1 · Agarwal et al. · 2018 [cited by applicant]
US 20180232038A1 · Surdu et al. · 2018 [cited by applicant]
US 20180314706A1 · Sirton et al. · 2018 [cited by applicant]
US 20190243671A1 · Yadav · 2019 [cited by examiner]
US 20190317796A1 · Kunt et al. · 2019 [cited by applicant]
US 20200034250A1 · Chandrasekaran et al. · 2020 [cited by applicant]
US 20200089409A1 · Ankireddypalle et al. · 2020 [cited by applicant]
US 20200133706A1 · Koehler et al. · 2020 [cited by applicant]
US 20200133718A1 · Koehler et al. · 2020 [cited by applicant]
US 20200192690A1 · Gupta et al. · 2020 [cited by applicant]
US 20200310848A1 · Kaul · 2020 [cited by applicant]
US 20200319694A1 · Mohanty et al. · 2020 [cited by applicant]
US 20210064408A1 · Gill · 2021 [cited by applicant]
US 20210067466A1 · Parab et al. · 2021 [cited by applicant]
US 20210067484A1 · Parab et al. · 2021 [cited by applicant]
US 20210157620A1 · Schütz · 2021 [cited by examiner]
US 20210263762A1 · Kachare et al. · 2021 [cited by applicant]
Non-Final Office Action for U.S. Appl. No. 17/086,393 dated Mar. 21, 2022. [cited by applicant]
“4.2 Hibernating a VM,” FusionCompute. [cited by applicant]
“Hibernate your On-Demand or Reserved Liux instance,” Amazon Elastic Compute Cloud, dated May 23, 2021. [cited by applicant]
Non-Final Office Action for U.S. Appl. No. 16/778,909 dated Mar. 18, 2022. [cited by applicant]
Extended European Search Report for EP Patent Appln. No. 20193502.0 dated Jan. 27, 2021. [cited by applicant]
Lee, B., “Types of Hyper-V Cluster Virtual Machine Migration,” Vembu, dated Aug. 17, 2018. [cited by applicant]
“Oracle VM Windows Paravirtual (PV) Drivers 2.0: New Features,” An Oracle Technical White Paper, dated Jun. 2010. [cited by applicant]
Gill, B., et al., “Technical Preview: Nutanix on AWS: Making Lift and Shift Invisible,” .NEXTConference, Nutanix Inc., Copyright 2019. [cited by applicant]
Devarapalli, S., et al., “Xi Clusters: Accelerate Your Hybrid Cloud Journey with Nutanix in Public Clouds,” .NEXTConference, Nutanix Inc., copyright 2019. [cited by applicant]
Poitras, Steven. “The Nutanix Bible” (Oct. 15, 2013), from http://stevenpoitras.com/the-nutanix-bible/ (Publication date based on indicated capture date by Archive.org; first publication date unknown). [cited by applicant]
Poitras, Steven. “The Nutanix Bible” (Jan. 11, 2014), from http://stevenpoitras.com/the-nutanix-bible/ (Publication date based on indicated capture date by Archive.org; first publication date unknown). [cited by applicant]
Poitras, Steven. “The Nutanix Bible” (Jun. 20, 2014), from http://stevenpoitras.com/the-nutanix-bible/ (Publication date based on indicated capture date by Archive.org; first publication date unknown). [cited by applicant]
Poitras, Steven. “The Nutanix Bible” (Jan. 7, 2015), from http://stevenpoitras.com/the-nutanix-bible/ (Publication date based on indicated capture date by Archive.org; first publication date unknown). [cited by applicant]
Poitras, Steven. “The Nutanix Bible” (Jun. 9, 2015), from http://stevenpoitras.com/the-nutanix-bible/ (Publication date based on indicated capture date by Archive.org; first publication date unknown). [cited by applicant]
Poitras, Steven. “The Nutanix Bible” (Sep. 4, 2015), from https://nutanixbible.com/. [cited by applicant]
Poitras, Steven. “The Nutanix Bible” (Jan. 12, 2016), from https://nutanixbible.com/. [cited by applicant]
Poitras, Steven. “The Nutanix Bible” (Jun. 9, 2016), from https://nutanixbible.com/. [cited by applicant]
Poitras, Steven. “The Nutanix Bible” (Jan. 3, 2017), from https://nutanixbible.com/. [cited by applicant]
Poitras, Steven. “The Nutanix Bible” (Jun. 8, 2017), from https://nutanixbible.com/. [cited by applicant]
Poitras, Steven. “The Nutanix Bible” (Jan. 3, 2018), from https://nutanixbible.com/. [cited by applicant]
Poitras, Steven. “The Nutanix Bible” (Jun. 25, 2018), from https://nutanixbible.com/. [cited by applicant]
Poitras, Steven. “The Nutanix Bible” (Jan. 8, 2019), from https://nutanixbible.com/. [cited by applicant]
Poitras, Steven. “The Nutanix Bible” (Jul. 9, 2019), from https://nutanixbible.com/ (Publication date based on indicated capture date by Archive.org; first publication date unknown). [cited by applicant]
Poitras, Steven. “The Nutanix Bible” (Feb. 3, 2020), from https://nutanixbible.com/ (Publication date based on indicated capture date by Archive.org; first publication date unknown). [cited by applicant]
Poitras, Steven. “The Nutanix Bible” (Aug. 1, 2020), from https://nutanixbible.com/ (Publication date based on indicated capture date by Archive.org; first publication date unknown). [cited by applicant]
Poitras, Steven. “The Nutanix Bible” (Jan. 30, 2021), from https://nutanixbible.com/ (Publication date based on indicated capture date by Archive.org; first publication date unknown). [cited by applicant]
Cano, I. et al., “Curator: Self-Managing Storage for Enterprise Clusters”, [cited by applicant]
Final Office Action for U.S. Appl. No. 17/086,393 dated Jul. 14, 2022. [cited by applicant]
Notice of Allowance for U.S. Appl. No. 16/778,909 dated Aug. 12, 2022. [cited by applicant]
Notice of Allowance for U.S. Appl. No. 16/778,909 dated Sep. 21, 2022. [cited by applicant]
Min Choi et al., “Cluster computing environment supporting single system Image”, Sep. 20, 2004, Cluster Computing, 2004 IEEE International Conference, San Diego CA USA. [cited by applicant]
Non-Final Office Action for U.S. Appl. No. 17/086,393 dated Mar. 31, 2023. [cited by applicant]
Non-Final Office Action dated Jun. 9, 2023 for U.S. Appl. No. 18/161,091. [cited by applicant]
“Aviatrix Transit Architecture for Azure,” Aviatrix, URL: https://read.docs.aviatrix.com/HowTos/nextgentransit_for_azure.html, date found via Google as Apr. 2, 2020. [cited by applicant]
“Aviatrix Next Gen Transit with customized SNAT and DNAT features,” Aviatrix, URL: https://read.docs.aviatrix.com/HowTos/transit_snat_dnat.html, date found via Google as Jun. 19, 2019. [cited by applicant]
“Launching a Gateway,” Aviatrix, URL: https://read.docs.aviatrix.com/HowTos/gateway.html, date found via Google as Jun. 24, 2017. [cited by applicant]
“NSX Public Cloud Gateway: Architecture and Modes of Deployment,” VMWare, URL: https://docs.vmware.com/en/VMware-NSX-T-Data-Center/3.1/installation/GUID-E761C567-937F-4E79-A98A-DEE306B1F9EF.html, dated Feb. 5, 2021. [cited by applicant]
“Azure VMware Solution Planning and Deployment Guide,” VMWare, URL: https://vmc.techzone.vmware.com/resource/avs-planning-and-deployment-guide#introduction, date found via Google as Aug. 26, 2021. [cited by applicant]
Chebaro S., “Step by Step Deploy VMware NSX with Microsoft Azure IPSEC VPN Site-to-Site,” Zero to Hero End User Computing, dated Mar. 17, 2017. [cited by applicant]
Poitras, Steven. “The Nutanix Bible” (Sep. 9, 2022), from https://nutanixbible.com/ (Publication date based on indicated capture date by Archive.org; first publication date unknown). [cited by applicant]
“Citrix XenDesktop 7.1 on Microsoft Hyper-V Server 2012 R2 on Nutanix Virtual Computing Platform—Solution Design,” Citrix Validated Solutions, Prepared by: Citrix APAC Solutions, dated Jun. 25, 2014. [cited by applicant]
Final Office Action dated Aug. 2, 2023 for U.S. Appl. No. 17/086,393. [cited by applicant]
Notice of Allowance for U.S. Appl. No. 18/161,091 dated Oct. 12, 2023. [cited by applicant]
Notice of Allowance for U.S. Appl. No. 17/086,393 dated Dec. 15, 2023. [cited by applicant]
Notice of Allowance for U.S. Appl. No. 18/161,091 dated Jan. 31, 2024. [cited by applicant]
Foreign Exam Report for EP Patent Appln. No. 20193502.0 dated May 8, 2024. [cited by applicant]
Lam, W., “How to move a VSAN Cluster from one vCenter Server to another?,” WilliamLam website, dated Sep. 26, 2014. [cited by applicant]
“Cluster Migrations Involving New Storage: Mount Points,” Microsoft, URL: https://learn.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2012-r2-and-2012/dn486776(v=ws.11), dated Aug. 31, 2016. [cited by applicant]
“Remove a Host from a Cluster,” VMWare Docs, URL: https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.vcenterhost.doc/GUID-32E79431-7DC9-48DB-A72C-CCA652A3B588.html, dated May 31, 2019. [cited by applicant]
“Relocate a Host,” VMWare Docs, URL: https://docs.vmware.com/en/VMware-vSphere/6.7/com.vmware.vsphere.vcenterhost.doc/GUID-32E79431-7DC9-48DB-A72C-CCA652A3B588.html, dated May 31, 2019. [cited by applicant]
“Preparing for Migration,” VMWare Docs, URL: https://docs.vmware.com/en/VMware-vSphere/8.0/vsphere-vcenter-upgrade/GUID-921FD4BA-F5F4-46D8-ABAD-500B9BE349B2.html, dated Jul. 20, 2019. [cited by applicant]
“Migrating Virtual Machines,” VMWare Docs, URL: https://docs.vmware.com/en/VMware-vSphere/6.7/com.vmware.vsphere.vm_admin.doc/GUID-FE2B516E-7366-4978-B75C-64BF0AC676EB.html, dated May 31, 2019. [cited by applicant]