IP Library › Granted Patent US 12,487,843
Granted Patent B2
US 12,487,843 · App. 17/559,004 · Granted Dec 2, 2025

Data protection for control planes in a virtualized computer system

Inventors: Brian Masao Oki (San Jose, CA); Mukesh Hira (Los Altos, CA); Konstantinos Roussos (Sunnyvale, CA)
Assignee: VMware LLC
G06F9/45558G06F9/541G06F11/1469H04L41/0895G06F2009/45579G06F2009/45595G06F2201/815
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,487,843
App. No.
17/559,004
Granted
Dec 2, 2025
Kind
B2
Abstract

An example method of data protection in a virtualized computing system, the virtualized computing system including a host cluster, a virtualization management server connected, and a network manager coupled to a physical network, the host cluster having hosts and a virtualization layer executing on hardware platforms of the hosts, is described. The method includes: receiving a backup request; executing, in response to the backup request, a coupled backup of the virtualization management server and the network manager, including: creating a backup of a first database in the virtualization management server, the first database storing first configuration data for a virtual infrastructure (VI) control plane of the host cluster; creating a backup of a second database in the network manager, the second database storing second configuration data for a logical network deployed in the host cluster; and storing the coupled backup in remote storage.

Claims (57)

1 . A method of data protection in a virtualized computing system, the virtualized computing system including a host cluster, a virtualization management server, and a network manager each coupled to a physical network, the host cluster having hosts and a virtualization layer executing on hardware platforms of the hosts, the method comprising:

receiving, at the virtualization management server, a backup request, the virtualization management server managing the host cluster, the virtualization layer, and virtual machines (VMs) executing on the virtualization layer, the virtualization management server including a service configured to cooperate with the network manager;

installing, by the virtualization management server, first agents in hypervisors of the virtualization layer to add the hosts as managed entities;

installing, by the network manager, second agents in the hypervisors of the virtualization layer to add the hosts as nodes of a logical network overlaid on the physical network, the logical network including edge nodes, managed by the network manager, configured to provide ingress and egress between the logical network and an external network;

executing, at the virtualization management server in response to the backup request, a coupled backup of the virtualization management server and the network manager, including:

creating, at a first time, a backup of a first database in the virtualization management server, the first database storing first configuration data for a virtual infrastructure (VI) control plane of the host cluster;

creating, at the first time, a backup of a second database in the network manager, the second database storing second configuration data for nodes and edge nodes of the logical network;

storing the coupled backup in storage remote from the virtualization management server and the network manager.

2 . The method of claim 1 , wherein the virtualized computing system includes an orchestration control plane integrated with the virtualization layer and including at least one master server managing nodes implemented as virtual machines (VMs) executing on the virtualization layer, and wherein the step of executing the coupled backup includes:

creating a backup of third configuration data for the orchestration control plane.

3 . The method of claim 2 , wherein the third configuration data is stored in the first database in the virtualization management server.

4 . The method of claim 1 , wherein the virtualization management server includes a network service configured to interface with an application programming interface (API) of the network manager, and wherein the step of creating the backup of the second database comprises requesting the backup of the second database through the network service.

5 . The method of claim 1 , wherein the first configuration data includes a portion of the second configuration data.

6 . The method of claim 1 , further comprising:

executing, by the virtualization management server, a coupled restore of the coupled backup to restore the backup of the first database to the virtualization management server and the backup of the second database to the network manager; and

executing, by the virtualization management server, a recovery to remediate one or more inconsistencies of the coupled restore based on state of the virtualization layer.

7 . The method of claim 1 , further comprising:

executing, by the virtualization management server, a decoupled restore of the coupled backup to restore the backup of one of the first database to the virtualization management server or the backup of the second database to the network manager; and

executing, by the virtualization management server, a recovery to remediate one or more inconsistencies of the decoupled restore based on state of the virtualization layer.

8 . A non-transitory computer readable medium comprising instructions to be executed in a computing device to cause the computing device to carry out a method of data protection in a virtualized computing system, the virtualized computing system including a host cluster, a virtualization management server, and a network manager each coupled to a physical network, the host cluster having hosts and a virtualization layer executing on hardware platforms of the hosts, the method comprising:

receiving, at the virtualization management server, a backup request, the virtualization management server managing the host cluster, the virtualization layer, and virtual machines (VMs) executing on the virtualization layer, the virtualization management server including a service configured to cooperate with the network manager;

installing, by the virtualization management server, first agents in hypervisors of the virtualization layer to add the hosts as managed entities;

installing, by the network manager, second agents in the hypervisors of the virtualization layer to add the hosts as nodes of a logical network overlaid on the physical network, the logical network including edge nodes, managed by the network manager, configured to provide ingress and egress between the logical network and an external network;

executing, at the virtualization management server in response to the backup request, a coupled backup of the virtualization management server and the network manager, including:

creating, at a first time, a backup of a first database in the virtualization management server, the first database storing first configuration data for a virtual infrastructure (VI) control plane of the host cluster;

creating, at the first time, a backup of a second database in the network manager, the second database storing second configuration data for nodes and edge nodes of the logical network;

storing the coupled backup in storage remote from the virtualization management server and the network manager.

9 . The non-transitory computer readable medium of claim 8 , wherein the virtualized computing system includes an orchestration control plane integrated with the virtualization layer and including at least one master server managing nodes implemented as virtual machines (VMs) executing on the virtualization layer, and wherein the step of executing the coupled backup includes:

creating a backup of third configuration data for the orchestration control plane.

10 . The non-transitory computer readable medium of claim 9 , wherein the third configuration data is stored in the first database in the virtualization management server.

11 . The non-transitory computer readable medium of claim 8 , wherein the virtualization management server includes a network service configured to interface with an application programming interface (API) of the network manager, and wherein the step of creating the backup of the second database comprises requesting the backup of the second database through the network service.

12 . The non-transitory computer readable medium of claim 8 , wherein the first configuration data includes a portion of the second configuration data.

13 . The non-transitory computer readable medium of claim 8 , further comprising:

executing, by the virtualization management server, a coupled restore of the coupled backup to restore the backup of the first database to the virtualization management server and the backup of the second database to the network manager; and

executing, by the virtualization management server, a recovery to remediate one or more inconsistencies of the coupled restore based on state of the virtualization layer.

14 . The non-transitory computer readable medium of claim 8 , further comprising:

executing, by the virtualization management server, a decoupled restore of the coupled backup to restore the backup of one of the first database to the virtualization management server or the backup of the second database to the network manager; and

executing, by the virtualization management server, a recovery to remediate one or more inconsistencies of the decoupled restore based on state of the virtualization layer.

15 . A virtualized computing system, comprising:

a host cluster, a virtualization management server, and a network manager each connected to a physical network, the virtualization management server managing the host cluster, the virtualization layer, and virtual machines (VMs) executing on the virtualization layer, the virtualization management server including a service configured to cooperate with the network manager;

the host cluster including hosts and a virtualization layer executing on hardware platforms of the hosts, the hardware platforms including central processing units (CPUs) and memories, the virtualization management server configured to install first agents in hypervisors of the virtualization layer to add the hosts as managed entities;

the network manager configured to manage a software-defined (SD) network for the host cluster, the network manager configured to install second agents in the hypervisors of the virtualization layer to add the hosts as nodes of a logical network overlaid on the physical network, the logical network including edge nodes, managed by the network manager, configured to provide ingress and egress between the logical network and an external network; and

the virtualization management server configured to:

receive a backup request;

execute, in response to the backup request, a coupled backup of the virtualization management server and the network manager, including:

creating a backup of a first database in the virtualization management server, the first database storing first configuration data for a virtual infrastructure (VI) control plane of the host cluster;

creating a backup of a second database in the network manager, the second database storing second configuration data for the nodes and the edge nodes of the logical network;

store the coupled backup in storage remote from the virtualization management server and the network manager.

16 . The virtualized computing system of claim 15 , wherein the virtualized computing system includes an orchestration control plane integrated with the virtualization layer and including at least one master server managing nodes implemented as virtual machines (VMs) executing on the virtualization layer, and wherein the virtualization management server executes the coupled backup by creating a backup of third configuration data for the orchestration control plane.

17 . The virtualized computing system of claim 16 , wherein the third configuration data is stored in the first database in the virtualization management server.

18 . The virtualized computing system of claim 15 , wherein the virtualization management server includes a network service configured to interface with an application programming interface (API) of the network manager, and wherein the virtualization management server creates the backup of the second database by requesting the backup of the second database through the network service.

19 . The virtualized computing system of claim 15 , wherein the virtualization management server is further configured to:

execute a coupled restore of the coupled backup to restore the backup of the first database to the virtualization management server and the backup of the second database to the network manager; and

execute a recovery to remediate one or more inconsistencies of the coupled restore based on state of the virtualization layer.

20 . The virtualized computing system of claim 15 , wherein the virtualization management server is further configured to:

execute a decoupled restore of the coupled backup to restore the backup of one of the first database to the virtualization management server or the backup of the second database to the network manager; and

execute a recovery to remediate one or more inconsistencies of the decoupled restore based on state of the virtualization layer.

Assignments (2)
CHANGE OF NAME Recorded Apr 15, 2024
From: VMWARE, INC.
To: VMWARE LLC
Reel/Frame 067102/0242 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 14, 2023
From: OKI, BRIAN MASAO; HIRA, MUKESH; ROUSSOS, KONSTANTINOS
To: VMWARE, INC.
Reel/Frame 062976/0307 →
Continuity (2)
Provisional Application 63129252 · Dec 22, 2020
Related Publication 20220197687A1 · Jun 23, 2022
References Cited (79)
US 8572679B1 · Wang et al. · 2013 [cited by applicant]
US 9276816B1 · Conte et al. · 2016 [cited by applicant]
US 9547562B1 · Feathergill · 2017 [cited by examiner]
US 9659040B1 · Bellingan et al. · 2017 [cited by applicant]
US 10289441B1 · Chopra · 2019 [cited by examiner]
US 10565160B1 · Gorelik et al. · 2020 [cited by applicant]
US 10754844B1 · Jain · 2020 [cited by examiner]
US 11070520B2 · Stabile · 2021 [cited by examiner]
US 11252157B1 · Khanna et al. · 2022 [cited by applicant]
US 11341104B1 · Prabhakaran · 2022 [cited by examiner]
US 11860743B1 · Willett · 2024 [cited by examiner]
US 20100011178A1 · Feathergill · 2010 [cited by examiner]
US 20120078855A1 · Beatty · 2012 [cited by examiner]
US 20150071110A1 · Kothari et al. · 2015 [cited by applicant]
US 20150103692A1 · Jain et al. · 2015 [cited by applicant]
US 20150280928A1 · Tessmer · 2015 [cited by examiner]
US 20160043905A1 · Fiedler · 2016 [cited by applicant]
US 20160179635A1 · Kondalsamy · 2016 [cited by examiner]
US 20160224363A1 · Joy · 2016 [cited by applicant]
US 20160254956A1 · Xu et al. · 2016 [cited by applicant]
US 20170244611A1 · Wang · 2017 [cited by applicant]
US 20180006902A1 · Bansal · 2018 [cited by examiner]
US 20180034747A1 · Nataraja et al. · 2018 [cited by applicant]
US 20180101395A1 · Aleksandrov et al. · 2018 [cited by applicant]
US 20180123932A1 · Shaw et al. · 2018 [cited by applicant]
US 20180159781A1 · Mehta et al. · 2018 [cited by applicant]
US 20180165122A1 · Dobrev · 2018 [cited by applicant]
US 20180260251A1 · Beveridge · 2018 [cited by examiner]
US 20180287938A1 · Han · 2018 [cited by examiner]
US 20180373961A1 · Wang · 2018 [cited by examiner]
US 20190020662A1 · Kumar · 2019 [cited by examiner]
US 20190028342A1 · Kommula et al. · 2019 [cited by applicant]
US 20190028345A1 · Kommula et al. · 2019 [cited by applicant]
US 20190036868A1 · Chandrashekhar et al. · 2019 [cited by applicant]
US 20190068622A1 · Lin · 2019 [cited by examiner]
US 20190102411A1 · Hung · 2019 [cited by examiner]
US 20190132197A1 · Saxena et al. · 2019 [cited by applicant]
US 20190171435A1 · Pande et al. · 2019 [cited by applicant]
US 20190229987A1 · Shelke et al. · 2019 [cited by applicant]
US 20190317751A1 · Ramsay et al. · 2019 [cited by applicant]
US 20200019468A1 · Chinnam et al. · 2020 [cited by applicant]
US 20200174845A1 · Toeroe · 2020 [cited by applicant]
US 20200213227A1 · Pianigiani et al. · 2020 [cited by applicant]
US 20210011781A1 · Wan · 2021 [cited by examiner]
US 20210081246A1 · Rajadurai · 2021 [cited by examiner]
US 20210089368A1 · Goosen et al. · 2021 [cited by applicant]
US 20210165694A1 · Nabi et al. · 2021 [cited by applicant]
US 20210311764A1 · Rosoff · 2021 [cited by examiner]
US 20210334004A1 · Krivenok · 2021 [cited by examiner]
US 20210334178A1 · Yang et al. · 2021 [cited by applicant]
US 20220004417A1 · Sinha et al. · 2022 [cited by applicant]
US 20220179760A1 · Manjunath · 2022 [cited by examiner]
Jeanna Matthews, Data Protection and Rapid Recovery From Attack With a Virtual Private File Server and Virtual Machine Appliances. (Year: 2005). [cited by examiner]
Zhe Wang, A Remote Backup Approach for Virtual Machine Images. (Year: 2016). [cited by examiner]
Oussama Soualah, Reliable embedding of virtual networks in Cloud's backbone network. (Year: 2015). [cited by examiner]
Teemu Koponen, Network Virtualization in Multi-tenant Datacenters. (Year: 2014). [cited by examiner]
ESXSI: “vSphere 7 with Kubernetes and Tanzu on VMware Cloud Foundation,” Mar. 10, 2020, ESXSI.com, Publication [online], Mar. 10, 2020 [retrieved Jul. 20, 2021], pp. 1-23. Retrieved from the Internet: <URL: https://esxs… [cited by applicant]
VMWARE, Inc. (Ramachandra et al.) “vCloud NFV Reference Architecture,” VMware vCloud NFV 3.0, Dec. 31, 2018, Publication [online], 2018 [retrieved May 31, 2021], pp. 8-9, 15-18, 27-43, 52-75. Retrieved from the Internet… [cited by applicant]
International Search Report and Written Opinion mailed Jul. 1, 2021 in corresponding International Application No. PCT/US2021/024412, 9 pages. [cited by applicant]
Final Office Action mailed Aug. 26, 2022 in U.S. Appl. No. 16/838,573, 25 pages. [cited by applicant]
Lee, Keith “PKS NSX-T Home Lab—Part 8: Configure NSX-T,” Nov. 24, 2018, 20 pages, URL: keithlee.ie/2018/11/24/pks-nsx-t-home-lab-part-8-configure-nsx-t/. [cited by applicant]
Shaik, Chand B. “Prepare ESXi host as fabric Node in NSX-T,” Jun. 22, 2018, 8 pages, URL: virtualbrigade.com/prepare-esxi-host-as-fabric-node/. [cited by applicant]
Smit, Martijn and Omkar Singh “VMware NSX-V Control and Management Plane Connections Diagram,” Aug. 21, 2018, 3 pages, URL: lostdomain.org/2018/08/21/vmware-nsx-v-control-and-management-plane-connections-diagram/. [cited by applicant]
Non-Final Office Action mailed Aug. 11, 2023 in U.S. Appl. No. 17/005,487, 14 pages. [cited by applicant]
Benson, T. et al. “CloudNaaS: A Cloud Networking Platform for Enterprise Applications,” Proceedings of the 2nd ACM Symposium on Cloud Computing (SOCC'11), Oct. 27-28, 2011, 13 pages. [cited by applicant]
Lee, B. “VMware vSphere management cluster role and benefits,” 2019, 5 pages, Retrieved from: www.vembu.com/blog/vmware-management-cluster-design-and-benefits/. [cited by applicant]
VMWARE, Inc. “NSX Upgrade Guide,” Online Product Documentation for VMware NSX Data Center for vSphere ver.6.4; selected sections, Front page: docs.vmware.com/en/VMware-NSX-Data-Center-for-vSphere/6.4/com.vmware.nsx.upgr… [cited by applicant]
VMWARE, Inc. “NSX-T Data Center Upgrade Guide,” VMware NSX-T Data Center 3.0, Apr. 7, 2020, 49 pages. [cited by applicant]
Non-Final Office Action mailed Jul. 14, 2022 in U.S. Appl. No. 17/005,487, 20 pages. [cited by applicant]
Awati, R. et al. TechTarget Definitions for “plane (in networking)” and “control plane (CP),” techtarget.com, 2022, 5 pages. [cited by applicant]
IBM “Get Control of Your Cloud with IBM Cloud Orchestrator and Juniper Networks Contrail,” IBM Redbooks, Document identifier: REDP-5125-00, 2014, 11 pages. [cited by applicant]
Gharbaoui, M. et al. “An Orchestrator of Network and Cloud Resources for Dynamic Provisioning of Mobile Virtual Network Functions,” IEEE 2016, pp. 98-101. [cited by applicant]
Martini, B. et al. “Cross-Functional Resource Orchestration in Optical Telco Clouds,” ICTON 2015, 5 pages. [cited by applicant]
Oliveira, C. et al. “VMware NSX® Automation Fundamentals,” VMware Press, 2018, 166 pages. [cited by applicant]
Final Office Action mailed May 3, 2024 in U.S. Appl. No. 17/005,487, 18 pages. [cited by applicant]
Pivotal Software, Inc. “Pivotal Container Service (PKS),” Version 1.3, Product Documentation, Jul. 17, 2019, 515 pages. [cited by applicant]
Thurgood, B. et al. “Cloud Computing With Kubernetes Cluster Elastic Scaling,” Proceedings of the 3rd International Conference on Future Networks and Distributed Systems, Jul. 1, 2019, pp. 1-7. [cited by applicant]
Non-Final Office Action mailed Aug. 28, 2024 in U.S. Appl. No. 17/559,019, 41 pages. [cited by applicant]
Notice of Allowance mailed Oct. 9, 2024 in U.S. Appl. No. 17/005,487, 25 pages. [cited by applicant]