IP Library Granted Patent US 12,265,849
Granted Patent B2
US 12,265,849 · App. 15/981,671 · Granted Apr 1, 2025

Use of nested hypervisors by a resource-exchange system to enhance data and operational security and to facilitate component installation

Inventors: Daniel James Beveridge (Apollo Beach, FL); Ricky Trigalo (Palo Alto, CA); Joerg Lew (Rettenberg, DE)
Assignee: VMWare LLC
G06F9/5027G06F9/45558H04L41/5051H04L41/5054H04L47/783H04L67/567G06F2009/45566G06F2009/4557G06F2009/45575G06F2009/45579G06F2009/45587H04L41/5096
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,265,849
App. No.
15/981,671
Granted
Apr 1, 2025
Kind
B2
Abstract

The current document is directed a resource-exchange system that facilitates resource exchange and sharing among computing facilities. The currently disclosed methods and systems employ efficient, distributed-search methods and subsystems within distributed computer systems that include large numbers of geographically distributed data centers to locate resource-provider computing facilities that match the resource needs of resource-consumer computing-facilities based on attribute values associated with the needed resources, the resource providers, and the resource consumers. Nested-hypervisor technology is employed, in disclosed implementations, to guarantee data security for, and prevent monitoring of operational states and characteristics of, resource-consumer virtual machines and virtual applications while they execute above leased computational resources in remote computing facilities. Nested-hypervisor technology is additionally employed to facilitate installation of resource-exchange system management components in participant computing facilities as well as to guarantee data and operational security for the management components.

Claims (33)

1. An automated resource-exchange system comprising:

multiple resource-exchange-system participants, including resource consumers and resource providers, that each

hosts a local cloud-exchange instance; and

a distributed cloud-exchange system that

is implemented using multiple physical server computers,

includes a cloud-exchange engine,

includes the local cloud-exchange instances within the multiple resource-exchange-system participants, and

automatically brokers and carries out transactions in each of which a resource consumer requests to lease computational resources from one or more resource providers, the distributed cloud-exchange system selects one or more resource providers from among the resource-exchange-system participants to lease the computational resources to the resource consumer, and the distributed cloud-exchange system arranges for use of the leased computational resources by the resource consumer by coordinating launching one or more virtual machines within each of the computing facilities of the one or more selected resource providers, each virtual machine executing a nested hypervisor that provides an execution environment for one or more computational entities, the nested hypervisor being configured to provide an execution environment through a virtual hardware interface for second-level virtual machines and to provide load balancing among the second-level virtual machines;

wherein the distributed cloud-exchange system monitors the one or more virtual machines to detect potential security leaks.

2. The automated resource-exchange system of claim 1 wherein, in addition to launching the one or more virtual machines, the distributed cloud-exchange system arranges for use of the leased computational resources by the resource consumer by extending an internal network within the resource-consumer's computing facility to the one or more virtual machines by creating one or more secure communications tunnels between the one or more virtual machines and the internal network.

3. The automated resource-exchange system of claim 1 wherein the nested hypervisors executing in the one or more virtual machines are controlled by a management server within the distributed cloud-exchange system to create a cloud-exchange management domain within each of the computing facilities of the one or more selected resource providers.

4. The automated resource-exchange system of claim 3 wherein, in the transaction in which the one or more virtual machines are launched, the resource consumer requested to lease computational resources to run a set of one or more particular computational entities, and the one or more virtual machines are configured to provide the computational resources to run the one or more particular computational entities.

5. The automated resource-exchange system of claim 1 wherein the nested hypervisors executing in the one or more virtual machines are controlled by a management server within the resource consumer's computing facility to create a resource-consumer management domain within each of the computing facilities of the one or more selected resource providers.

6. The automated resource-exchange system of claim 5 wherein, in the transaction in which the one or more virtual machines are launched, the resource consumer requested to lease a block of computational resources, and the one or more virtual machines are configured to provide the requested block of computational resources to the resource consumer to use at the resource consumer's discretion.

7. The automated resource-exchange system of claim 1 wherein each nested hypervisor runs within an execution environment provided by a base hypervisor controlled by a management server within the resource provider's computing facility in which the nested hypervisor executes, the base hypervisor configured to provide separate management domains for the base hypervisor and the nested hypervisor.

8. The automated resource-exchange system of claim 7 wherein the nested hypervisor and base hypervisor that provides the execution environment for the nested hypervisor are configured so that, for network traffic flowing to the nested hypervisor from a network and for network traffic emanating from source entities executing in the execution environment provided by the nested hypervisor, the network traffic passes through the base hypervisor in encrypted form so that the base hypervisor cannot access the contents of the messages that together comprise the network traffic.

9. The automated resource-exchange system of claim 7 wherein the nested hypervisor and base hypervisor that provides the execution environment for the nested hypervisor are configured so that, for data flowing to the nested hypervisor from data-storage devices and for data transmitted to data-storage devices from source entities executing in the execution environment provided by the nested hypervisor, the data passes through the base hypervisor in encrypted form so that the base hypervisor cannot access the data.

10. The automated resource-exchange system of claim 1 wherein the distributed cloud-exchange system detects exposure of nested-hypervisor memory or state by examining logged events maintained in hypervisor logs to identify logged events corresponding to potential access to nested-hypervisor memory or state.

11. The automated resource-exchange system of claim 1 wherein the distributed cloud-exchange system detects exposure of memory or state of resource consumer VMs by examining the resource provider's hypervisor to determine whether or not any of the resource consumer's VMs are executing in execution environments provided by resource-provider-managed nested hypervisors.

12. The automated resource-exchange system of claim 1 wherein each local cloud-exchange instance is installed in a resource-provider computing facility as a virtual machine running a nested hypervisor controlled by a cloud-exchange-system management server and multiple second-level virtual machines running in the execution environment provided by the nested hypervisor.

13. The automated resource-exchange system of claim 12 wherein environment-specific configurations of the virtual machine running a nested hypervisor controlled by a cloud-exchange-system management server are carried out by an automated process invoked by the distributed cloud-exchange system following launching of the virtual machine within the resource-provider computing facility.

14. A method that creates two management domains within a resource-exchange-system-participant computing facility of an automated resource-exchange system, the method comprising:

launching, by a distributed cloud-exchange system implemented on multiple physical server computers, one or more virtual machines within each of the computing facilities of one or more resource providers selected by the distributed cloud-exchange system to provide computational resources to a resource consumer, each virtual machine executing in an execution environment provided by a base hypervisor controlled by a resource-provider management server, the execution environment supporting a nested hypervisor controlled by a management server external to the resource-provider computing facility; and

configuring the base hypervisor to maintain the base hypervisor in a resource-provider control domain separate from the control domain that includes the nested hypervisor and external management server, the nested hypervisor being configured to provide an execution environment through a virtual hardware interface for second-level virtual machines and to provide load balancing among the second-level virtual machines;

wherein the distributed cloud-exchange system monitors the one or more virtual machines to detect potential security leaks.

15. The method of claim 14 wherein the nested hypervisor and base hypervisor are configured so that, for network traffic flowing to the nested hypervisor from a network and for network traffic emanating from source entities executing in the execution environment provided by the nested hypervisor, the network traffic passes through the base hypervisor in encrypted form so that the base hypervisor cannot access the contents of the messages that together comprise the network traffic.

16. The method of claim 14 wherein the nested hypervisor and base hypervisor are configured so that, for data flowing to the nested hypervisor from data-storage devices and for data transmitted to data-storage devices from source entities executing in the execution environment provided by the nested hypervisor, the data passes through the base hypervisor in encrypted form so that the base hypervisor cannot access the data.

17. The method of claim 14 further including installing, by the cloud-exchange system, a local cloud-exchange instance in a resource-provider computing facility as a virtual machine running a nested hypervisor controlled by a cloud-exchange-system management server and multiple second-level virtual machines running in the execution environment provided by the nested hypervisor.

18. The method of claim 17 wherein environment-specific configurations of the virtual machine running a nested hypervisor controlled by a cloud-exchange-system management server are carried out by an automated process invoked by the distributed cloud-exchange system following launching of the virtual machine within the resource-provider computing facility.

19. A physical data-storage device encoded with computer instructions that, when executed by processors with an automated resource-exchange system comprising multiple resource-exchange participants and a cloud-exchange system, control the automated resource-exchange system to create two management domains within a resource-exchange-system-participant computing facility, the method comprising:

launching, by the cloud-exchange system, one or more virtual machines within each of the computing facilities of one or more resource-exchange-system resource-provider participants selected by the distributed cloud-exchange system to provide computational resources to a resource-exchange-system resource-consumer participant, each virtual machine executing in an execution environment provided by a base hypervisor controlled by a resource-exchange-system-resource-provider-participant management server, the execution environment supporting a nested hypervisor controlled by a management server external to the resource-exchange-system-resource-provider-participant computing facility; and

configuring the base hypervisor to maintain the base hypervisor in a resource-provider control domain separate from the control domain that includes the nested hypervisor and external management server, the nested hypervisor being configured to provide an execution environment through a virtual hardware interface for second-level virtual machines and to provide load balancing among the second-level virtual machines;

wherein the distributed cloud-exchange system monitors the one or more virtual machines to detect potential security leaks.

Assignments (2)
CHANGE OF NAME Recorded Feb 27, 2024
From: VMWARE, INC.
To: VMWARE LLC
Reel/Frame 066692/0103 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 16, 2018
From: BEVERIDGE, DANIEL JAMES; TRIGALO, RICKY; LEW, JOERG
To: VMWARE, INC.
Reel/Frame 045824/0876 →
Continuity (3)
Continuation In Part 15285355 · Oct 4, 2016
Provisional Application 62380450 · Aug 28, 2016
Related Publication 20180260251A1 · Sep 13, 2018
References Cited (34)
US 5381407A · Chao · 1995 [cited by applicant]
US 6693651B2 · Biebesheimer et al. · 2004 [cited by applicant]
US 6934702B2 · Faybishenko et al. · 2005 [cited by applicant]
US 7013033B2 · Arena et al. · 2006 [cited by applicant]
US 20020002512A1 · Harpale · 2002 [cited by applicant]
US 20030050924A1 · Faybishenko et al. · 2003 [cited by applicant]
US 20060155633A1 · Fellenstein et al. · 2006 [cited by applicant]
US 20070038601A1 · Guha · 2007 [cited by applicant]
US 20070297414A1 · Gupta et al. · 2007 [cited by applicant]
US 20080059466A1 · Luo et al. · 2008 [cited by applicant]
US 20090030833A1 · Leung · 2009 [cited by applicant]
US 20110055385A1 · Tung et al. · 2011 [cited by applicant]
US 20120131591A1 · Moorthi · 2012 [cited by examiner]
US 20120198073A1 · Srikanth et al. · 2012 [cited by applicant]
US 20120221454A1 · Morgan · 2012 [cited by applicant]
US 20140053272A1 · Lukacs · 2014 [cited by examiner]
US 20150067171A1 · Yum et al. · 2015 [cited by applicant]
US 20150326449A1 · Melander et al. · 2015 [cited by applicant]
US 20160147556A1 · Hu · 2016 [cited by examiner]
US 20170063708A1 · Hsu et al. · 2017 [cited by applicant]
US 20170097841A1 · Chang · 2017 [cited by examiner]
Sharma P, Lee S, Guo T, Irwin D, Shenoy P. Spotcheck: Designing a derivative iaas cloud on the spot market. InProceedings of the Tenth European Conference on Computer Systems Apr. 17, 2015 (pp. 1-15). (Year: 2015). [cited by examiner]
Ben-Yehuda M, Day MD, Dubitzky Z, Factor M, Har'El N, Gordon A, Liguori A, Wasserman O, Yassour BA. The Turtles Project: Design and Implementation of Nested Virtualization. InOsdi Oct. 4, 2010 (vol. 10, pp. 423-436). (Y… [cited by examiner]
Williams D, Jamjoom H, Weatherspoon H. The Xen-Blanket: virtualize once, run everywhere. InProceedings of the 7th ACM european conference on Computer Systems Apr. 10, 2012 (pp. 113-126). (Year: 2012). [cited by examiner]
Zhang F, Chen J, Chen H, Zang B. Cloudvisor: retrofitting protection of virtual machines in multi-tenant cloud with nested virtualization. InProceedings of the Twenty-Third ACM Symposium on Operating Systems Principles … [cited by examiner]
Aliyu, S. et al., A Self-Tuning Procedure for Resource Management in InterCloud Computing, In 2016 IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C), 2016, pp. 326-333. [cited by applicant]
Amin, M. et al., Intercloud Message Exchange Middleware, In Proceedings of the 6th International Conference on Ubiquitous Information Management and Communication, 2012, pp. 1-7. [cited by applicant]
Bernstein, D. et al., Intercloud Directory and Exchange Protocol Detail Using XMPP and RDF, In 2010 6th World Congress on Services, IEEE, 2010, pp. 431-438. [cited by applicant]
Bijon, K. et al., Mitigating Multi-Tenancy Risks in laaS Cloud Through Constraints-Driven Virtual Resource Scheduling, In Proceedings of the 20th ACM Symposium on Access Control Models and Technologies, 2015, pp. 63-74. [cited by applicant]
Di Martino, B. et al., Towards an Ontology-Based Intercloud Resource Catalogue—The IEEE P2302 Intercloud Approach for a Semantic Resource Exchange, In 2015 IEEE International Conference on Cloud Engineering, 2015, pp. 4… [cited by applicant]
Grozev, N. et al., Inter-Cloud Architectures and Application Brokering: Taxonomy and Survey, Software: Practice and Experience, 2014, 44(3):369-390. [cited by applicant]
Hsu, C. et al., Algorithms and Architectures for Parallel Processing, 10th International Conference, ICA3PP 2010, Busan, Korea, May 21-23, 2010, Proceedings, Part I, Conference Proceedings ICA3PP, 2010. [cited by applicant]
Sotiriadis, S. et al., SimIC: Designing a New Inter-cloud Simulation Platform for Integrating Large-Scale Resource Management, In 2013 IEEE 27th International Conference on Advanced Information Networking and Applicatio… [cited by applicant]
Vilutis, G. et al., The QoGS Method Application for Selection of Computing Resources in Intercloud. Elektronika ir Elektrotechnika, 2013, 19(7):98-103. [cited by applicant]