IP Library Granted Patent US 11,616,686
Granted Patent B1
US 11,616,686 · App. 15/820,089 · Granted Mar 28, 2023

Cluster management

Inventors: Santosh Chandrachood (Milpitas, CA); Gayatri Ramesh Deo (Mountain View, CA); Ankit Kamboj (Redmond, WA); Lukasz Misiuda (Redmond, WA); Amit Purohit (Issaquah, WA); Aravind Ramarathinam (Sammamish, WA); Ramkumar Kamalapuram Sugavanam (Bothell, WA); Vinayak Thapliyal (Seattle, WA); Linchi Zhang (Sunnyvale, CA); Min Zhou (Sunnyvale, CA)
Assignee: Amazon Technologies, Inc.
H04L41/0803G06F9/455G06F9/4881G06F9/5005H04L63/102H04L63/108G06F2209/505G06F2209/5011
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,616,686
App. No.
15/820,089
Granted
Mar 28, 2023
Kind
B1
Abstract

In response to receiving a parallel processing job from a customer, a system operated by a computing resource service provider allocates and configures a cluster of computer systems capable of executing the job. In an embodiment, each computer system is configured with a first network stack that allows access to resources of the computing resource service provider and a second network stack that allows access to resources of the customer. In an embodiment, the state of the cluster is monitored by the system via the first network stack. In an embodiment, the system deploys a set of tasks on the cluster for fulfilling the processing job. In an embodiment, the tasks have access to the second network stack so that each task can access resources of the customer.

Claims (55)

1. A system, comprising:

one or more processors; and

memory storing instructions that, as a result of being executed by the one or more processors, cause the system to:

obtain a request to perform a job using data maintained by a computing resource of first computing resources, where the system is prevented from accessing the first computing resources within a first network;

configure a gateway and a group of computer systems via a second network, where:

the system has access to second computing resources within the second network;

the gateway has access to the second network; and

a first computer system in the group of computer systems has access to the first network and the second network;

cause, by execution of a job agent at the first computer system, the first computer system to obtain, from the system, a set of operations identified in the job and further load, over the second network, the set of operations onto computer systems of the group of computer systems; and

cause the set of operations to be executed on the group of computer systems, a first operation in the set of operations causing the first computer system to access the computing resource within the first network.

2. The system of claim 1 , wherein the instructions, as a result of being executed on the one or more processors, further cause the system to determine a set of parameters for the group of computer systems based at least in part on a characteristic of the request.

3. The system of claim 2 , wherein the instructions, as a result of being executed on the one or more processors, further cause the system to allocate a number of computer systems to the group of computer systems so that the number of computer systems in the group of computer systems is sufficient to fulfill the request.

4. The system of claim 3 , wherein:

the group of computer systems is a group of virtual computer systems; and

the system allocates the number of computer systems by instantiating a number of virtual computer systems.

5. The system of claim 2 , wherein:

the request identifies a MapReduce job to be performed on the group of computer systems; and

the set of operations is a set of tasks that, as a result of being performed, complete the MapReduce job.

6. The system of claim 1 , wherein:

the first computer system in the group of computer systems includes a first network namespace to access the first network; and

the first computer system in the group of computer systems includes a second network namespace to access the second network.

7. The system of claim 1 , wherein:

the group of computer systems is monitored from the first computer system via the second network;

a second computer system is connected to the first computer system; and

the set of operations is monitored by the second computer system via the first network.

8. The system of claim 1 , wherein:

the group of computer systems includes a first computer system and one or more second computer systems, where the first computer system coordinates the execution of operations on the one or more second computer systems; and

the set of operations is executed on the one or more second computer systems.

9. The system of claim 1 , wherein the instructions that cause the system to configure the gateway further configure the gateway to obtain a status of the set of operations from the first computer system and further provide the status to the system.

10. A computer-implemented method, comprising:

providing a request to a computer system to perform a job using data maintained by a computing resource of first computing resources, where the computer system is prevented from accessing the first computing resources within a first network;

configuring a gateway and a group of computer systems via a second network, where:

the system has access to second computing resources within the second network;

the gateway has access to the second network; and

a first computer system in the group of computer systems has access to the first network and the second network;

causing, by execution of a job agent at the first computer system, the first computer system to obtain, from the computer system, a set of operations identified in the job and further load, over the second network, the set of operations onto computer systems of the group of computer systems; and

causing the set of operations to be executed on the group of computer systems, a first operation in the set of operations causing the first computer system to access the computing resource within the first network.

11. The computer-implemented method of claim 10 , wherein a subset of operations of the set of operations is performed in parallel by the group of computer systems.

12. The computer-implemented method of claim 10 , wherein the first network is isolated from the second network.

13. The computer-implemented method of claim 12 , wherein the first network is within a data plane namespace and the second network is within a control plane namespace.

14. The computer-implemented method of claim 10 , further comprising determining that the group of computer systems has access to the computer resource over the first network.

15. The computer-implemented method of claim 10 , wherein at least one of the second computing resources is provided by a storage service of a service provider associated with the computer system.

16. A non-transitory computer-readable storage medium having stored thereon executable instructions that, as a result of being executed by one or more processors of a computer system, cause the computer system to:

obtain a request to perform a job using data maintained by a computing resource of first computing resources, where the computer system is prevented from accessing the first computing resources within a first network;

configure a gateway and a group of computer systems via a second network, where:

the system has access to second computing resources within the second network;

the gateway has access to the second network; and

a first computer system in the group of computer systems has access to the first network and the second network;

cause, by execution of a job agent at the first computer system, the first computer system to obtain, from the computer system, a set of operations identified in the job and further load, over the second network, the set of operations onto computer systems of the group of computer systems; and

cause the set of operations to be executed on the group of computer systems, a first operation in the set of operations causing the first computer system to access the computing resource within the first network.

17. The non-transitory computer-readable storage medium of claim 16 , wherein the instructions further include instructions that, as a result of being executed by the one or more processors, cause the computer system to determine that the group of computer systems has access to the computer resource over the first network.

18. The non-transitory computer-readable storage medium of claim 16 , wherein the instructions further include instructions that, as a result of being executed by the one or more processors, cause the computer system to instantiate the group of computer systems in response to a job description that is included in the request.

19. The non-transitory computer-readable storage medium of claim 18 , wherein the instructions that cause the computer system to instantiate the group of computer systems further include instructions that, as a result of being executed by the one or more processors, cause the computer system to determine a set of parameters for the group of computer systems based at least in part on information included in the job description.

20. The non-transitory computer-readable storage medium of claim 19 , wherein the set of parameters identifies a quantity of processing nodes of the group of computer systems to satisfy the job description.

21. The non-transitory computer-readable storage medium of claim 19 , wherein the request includes a job description that describes a MapReduce job.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 21, 2017
From: CHANDRACHOOD, SANTOSH; DEO, GAYATRI RAMESH; KAMBOJ, ANKIT; MISIUDA, LUKASZ; PUROHIT, AMIT; RAMARATHINAM, ARAVIND; KAMALAPURAM SUGAVANAM, RAMKUMAR; THAPLIYAL, VINAYAK; ZHANG, LINCHI; ZHOU, MIN
To: AMAZON TECHNOLOGIES, INC.
Reel/Frame 044194/0638 →
Cited By (6)
US 12,204,946 US 12,231,351 US 12,423,124 US 12,423,141 US 12,687,823 US 12,705,093