Cloud computing platform architecture
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating recipes having platform-independent, resource-agnostic commands for performing work in a cloud computing platform. One of the systems includes a plurality of executors, each executor being operable to receive a recipe comprising one or more platform independent, resource agnostic commands, each command corresponding to a requested service to be performed by an execution engine on an underlying computing platform, translate each platform independent, resource agnostic command of the recipe into one or more platform independent, resource specific requests that satisfy the requested service on the underlying computing platform and make each of the one or more platform independent, resource specific requests to one of the one or more execution engines using a platform independent execution engine application programming interface (API).
1 . A system comprising:
a plurality of first computers and a plurality of first storage devices storing instructions that are operable, when executed by the plurality of first computers, to cause the plurality of first computers to perform operations comprising:
receiving a sequence of platform-independent, resource-agnostic commands for execution of a computing service on any one of multiple, different underlying computing platforms, wherein:
the platform-independent, resource-agnostic commands are a) independent of all of the multiple, different underlying computing platforms and b) independent of a plurality of resource types included in the multiple, different underlying computing platforms and independent of instances of the plurality of resource types; and
the platform-independent, resource-agnostic commands do not identify any particular resource type of the plurality of resource types, wherein the plurality of resource types comprises containers and virtual machines;
generating, from the sequence of platform-independent, resource-agnostic commands, a sequence of platform-independent, resource-specific requests, wherein:
the platform-independent, resource-specific requests are a) independent of all of the multiple, different underlying computing platforms and b) specific to a particular resource type, of the plurality of resource types, that is included in one or more of the multiple, different underlying computing platforms; and
the platform-independent, resource-specific requests comprise requests for allocating cloud computing resources of the particular resource type and using the allocated cloud computing resources to effectuate the platform-independent, resource-agnostic commands on one of the multiple, different underlying distributed computing platforms; and
providing, to an underlying computing platform from the multiple, different underlying computing platforms, the sequence of generated platform-independent, resource-specific requests to cause the underlying computing platform to perform platform-specific, resource instance-specific operations that correspond to the sequence of platform-independent, resource-specific requests; and
a plurality of second computers that implement the underlying computing platform and a plurality of second storage devices storing instructions that are operable, when executed by the plurality of second computers, to cause the plurality of second computers to perform operations comprising:
receiving, from at least one of the plurality of first computers, the sequence of generated platform-independent, resource-specific requests;
determining, for each request in the sequence of platform-independent, resource-specific requests, one or more platform-specific, resource instance-specific operations, wherein each platform-specific, resource instance-specific operation specifies one or more instances on the underlying computing platform of the particular resource type that is identified by the corresponding platform-independent, resource-specific request; and
performing the platform-specific, resource instance-specific operations to cause execution of the computing service.
2 . The system of claim 1 , wherein performing the platform-specific, resource instance-specific operations to cause execution of the computing service comprises:
performing one or more of the platform-specific, resource instance-specific operations to allocate, on the underlying computing platform, a particular instance of the particular resource type identified by the respective platform-independent, resource-specific request.
3 . The system of claim 2 , wherein performing the platform-specific, resource instance-specific operations to cause execution of the computing service further comprises:
performing one or more of the platform-specific, resource instance-specific operations to copy a target of one of the platform-independent, resource-agnostic commands in the sequence of platform-independent, resource-agnostic commands into the particular instance of the particular resource type.
4 . The system of claim 3 , wherein performing the platform-specific, resource instance-specific operations to cause execution of the computing service further comprises:
performing one or more of the platform-specific, resource instance-specific operations to execute the target.
5 . The system of claim 1 , further comprising:
a plurality of third computers and a plurality of third storage devices storing instructions that are operable, when executed by the plurality of third computers, to cause the plurality of third computers to perform operations comprising:
receiving a request to prepare an application for execution on one of the multiple, different underlying computing platforms;
in response to receiving the request to prepare the application for execution on one of the multiple, different underlying computing platforms, generating the sequence of platform-independent, resource-agnostic commands for preparing the application for execution; and
providing the sequence of platform-independent, resource-agnostic commands to at least one of the plurality of first computers.
6 . The system of claim 1 , further comprising:
a plurality of fourth computers and a plurality of fourth storage devices storing instructions that are operable, when executed by the plurality of fourth computers, to cause the plurality of fourth computers to perform operations comprising:
receiving a request to execute an application on one of the multiple, different underlying computing platforms;
in response to receiving the request to execute the application on one of the multiple, different underlying computing platforms, generating the sequence of platform-independent, resource-agnostic commands for executing the application; and
providing the sequence of platform-independent, resource-agnostic commands to at least one of the plurality of first computers.
7 . The system of claim 1 , wherein providing, to the underlying computing platform from the multiple, different underlying computing platforms, the sequence of generated platform-independent, resource-specific requests comprises:
determining whether data for the sequence of platform-independent, resource-agnostic commands identifies a particular one of the multiple, different underlying computing platforms for execution of the computing service; and
in response to determining that the data for the sequence of platform-independent, resource-agnostic commands identifies a particular one of the multiple, different underlying computing platforms for execution of the computing service, providing, to the particular one of the multiple, different underlying computing platforms, the sequence of generated platform-independent, resource-specific requests.
8 . The system of claim 1 , wherein providing, to the underlying computing platform from the multiple, different underlying computing platforms, the sequence of generated platform-independent, resource-specific requests comprises:
providing, to two or more of the multiple, different underlying computing platforms, the sequence of generated platform-independent, resource-specific requests to cause each of the two or more underlying computing platforms to perform respective different platform-specific, resource instance-specific operations that correspond to the sequence of platform-independent, resource-specific requests.
9 . The system of claim 1 , wherein the computing service comprises one or more of:
a task that will be performed at least once; or
a long-running process that will be performed for an indefinite length of time.
10 . A computer-implemented method comprising:
receiving, by at least one of a plurality of executors, a sequence of platform-independent, resource-agnostic commands for execution of a computing service on any one of multiple, different underlying computing platforms, wherein:
the platform-independent, resource-agnostic commands are a) independent of all of the multiple, different underlying computing platforms and b) independent of a plurality of resource types included the multiple, different underlying computing platforms and independent of instances of the plurality of resource types; and
the platform-independent, resource-agnostic commands do not identify any particular resource type of the plurality of resource types, wherein the plurality of resource types comprises containers and virtual machines;
generating, by one or more of the plurality of executors and from the sequence of platform-independent, resource-agnostic commands, a sequence of platform-independent, resource-specific requests, wherein:
the platform-independent, resource-specific requests are a) independent of all of the multiple, different underlying computing platforms and b) specific to a particular resource type, of the plurality of resource types, that is included in one or more of the multiple, different underlying computing platforms; and
the platform-independent, resource-specific requests comprise requests for allocating cloud computing resources of the particular resource type and using the allocated cloud computing resources to effectuate the platform-independent, resource-agnostic commands on one of the multiple, different underlying distributed computing platforms; and
providing, by one or more of the plurality of executors and to at least one of a plurality of execution engines each having an underlying computing platform from the multiple, different underlying computing platforms, the sequence of generated platform-independent, resource-specific requests to cause the execution engines to perform platform-specific, resource instance-specific operations for the underlying computing platform that correspond to the sequence of platform-independent, resource-specific requests;
determining, by one or more of the plurality of execution engines and for each request in the sequence of platform-independent, resource-specific requests, one or more platform-specific, resource instance-specific operations, wherein each platform-specific, resource instance-specific operation specifies one or more instances on the underlying computing platform of the particular resource type that is identified by the corresponding platform-independent, resource-specific request; and
performing, by one or more of the plurality of execution engines, the platform-specific, resource instance-specific operations to cause execution of the computing service.
11 . The method of claim 10 , wherein performing the platform-specific, resource instance-specific operations to cause execution of the computing service comprises:
performing, by one or more of the plurality of execution engines, one or more of the platform-specific, resource instance-specific operations to allocate, on the underlying computing platform, a particular instance of the particular resource type identified by the respective platform-independent, resource-specific request.
12 . The method of claim 11 , wherein performing the platform-specific, resource instance-specific operations to cause execution of the computing service further comprises:
performing, by one or more of the plurality of execution engines, one or more of the platform-specific, resource instance-specific operations to copy a target of one of the platform-independent, resource-agnostic commands in the sequence of platform-independent, resource-agnostic commands into the particular instance of the particular resource type.
13 . The method of claim 12 , wherein performing the platform-specific, resource instance-specific operations to cause execution of the computing service further comprises:
performing, by one or more of the plurality of execution engines, one or more of the platform-specific, resource instance-specific operations to execute the target.
14 . The method of claim 10 , further comprising:
receiving, by at least one of a plurality of stagers, a request to prepare an application for execution on one of the multiple, different underlying computing platforms;
in response to receiving the request to prepare the application for execution on one of the multiple, different underlying computing platforms, generating, by one or more of the plurality of stagers, the sequence of platform-independent, resource-agnostic commands for preparing the application for execution; and
providing, by one or more of the plurality of stagers to at least one of the plurality of executors, the sequence of platform-independent, resource-agnostic commands.
15 . The method of claim 10 , further comprising:
receiving, by at least one of a plurality of application managers, a request to execute an application on one of the multiple, different underlying computing platforms;
in response to receiving the request to execute the application on one of the multiple, different underlying computing platforms, generating, by one or more of the plurality of application managers, the sequence of platform-independent, resource-agnostic commands for executing the application; and
providing, by one or more of the plurality of application managers to at least one of the plurality of executors, the sequence of platform-independent, resource-agnostic commands.
16 . The method of claim 10 , wherein providing, by one or more of the plurality of executors and to at least one of a plurality of execution engines each having an underlying computing platform from the multiple, different underlying computing platforms, the sequence of generated platform-independent, resource-specific requests comprises:
determining whether data for the sequence of platform-independent, resource-agnostic commands identifies a particular one of the multiple, different underlying computing platforms for execution of the computing service; and
in response to determining that the data for the sequence of platform-independent, resource-agnostic commands identifies a particular one of the multiple, different underlying computing platforms for execution of the computing service, providing, to at least one execution engine having the particular one of the multiple, different underlying computing platforms, the sequence of generated platform-independent, resource-specific requests.
17 . The method of claim 10 , wherein providing, by one or more of the plurality of executors and to at least one of a plurality of execution engines each having an underlying computing platform from the multiple, different underlying computing platforms, the sequence of generated platform-independent, resource-specific requests comprises:
providing, to two or more execution engines each having a different underlying computing platform from the multiple underlying computing platforms, the sequence of generated platform-independent, resource-specific requests to cause each of the two or more underlying computing platforms to perform respective different platform-specific, resource instance-specific operations that correspond to the sequence of platform-independent, resource-specific requests.
18 . A computer program product, encoded on one or more non-transitory computer storage media, comprising instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:
receiving, by at least one of a plurality of executors, a sequence of platform-independent, resource-agnostic commands for execution of a computing service on any one of multiple, different underlying computing platforms, wherein:
the platform-independent, resource-agnostic commands are a) independent of all of the multiple, different underlying computing platforms and b) independent of a plurality of resource types included in the multiple, different underlying computing platforms and independent of instances of the plurality of resoure types; and
the platform-independent, resource-agnostic commands do not identify any particular resource type of the plurality of resource types, wherein the plurality of resource types comprises containers and virtual machines;
generating, by one or more of the plurality of executors and from the sequence of platform-independent, resource-agnostic commands, a sequence of platform-independent, resource-specific requests, wherein:
the platform-independent, resource-specific requests are a) independent of all of the multiple, different underlying computing platforms and b) specific to a particular resource type, of the plurality of resource types, that is included in one or more of the multiple, different underlying computing platforms; and
the platform-independent, resource-specific requests comprise requests for allocating cloud computing resources of the particular resource type and using the allocated cloud computing resources to effectuate the platform-independent, resource-agnostic commands on one of the multiple, different underlying distributed computing platforms; and
providing, by one or more of the plurality of executors and to at least one of a plurality of execution engines each having an underlying computing platform from the multiple, different underlying computing platforms, the sequence of generated platform-independent, resource-specific requests to cause the execution engines to perform platform-specific, resource instance-specific operations for the underlying computing platform that correspond to the sequence of platform-independent, resource-specific requests;
determining, by one or more of the plurality of execution engines and for each request in the sequence of platform-independent, resource-specific requests, one or more platform-specific, resource instance-specific operations, wherein each platform-specific, resource instance-specific operation specifies one or more instances on the underlying computing platform of the particular resource type that is identified by the corresponding platform-independent, resource-specific request; and
performing, by one or more of the plurality of execution engines, the platform-specific, resource instance-specific operations to cause execution of the computing service.
19 . The computer program product of claim 18 , wherein performing the platform-specific, resource instance-specific operations to cause execution of the computing service comprises:
performing, by one or more of the plurality of execution engines, one or more of the platform-specific, resource instance-specific operations to allocate, on the underlying computing platform, a particular instance of the particular resource type identified by the respective platform-independent, resource-specific request.
20 . The computer program product of claim 19 , wherein performing the platform-specific, resource instance-specific operations to cause execution of the computing service further comprises:
performing, by one or more of the plurality of execution engines, one or more of the platform-specific, resource instance-specific operations to copy a target of one of the platform-independent, resource-agnostic commands in the sequence of platform-independent, resource-agnostic commands into the particular instance of the particular resource type.
21 . The computer program product of claim 20 , wherein performing the platform-specific, resource instance-specific operations to cause execution of the computing service further comprises:
performing, by one or more of the plurality of execution engines, one or more of the platform-specific, resource instance-specific operations to execute the target.
22 . The computer program product of claim 18 , the operations further comprising:
receiving, by at least one of a plurality of stagers, a request to prepare an application for execution on one of the multiple, different underlying computing platforms;
in response to receiving the request to prepare the application for execution on one of the multiple, different underlying computing platform, generating, by one or more of the plurality of stagers, the sequence of platform-independent, resource-agnostic commands for preparing the application for execution; and
providing, by one or more of the plurality of stagers to at least one of the plurality of executors, the sequence of platform-independent, resource-agnostic commands.
23 . The computer program product of claim 18 , the operations further comprising:
receiving, by at least one of a plurality of application managers, a request to execute an application on one of the multiple, different underlying computing platforms;
in response to receiving the request to execute the application on one of the multiple, different underlying computing platforms, generating, by one or more of the plurality of application managers, the sequence of platform-independent, resource-agnostic commands for executing the application; and
providing, by one or more of the plurality of application managers to at least one of the plurality of executors, the sequence of platform-independent, resource-agnostic commands.
24 . The computer program product of claim 18 , wherein providing, by one or more of the plurality of executors and to at least one of a plurality of execution engines each having an underlying computing platform from the multiple, different underlying computing platforms, the sequence of generated platform-independent, resource-specific requests comprises:
determining whether data for the sequence of platform-independent, resource-agnostic commands identifies a particular one of the multiple, different underlying computing platforms for execution of the computing service; and
in response to determining that the data for the sequence of platform-independent, resource-agnostic commands identifies a particular one of the multiple, different underlying computing platforms for execution of the computing service, providing, to at least one execution engine having the particular one of the multiple, different underlying computing platforms, the sequence of generated platform-independent, resource-specific requests.
25 . The computer program product of claim 18 , wherein providing, by one or more of the plurality of executors and to at least one of a plurality of execution engines each having an underlying computing platform from the multiple, different underlying computing platforms, the sequence of generated platform-independent, resource-specific requests comprises:
providing, to two or more execution engines each having a different underlying computing platform from the multiple underlying computing platforms, the sequence of generated platform-independent, resource-specific requests to cause each of the two or more underlying computing platforms to perform respective different platform-specific, resource instance-specific operations that correspond to the sequence of platform-independent, resource-specific requests.