Managing user identities in a managed multi-tenant service
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for processing data in a multi-tenant system. One of the methods includes receiving a data processing job associated with a user account of a user; determining to launch the data processing job on one or more cloud clusters of a cloud services provider; identifying a mirror account corresponding to the user, wherein the mirror account defines which cloud resources of the cloud services provider the user is permitted to access; obtaining a key for the mirror account; sending a request to launch the data processing job on the one or more cloud clusters, comprising sending data characterizing the data processing job, the mirror account of the user, and the obtained key to the one or more cloud clusters; and receiving output data associated with the data processing job from the one or more cloud clusters.
1. A method comprising:
receiving a data processing job associated with a user account of a user, wherein the user account is used to authenticate requests for accessing resources of an enterprise environment;
determining to launch the data processing job on one or more cloud clusters of a cloud services provider, wherein a cloud services account of the user is used to authenticate requests for accessing a first set of resources of the cloud services provider;
identifying a mirror account corresponding to the user account of the user, wherein:
the mirror account of the user is used to authenticate requests for accessing a second set of resources of the cloud services provider, and
the mirror account is distinct from the user account and the cloud services account;
obtaining a key for the mirror account;
sending a request to launch the data processing job on the one or more cloud clusters, comprising sending data characterizing the data processing job, the mirror account of the user, and the obtained key to the one or more cloud clusters, wherein the obtained key authenticates the request with the one or more cloud clusters; and
receiving output data associated with the data processing job from the one or more cloud clusters.
2. The method of claim 1 , wherein the one or more cloud clusters use the mirror account to authorize one or more requests submitted by the data processing job, the one or more requests being associated with at least a subset of the second set of resources of the cloud services provider, the authorizing comprising determining whether the user is permitted to access the subject.
3. The method of claim 1 , wherein the one or more cloud clusters use the mirror account to audit one or more requests submitted by the data processing job, the auditing comprising generating one or more logs associated with the requests and with the user.
4. The method of claim 1 , wherein obtaining the key for the mirror account comprises obtaining the key from a secure key store, and wherein the key is updated periodically.
5. The method of claim 1 , wherein determining to launch the data processing job on the one or more cloud clusters comprises identifying a user input associated with the data processing job, wherein the user input specifies that the data processing job should be launched on the one or more cloud clusters.
6. The method of claim 1 , wherein determining to launch the data processing job on the one or more cloud clusters comprises determining an ability of one or more clusters of the enterprise environment to execute the data processing job.
7. The method of claim 1 , further comprising:
receiving a second data processing job associated with a second user account of a second user;
determining to launch the data processing job on one or more clusters of the enterprise environment; and
executing the data processing job on the one or more clusters of the enterprise environment using the second user account.
8. The method of claim 1 , further comprising:
generating, by the one or more cloud clusters of the cloud services provider or by one or more processing resources of the enterprise environment, the mirror account for the user.
9. A system comprising:
an enterprise environment comprising one or more data centers, each data center having one or more data processing clusters; and
a multi-tenant cloud environment comprising one or more cloud-based data processing clusters and one or more cloud-based services,
wherein each data processing cluster of the enterprise environment is configured to perform operations comprising:
receiving a data processing job associated with a user account of a user, wherein the user account is used to authenticate requests for accessing resources of the enterprise environment;
determining to launch the data processing job on the one or more cloud-based data processing clusters of the multi-tenant cloud environment, wherein a cloud services account of the user is used to authenticate requests for accessing a first set of resources of the multi-tenant cloud environment;
identifying a mirror account corresponding to the user account of the user, wherein:
the mirror account of the user is used to authenticate requests for accessing a second set of resources of the multi-tenant cloud environment, and
the mirror account is distinct from the user account and the cloud services account;
obtaining a key for the mirror account;
sending a request to launch the data processing job on the one or more cloud-based data processing clusters, comprising sending data characterizing the data processing job, the mirror account of the user, and the obtained key to the one or more cloud-based data processing clusters, wherein the obtained key authenticates the request with the one or more cloud-based data processing clusters; and
receiving output data associated with the data processing job from the one or more cloud-based data processing clusters.
10. The system of claim 9 , wherein the one or more cloud-based data processing clusters use the mirror account to authorize one or more requests submitted by the data processing job, the one or more requests being associated with at least a subset of the second set of resources of the multi-tenant cloud environment, the authorizing comprising determining whether the user is permitted to access the subset.
11. The system of claim 9 , wherein the one or more cloud-based data processing clusters use the mirror account to audit one or more requests submitted by the data processing job, the auditing comprising generating one or more logs associated with the requests and with the user.
12. The system of claim 9 , wherein obtaining the key for the mirror account comprises obtaining the key from a secure key store, and wherein the key is updated periodically.
13. The system of claim 9 , wherein determining to launch the data processing job on the one or more cloud-based data processing clusters comprises identifying a user input associated with the data processing job, wherein the user input specifies that the data processing job should be launched on the one or more cloud-based data processing clusters.
14. The system of claim 9 , wherein determining to launch the data processing job on the one or more cloud-based data processing clusters comprises determining an ability of the data centers of the enterprise environment to execute the data processing job.
15. The system of claim 9 , wherein each data processing cluster of the enterprise environment is configured to perform operations further comprising:
receiving a second data processing job associated with a second user account of a second user;
determining to launch the data processing job on one or more particular data processing clusters of a particular data center of the enterprise environment; and
executing the data processing job on the one or more particular data processing clusters of the particular data center of the enterprise environment using the second user account.
16. The system of claim 9 , wherein each data processing cluster of the enterprise environment is configured to perform operations further comprising:
generating, by the one or more cloud-based data processing clusters of the multi-tenant cloud environment or by the one or more data processing clusters of a respective data center of the enterprise environment, the mirror account for the user.
17. One or more non-transitory computer storage media encoded with computer program instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:
receiving a data processing job associated with a user account of a user, wherein the user account is used to authenticate requests for accessing resources of an enterprise environment;
determining to launch the data processing job on one or more cloud clusters of a cloud services provider, wherein a cloud services account of the user is used to authenticate requests for accessing a first set of resources of the cloud services provider;
identifying a mirror account corresponding to the user account of the user, wherein:
the mirror account of the user is used to authenticate requests for accessing a second set of resources of the cloud services provider, and
the mirror account is distinct from the user account and the cloud services account;
obtaining a key for the mirror account;
sending a request to launch the data processing job on the one or more cloud clusters, comprising sending data characterizing the data processing job, the mirror account of the user, and the obtained key to the one or more cloud clusters, wherein the obtained key authenticates the request with the one or more cloud clusters; and
receiving output data associated with the data processing job from the one or more cloud clusters.
18. The one or more non-transitory computer storage media of claim 17 , wherein the one or more cloud clusters use the mirror account to authorize one or more requests submitted by the data processing job, the one or more requests being associated with at least a subset of the second set of resources cloud services provider, the authorizing comprising determining whether the user is permitted to access the subset.
19. The one or more non-transitory computer storage media of claim 17 , wherein the one or more cloud clusters use the mirror account to audit one or more requests submitted by the data processing job, the auditing comprising generating one or more logs associated with the requests and with the user.
20. The one or more non-transitory computer storage media of claim 17 , wherein obtaining the key for the mirror account comprises obtaining the key from a secure key store, and wherein the key is updated periodically.
21. The one or more non-transitory computer storage media of claim 17 , wherein determining to launch the data processing job on the one or more cloud clusters comprises identifying a user input associated with the data processing job, wherein the user input specifies that the data processing job should be launched on the one or more cloud clusters.
22. The one or more non-transitory computer storage media of claim 17 , wherein determining to launch the data processing job on the one or more cloud clusters comprises determining an ability of one or more clusters of the enterprise environment to execute the data processing job.
23. The one or more non-transitory computer storage media of claim 17 , wherein the operations further comprise:
receiving a second data processing job associated with a second user account of a second user;
determining to launch the data processing job on one or more clusters of the enterprise environment; and
executing the data processing job on the one or more clusters of the enterprise environment using the second user account.
24. The one or more non-transitory computer storage media of claim 17 , wherein the operations further comprise:
generating, by the one or more cloud clusters of the cloud services provider or by one or more processing resources of the enterprise environment, the mirror account for the user.