System and methods for securely storing data for efficient access by cloud-based computing instances
Systems and methods for securely storing data for efficient access by cloud-based computing instances is provided. In one or more examples, a computing hub can receive one or more access requests to data stored within a persistent data storage computing resources that in connected to the computing hub. The computing hub can be configured to determine if the access request is from an authorized computing resource, and can then generate one or more tokens that provide access to the computing resource. The one or more tokens can include information regarding the IP address of the requesting cloud-based computing resource, and each time that the cloud-based computing resource uses the token to request access to the stored data, the computing hub can check the IP address of the computing resource against the IP address indicated on the token to decide whether or not to grant access to the data.
1. A method for providing secure access to data stored on a computing resource to one or more cloud-based computing resources, the method comprising:
receiving a data set;
receiving one or more processing instructions associated with the received data set, wherein the one or more processing instructions corresponds to the received data set and includes one or more processing tasks to be performed on the received data set;
generating one or more data containers within a first computing resource based on the received one or more processing instructions;
storing the data set in the one or more containers within the first computing resource;
configuring a second computing resource on one or more cloud-based computing platforms based on the received one or more processing instructions, wherein generating the computing resource comprises selecting one or more pre-defined specifications of computing clusters based on the received one or more processing instructions, and wherein configuring a second computing resource on the one or more cloud-based computing platforms comprises embedding one or more hooks in the second computing resource configured to record lineage data about one or more transactions processed by the second computing resource;
receiving one or more requests from the second computer resource for access to the data set stored in the one or more containers within the first computing resource;
generating one or more tokens for access to the data set stored in the one or more containers within the first computing resource, wherein the generated tokens are based on one or more parameters associated with the second computing resource that is provisioned on one or more cloud-based computing platforms; and
transmitting the one or more tokens for access to the data set stored in the one or more containers within first computing resource to the second computing resource.
2. The method of claim 1 , wherein the one or more parameters includes an IP address of the second computing resource that transmitted the request, and wherein generating one or more tokens for access to the data set stored in the one or more containers within the first computing resource includes generating the one or more tokens for access to the data set based on the IP address of the request.
3. The method of claim 1 , wherein the method further comprises generating a storage account within the first computing resource based on a source of the received data set.
4. The method of claim 1 , wherein the method further comprises encrypting the data set.
5. A system for providing secure access to data stored on a computing resource to one or more cloud-based computing resources, the system comprising:
one or more processors;
a memory; and
a first computing resource;
wherein the one or more processors are configured to execute instructions stored on the memory that cause the system to:
receive a data set;
receive one or more processing instructions associated with the received data set, wherein the one or more processing instructions corresponds to the received data set and includes one or more processing tasks to be performed on the received data set;
generate one or more data containers within the first computing resource based on the received one or more processing instructions;
store the data set in the one or more containers within the first computing resource;
configure a second computing resource on one or more cloud-based computing platforms based on the received one or more processing instructions, wherein generating the computing resource comprises selecting one or more pre-defined specifications of computing clusters based on the received one or more processing instructions, and wherein configuring a second computing resource on the one or more cloud-based computing platforms comprises embedding one or more hooks in the second computing resource configured to record lineage data about one or more transactions processed by the second computing resource;
receive one or more requests from the second computer resource for access to the data set stored in the one or more containers within the first computing resource;
generate one or more tokens for access to the data set stored in the one or more containers within the first computing resource, wherein the generated tokens are based on one or more parameters associated with the second computing resource that is provisioned on one or more cloud-based computing platforms; and
transmit the one or more tokens for access to the data set stored in the one or more containers within first computing resource to the second computing resource.
6. The system of claim 5 , wherein the one or more parameters includes an IP address of the second computing resource that transmitted the request, and wherein generating one or more tokens for access to the data set stored in the one or more containers within the first computing resource includes generating the one or more tokens for access to the data set based on the IP address of the request.
7. The system of claim 5 , wherein the system is further caused to generate a storage account within the first computing resource based on a source of the received data set.
8. The system of claim 5 , wherein the system is further caused to encrypt the data set.
9. The system of claim 5 , wherein receiving one or more requests from the second computing resource for access to the data set stored in the one or more containers within the first computing resource includes receiving requests from the second computing resource, wherein the requests are based on the received one or more processing instructions.
10. A non-transitory computer-readable storage medium storing one or more programs for providing secure access to data stored on a computing resource to one or more cloud-based computing resources the one or more programs configured to be executed by one or more processors communicatively coupled to one or more processors of a system, the one or more programs including instructions to cause the system to:
receive a data set;
receive one or more processing instructions associated with the received data set, wherein the one or more processing instructions corresponds to the received data set and includes one or more processing tasks to be performed on the received data set;
generate one or more data containers within the first computing resource based on the received one or more processing instructions;
store the data set in the one or more containers within the first computing resource;
configure a second computing resource on one or more cloud-based computing platforms based on the received one or more processing instructions, wherein generating the computing resource comprises selecting one or more pre-defined specifications of computing clusters based on the received one or more processing instructions, and wherein configuring a second computing resource on the one or more cloud-based computing platforms comprises embedding one or more hooks in the second computing resource configured to record lineage data about one or more transactions processed by the second computing resource;
receive one or more requests from the second computer resource for access to the data set stored in the one or more containers within the first computing resource;
generate one or more tokens for access to the data set stored in the one or more containers within the first computing resource, wherein the generated tokens are based on one or more parameters associated with the second computing resource that is provisioned on one or more cloud-based computing platforms; and
transmit the one or more tokens for access to the data set stored in the one or more containers within first computing resource to the second computing resource.
11. The non-transitory computer-readable storage medium of claim 10 , wherein the one or more parameters includes an IP address of the second computing resource that transmitted the request, and wherein generating one or more tokens for access to the data set stored in the one or more containers within the first computing resource includes generating the one or more tokens for access to the data set based on the IP address of the request.
12. The non-transitory computer-readable storage medium of claim 10 , wherein the system is further caused to generate a storage account within the first computing resource based on a source of the received data set.
13. The non-transitory computer-readable storage medium of claim 10 , wherein the system is further caused to encrypt the data set.
14. The non-transitory computer-readable storage medium of claim 10 , wherein receiving one or more requests from the second computing resource for access to the data set stored in the one or more containers within the first computing resource includes receiving requests from the second computing resource, wherein the requests are based on the received one or more processing instructions.