Providing external access to a processing platform
An apparatus, and a method, performed by one or more processors are disclosed. The method may comprise receiving a build request associated with performing an external data processing task on a first data set, the first data set being stored in memory associated with a data processing platform to be performed at a system external to the data processing platform. The method may also comprise generating a task identifier for the data processing task, and providing, in association with the task identifier, the first data set to an agent associated with the external system with an indication of the data processing task, the agent being arranged to cause performance of the task at the external system, to receive a second data set resulting from performance of the task, and to provide the second data set and associated metadata indicative of the transformation. The method may also comprise receiving the second data set and metadata from the agent associated with the external system and storing the second data set and associated metadata.
1. A method performed by one or more processors configured to implement an agent of an external system, the method comprising:
identifying a request associated with performing a data processing task to be performed on a first dataset;
receiving, from a data processing platform, the first dataset and a task identifier for the data processing task to be performed on the first dataset, wherein the first dataset is transmitted from the data processing platform to the agent;
causing performance of the data processing task at the external system; and
transmitting an indication of one or more updates made to code to perform the data processing task.
2. The method of claim 1 further comprising:
receiving a second dataset resulting from performance of the data processing task; and
sending the second dataset and associated metadata to the data processing platform.
3. The method of claim 2 , wherein the first dataset is stored in memory associated with the data processing platform.
4. The method of claim 2 , further comprising transmitting one or more build queries to the data processing platform for causing receiving of the request,
wherein the one or more build queries are polling queries periodically transmitted to the data processing platform.
5. The method of claim 2 , further comprising receiving a token associated with the request for transmitting to the data processing platform when requesting the first dataset and/or sending the second dataset such that the data processing platform may determine that they correspond to the request.
6. The method of claim 1 , further comprising receiving an update query from the data processing platform, and responsive thereto, transmitting the indication of the one or more updates.
7. The method of claim 1 , further comprising, upon receiving the request, creating a directory for holding the first dataset and the second dataset and associated metadata, and removing the directory subsequent to the second dataset and associated metadata being sent to the data processing platform.
8. A computer system comprising:
one or more processors configured to execute program instructions to cause the computer system to implement an agent configured to:
identify a request associated with performing a data processing task to be performed on a first dataset;
receive, from a data processing platform, the first dataset and a task identifier for the data processing task to be performed on the first dataset, wherein the first dataset is transmitted from the data processing platform to the agent;
cause performance of the data processing task at an external system of the computer system; and
transmit an indication of one or more updates made to code to perform the data processing task.
9. The computer system of claim 8 , wherein the agent is further configured to:
receive a second dataset resulting from performance of the data processing task; and
send the second dataset and associated metadata to the data processing platform.
10. The computer system of claim 9 , wherein the first dataset is stored in memory associated with the data processing platform.
11. The computer system of claim 9 , wherein the one or more processors are configured to execute program instructions to further cause the computer system to implement the external system configured to:
receive the data processing task from the agent;
perform the data processing task;
generate the second dataset resulting from performance of the data processing task; and
provide the second dataset to the agent.
12. The computer system of claim 9 , wherein the agent is further configured to:
transmit one or more build queries to the data processing platform for causing receiving of the request,
wherein the one or more build queries are polling queries periodically transmitted to the data processing platform.
13. The computer system of claim 9 , wherein the agent is further configured to:
receive a token associated with the request for transmitting to the data processing platform when requesting the first dataset and/or sending the second dataset such that the data processing platform may determine that they correspond to the request.
14. The computer system of claim 8 , wherein the agent is further configured to:
receive an update query from the data processing platform, and responsive thereto, transmit the indication of the one or more updates.
15. The computer system of claim 8 , wherein the agent is further configured to:
upon receiving the request, create a directory for holding the first dataset and the second dataset and associated metadata; and
remove the directory subsequent to the second dataset and associated metadata being sent to the data processing platform.