Providing external access to a processing platform
An apparatus, and a method, performed by one or more processors are disclosed. The method may comprise receiving a build request associated with performing an external data processing task on a first data set, the first data set being stored in memory associated with a data processing platform to be performed at a system external to the data processing platform. The method may also comprise generating a task identifier for the data processing task, and providing, in association with the task identifier, the first data set to an agent associated with the external system with an indication of the data processing task, the agent being arranged to cause performance of the task at the external system, to receive a second data set resulting from performance of the task, and to provide the second data set and associated metadata indicative of the transformation. The method may also comprise receiving the second data set and metadata from the agent associated with the external system and storing the second data set and associated metadata.
1. A method, performed by one or more processors, the method comprising:
receiving a build request associated with performing an external data processing task on a first dataset, the first dataset being stored in memory associated with a data processing platform to be performed at a system external to the data processing platform;
generating a task identifier for the data processing task; and
providing, in association with the task identifier, a build request comprising an indication of the first dataset and the data processing task to an agent associated with the external system, the agent being arranged to cause performance of the task at the external system, to receive a second dataset resulting from performance of the task, and to provide the second dataset and associated metadata indicative of the transformation to the data processing platform.
2. The method of claim 1 , further comprising:
receiving the second dataset and associated metadata from the agent associated with the external system; and
storing the second dataset and associated metadata in the data processing platform.
3. The method of claim 2 , further comprising storing the task identifier for the build request in a co-ordinator function that receives one or more build queries from the external system, or a service associated with the external system, for providing the build request to the agent.
4. The method of claim 3 , wherein the co-ordinator function stores multiple task identifiers for multiple other build requests for providing one or more build requests to the agent.
5. The method of claim 4 , wherein the co-ordinator function removes task identifiers, or flags task identifiers as used, responsive to receiving a completion message from the agent, such that subsequent build queries from the same external system do not cause re-provision of the same datasets.
6. The method of claim 5 , wherein the one or more build queries are polling queries received periodically from the external system, or a service associated with the external system.
7. The method of claim 6 , further comprising generating a token associated with the build request for enabling verification at the data processing platform that the received second dataset corresponds to the build request.
8. The method of claim 7 , wherein the metadata comprises one or more of: at least part of the code used on the external system to perform the data processing task; the version number of the code used on the external system to perform the data processing task; the date the code was last updated; and a hash of the code.
9. The method of claim 8 , further comprising transmitting an update query to the external system, and responsive thereto, receiving an indication of one or more updates made to code used on the external system to perform the data processing task.
10. The method of claim 9 , wherein the update query is transmitted periodically as a polling request.
11. The method of claim 10 , further comprising, responsive to receiving an indication of one or more updates made to the code on the external system, flagging the second dataset, and/or any other dataset generated as a result of performance of the data processing task, as being out of date.
12. The method of claim 11 , further comprising displaying a prompt on a user interface indicating the second and/or other datasets flagged as out of date.
13. The method of claim 12 , further comprising preventing further data processing tasks being performed on the second and/or other datasets flagged as out of date.
14. The method of claim 13 , further comprising automatically generating a new build request for performing the data processing task on the first dataset for generating a new dataset to replace the second and/or other datasets flagged as out of date.
15. The method of claim 14 , wherein the external data processing task comprises one task of a data processing pipeline comprising two or more tasks, and wherein the received second dataset is provided as input to a further task which performs a further data processing task to generate a third dataset.
16. A method, performed by one or more processors, the method comprising:
identifying, at a system external to a data processing platform, a build request derived from the data processing platform associated with performing a data processing task on a first dataset, the first dataset being stored in memory associated with the data processing platform; receiving, responsive to the identification, the build request, including an identification of the first dataset and the data processing task; receiving the first dataset from the data processing platform;
causing performance of the data processing task at the external system; receiving a second dataset resulting from performance of the task; and
sending the second dataset and associated metadata indicative of the transformation to the data processing platform.
17. The method of claim 16 , further comprising transmitting one or more build queries to the data processing platform for causing receiving of the build request,
wherein the one or more build queries are polling queries periodically transmitted to the data processing platform.
18. The method of claim 17 , further comprising receiving a token associated with the build request for transmitting to the data processing platform when requesting the first dataset and/or sending the second dataset such that the data processing platform may determine that they correspond to the build request.
19. The method of claim 18 , further comprising receiving an update query from the data processing platform, and responsive thereto, transmitting an indication of one or more updates made to code to perform the data processing task.
20. The method of claim 19 , further comprising, upon receiving the build request from the data processing platform, creating a directory for holding the first dataset and the second dataset and associated metadata, and removing the directory subsequent to the second dataset and associated metadata being sent to the data processing platform.