Pipeline task verification for a data processing platform
A pipeline task verification method and system is disclosed, and may use one or more processors. The method may comprise providing a data processing pipeline specification, wherein the data processing pipeline specification defines a plurality of data elements of a data processing pipeline. The method may further comprise identifying from the data processing pipeline specification one or more tasks defining a relationship between a first data element and a second data element. The method may further comprise receiving for a given task one or more data processing elements intended to receive the first data element and to produce the second data element. The method may further comprise verifying that the received one or more data processing elements receive the first data element and produce the second data element according to the defined relationship.
1. A method of task validation in a software platform, wherein the method is performed using one or more processors, the method comprising:
providing a data processing pipeline specification, wherein the data processing pipeline specification defines a plurality of data elements of a data processing pipeline;
identifying from the data processing pipeline specification one or more tasks defining or having a defined relationship between a first data element and a second data element, wherein the one or more tasks are associated with metadata indicating an expected language or transformation;
receiving for a given task one or more data processing elements intended to receive the first data element and to produce the second data element; and
verifying that the received one or more data processing elements receive the first data element and produce the second data element according to the defined relationship.
2. The method of claim 1 , wherein the data processing pipeline specification defines pipeline dependencies of the plurality of data elements from which the one or more tasks may be identified.
3. The method of claim 2 , wherein the one or more tasks comprise metadata defining the relationship between the first data element and the second data element of the task.
4. The method of claim 3 , wherein the metadata defines expected criteria for the one or more data processing elements and wherein the verification is based on whether the received one or more data processing elements agree with the expected criteria.
5. The method of claim 4 , wherein the metadata comprises one or more of:
expected dependencies of the one or more tasks; and
expected row and/or column names.
6. The method of claim 5 , wherein the data processing pipeline specification is provided by means of a graphical user interface receiving user-defined placeholders representing one or more of data elements and tasks with a graphical indication of dependencies between data elements and/or tasks.
7. The method of claim 6 , wherein the one or more data elements and/or tasks are user selectable to open an interface therefor, for receiving code or one or more links to code for performing one or more operations.
8. The method of claim 7 , further comprising providing on the graphical user interface a graphical indication of the verification result.
9. The method of claim 8 , wherein the graphical indication of the verification result is positioned in or adjacent a placeholder associated with the verification operation.
10. The method of claim 9 , further comprising providing on the graphical user interface an indication of the reason for a verification failure.
11. The method of claim 10 , wherein the indication of the reason for a verification failure is made responsive to a user hovering over, or selecting a placeholder associated with the verification operation.
12. The method of claim 11 , further comprising providing an indication of the expected criteria.
13. The method of claim 6 , wherein at least some of the metadata is automatically inferred based on the user defined graphical identification of dependencies.
14. The method of claim 13 , further comprising controlling an electrical system based on the verification.
15. The method of claim 13 , wherein at least one of the data elements is a first data set representing sensed data from one or more sensors, wherein the data processing element produces a second data set from the first data set, and wherein a positive verification controls the electrical system to perform a predetermined operation.
16. A method of task validation in a software platform, wherein the method is performed using one or more processors, the method comprising:
providing a data processing pipeline specification, wherein the data processing pipeline specification defines a plurality of data elements of a data processing pipeline;
identifying from the data processing pipeline specification one or more tasks defining or having a defined relationship between a first data element and a second data element, wherein the one or more tasks are associated with metadata indicating expected row and/or column names;
receiving for a given task one or more data processing elements intended to receive the first data element and to produce the second data element; and
verifying that the received one or more data processing elements receive the first data element and produce the second data element according to the defined relationship.
17. A method of task validation in a software platform, wherein the method is performed using one or more processors, the method comprising:
providing a data processing pipeline specification, wherein the data processing pipeline specification defines a plurality of data elements of a data processing pipeline;
identifying from the data processing pipeline specification one or more tasks defining or having a defined relationship between a first data element and a second data element, wherein the one or more tasks are associated with metadata indicating expected dependencies of the one or more tasks;
receiving for a given task one or more data processing elements intended to receive the first data element and to produce the second data element;
verifying that the received one or more data processing elements receive the first data element and produce the second data element according to the defined relationship; and
displaying, on a graphical user interface, the data processing pipeline specification using placeholders, wherein the placeholders comprise one or more task placeholders for the one or more tasks.
18. The method of claim 16 , further comprising:
displaying, on a graphical user interface, the data processing pipeline specification using placeholders, wherein the placeholders comprise one or more task placeholders for the one or more tasks.
19. The method of claim 17 , wherein said verifying that the received one or more data processing elements receive the first data element and produce the second data element according to the defined relationship is performed in response to receiving the one or more data processing elements for the given task from the user.