Pipeline task verification for a data processing platform
A pipeline task verification method and system is disclosed, and may use one or more processors. The method may comprise providing a data processing pipeline specification, wherein the data processing pipeline specification defines a plurality of data elements of a data processing pipeline. The method may further comprise identifying from the data processing pipeline specification one or more tasks defining a relationship between a first data element and a second data element. The method may further comprise receiving for a given task one or more data processing elements intended to receive the first data element and to produce the second data element. The method may further comprise verifying that the received one or more data processing elements receive the first data element and produce the second data element according to the defined relationship.
1. A computerized method, performed by a computing system having one or more hardware computer processors and one or more non-transitory computer readable storage device storing software instructions executable by the computing system to perform the computerized method comprising:
providing a data processing pipeline specification, wherein the data processing pipeline specification defines a plurality of data elements of a data processing pipeline;
identifying from the data processing pipeline specification one or more tasks defining or having a defined relationship between a first data element and a second data element, wherein the one or more tasks are associated with metadata indicating an expected language or transformation, expected dependencies of the one or more tasks, or expected row and/or column names;
determining, for a given task, one or more data processing elements intended to receive the first data element and to produce the second data element; and
displaying, on a graphical user interface, the data processing pipeline specification including a task placeholder associated with a first task.
2. The computerized method of claim 1 , wherein the data processing pipeline specification defines pipeline dependencies of the plurality of data elements from which the one or more tasks may be identified.
3. The computerized method of claim 2 , wherein the metadata defines a relationship between the first data element and the second data element.
4. The computerized method of claim 3 , wherein the metadata defines expected criteria for the one or more data processing elements.
5. The computerized method of claim 1 , wherein the first data element is a first data set representing sensed data from one or more sensors, wherein the one or more data processing elements produce the second data element comprising a second data set from the first data set.
6. The computerized method of claim 1 , further comprising:
receiving user input of the data processing elements intended to receive the first data element and to produce the second data element according to a defined relationship; and
accessing the first and second data elements from a data source.
7. The computerized method of claim 6 , wherein the task placeholder is user selectable to open an interface therefor, for receiving code or one or more links to code for performing one or more operations.
8. A computing system comprising:
a hardware computer processor;
a non-transitory computer-readable medium having software instructions stored thereon, the software instructions executable by the hardware computer processor to cause the computing system to perform operations comprising:
providing a data processing pipeline specification, wherein the data processing pipeline specification defines a plurality of data elements of a data processing pipeline;
identifying from the data processing pipeline specification one or more tasks defining or having a defined relationship between a first data element and a second data element, wherein the one or more tasks are associated with metadata indicating an expected language or transformation, expected dependencies of the one or more tasks, or expected row and/or column names;
determining, for a given task, one or more data processing elements intended to receive the first data element and to produce the second data element; and
displaying, on a graphical user interface, the data processing pipeline specification including a task placeholder associated with a first task.
9. The computing system of claim 8 , wherein the data processing pipeline specification defines pipeline dependencies of the plurality of data elements from which the one or more tasks may be identified.
10. The computing system of claim 9 , wherein the metadata defines a relationship between the first data element and the second data element.
11. The computing system of claim 10 , wherein the metadata defines expected criteria for the one or more data processing elements.
12. The computing system of claim 8 , wherein the first data element is a first data set representing sensed data from one or more sensors, wherein the one or more data processing elements produce the second data element comprising a second data set from the first data set.
13. The computing system of claim 8 , further comprising:
receiving user input of the data processing elements intended to receive the first data element and to produce the second data element according to a defined relationship; and
accessing the first and second data elements from a data source.
14. The computing system of claim 8 , wherein the task placeholder is user selectable to open an interface therefor, for receiving code or one or more links to code for performing one or more operations.