Pipeline task verification for a data processing platform
A pipeline task verification method and system is disclosed, and may use one or more processors. The method may comprise providing a data processing pipeline specification, wherein the data processing pipeline specification defines a plurality of data elements of a data processing pipeline. The method may further comprise identifying from the data processing pipeline specification one or more tasks defining a relationship between a first data element and a second data element. The method may further comprise receiving for a given task one or more data processing elements intended to receive the first data element and to produce the second data element. The method may further comprise verifying that the received one or more data processing elements receive the first data element and produce the second data element according to the defined relationship.
1. A computerized method, performed by a computing system having one or more hardware computer processors and one or more non-transitory computer readable storage device storing software instructions executable by the computing system to perform the computerized method comprising:
accessing a data processing pipeline specification, wherein the data processing pipeline specification defines a plurality of data elements of a data processing pipeline;
identifying from the data processing pipeline specification a task indicating a defined relationship between a first data element and a second data element, wherein the task is associated with metadata indicating an expected language or transformation, an expected dependency, or an expected row and/or column name;
determining, for the task, one or more data processing elements intended to receive the first data element and to produce the second data element; and
verifying that the one or more data processing elements receive the first data element and produce the second data element according to the defined relationship.
2. The computerized method of claim 1 , wherein the data processing pipeline specification defines pipeline dependencies of the plurality of data elements from which the task may be identified.
3. The computerized method of claim 2 , wherein the metadata defines a relationship between the first data element and the second data element.
4. The computerized method of claim 3 , wherein the metadata defines expected criteria for the one or more data processing elements.
5. The computerized method of claim 1 , wherein the first data element is a first data set representing sensed data from one or more sensors, wherein the one or more data processing elements produce the second data element comprising a second data set from the first data set.
6. The computerized method of claim 1 , further comprising:
receiving user input of the one or more data processing elements intended to receive the first data element and to produce the second data element according to the defined relationship; and
accessing the first and second data elements from a data source.
7. The computerized method of claim 1 , wherein the expected dependency is associated with the task or one more of the first and second data elements.
8. The computerized method of claim 1 , wherein the verifying comprises analyzing the one or more data processing elements with reference to the metadata.
9. A computing system comprising:
a hardware computer processor;
a non-transitory computer-readable medium having software instructions stored thereon, the software instructions executable by the hardware computer processor to cause the computing system to perform operations comprising:
accessing a data processing pipeline specification, wherein the data processing pipeline specification defines a plurality of data elements of a data processing pipeline;
identifying from the data processing pipeline specification a task indicating a defined relationship between a first data element and a second data element, wherein the task is associated with metadata indicating an expected language or transformation, an expected dependency, or an expected row and/or column name;
determining, for the task, one or more data processing elements intended to receive the first data element and to produce the second data element; and
verifying that the one or more data processing elements receive the first data element and produce the second data element according to the defined relationship.
10. The computing system of claim 9 , wherein the data processing pipeline specification defines pipeline dependencies of the plurality of data elements from which the task may be identified.
11. The computing system of claim 10 , wherein the metadata defines a relationship between the first data element and the second data element.
12. The computing system of claim 11 , wherein the metadata defines expected criteria for the one or more data processing elements.
13. The computing system of claim 11 , wherein the first data element is a first data set representing sensed data from one or more sensors, wherein the one or more data processing elements produce the second data element comprising a second data set from the first data set.
14. The computing system of claim 11 , further comprising:
receiving user input of the one or more data processing elements intended to receive the first data element and to produce the second data element according to the defined relationship; and
accessing the first and second data elements from a data source.
15. The computing system of claim 9 , wherein the expected dependency is associated with the task or one more of the first and second data elements.
16. The computing system of claim 9 , wherein the verifying comprises analyzing the one or more data processing elements with reference to the metadata.