End to end validation of data transformation accuracy
Data is validated as it travels through the different nodes of a data pipeline. Instead of having to wait to validate the data when the data reaches an end of the data pipeline, each node in the pipeline may validate the data. Different methods may be used to validate the data. For example, each node may determine metadata about the received data and/or the transformed data. This metadata may be used to determine if the node is receiving the same amount of data as it usually receives, whether the data is in a same format, and the like. A timing of the data through one or more of the nodes may also be used in determining when the data is valid. When a problem is detected at any of the nodes in the pipeline, a report may be sent to one or more users.
1. A method for validating data at a node in a data pipeline that includes a plurality of nodes, comprising:
receiving data at the node;
determining metadata for the received data, the metadata including characteristics about the received data;
comparing the metadata to previously determined metadata about previous data received at the node to determine whether the received data is expected;
determining, based on a transformed version of the received data at the node, transformation metadata, the transformation metadata being characteristics about the transformed version of the received data at the node;
comparing the transformation metadata to previously determined transformation metadata about data previously transformed by the node to determine if the transformed version of the received data is valid;
and
providing a report in a case where the transformed version of the received data is invalid.
2. The method of claim 1 , further comprising transforming the received data at the node to the transformed version of the received data.
3. The method of claim 1 , wherein determining the transformation metadata includes determining the characteristics about the transformed version of the received data.
4. The method of claim 1 , wherein the received data is received from a trusted data source.
5. The method of claim 1 , wherein determining the metadata further comprises determining at least one of: a number of rows of the data, a number of columns of the data, or a memory size of the received data.
6. The method of claim 1 , further comprising determining a timing of the data through the node.
7. The method of claim 1 , further comprising storing the transformation metadata.
8. The method of claim 1 , further comprising providing the report to include a graphical representation of each node in the data pipeline and a graphical representation of the data pipeline and including a graphical indication for each of the graphical representations that represent a validity of the associated node or graphical pipeline.
9. A computer-readable storage medium storing computer-executable instructions validating data at a node in a data pipeline, comprising:
receiving data at the node;
determining metadata for the received data, the metadata including characteristics about the received data;
comparing the metadata to previously determined metadata about previous data received at the node to determine whether the received data is expected;
determining, based on a transformed version of the received data at the node, transformation metadata, the transformation metadata being characteristics about the transformed version of the received data at the node;
comparing the transformation metadata to previously determined transformation metadata about data previously transformed by the node to determine if the transformed version of the received data is valid;
providing a report in a case where the transformed version of the received data is invalid.
10. The computer-readable storage medium of claim 9 , wherein the metadata is determined from the received data by the node.
11. The computer-readable storage medium of claim 9 , further comprising determining whether a timing of data through the node is expected.
12. The computer-readable storage medium of claim 9 , wherein the received data is received from a trusted data source.
13. The computer-readable storage medium of claim 9 , wherein the determining of the metadata data further comprises determining at least one of: a size of the data; a number of rows of the data; and a number of columns of the data.
14. The computer-readable storage medium of claim 9 , further comprising determining a timing of the data through the node to determine if the transformed version of the received data is valid.
15. The computer-readable storage medium of claim 9 , further comprising storing the transformation metadata.
16. The computer-readable storage medium of claim 9 , further comprising providing the report to include a graphical representation of each node in the data pipeline.
17. A system for validating data at a node in a data pipeline, comprising:
a processor and memory;
an operating environment executing using the processor; and
a data validator that is configured to perform actions comprising:
receiving data at the node;
determining metadata for the received data, the metadata including characteristics about the received data;
comparing the metadata to previously determined metadata about previous data received at the node to determine whether the received data is expected;
determining, based on a transformed version of the received data at the node, transformation metadata, the transformation metadata being characteristics about the transformed version of the received data at the node;
comparing the transformation metadata to previously determined transformation metadata about data previously transformed by the node to determine if the transformed version of the received data is valid;
and
providing a report in a case where the transformed version of the received data is invalid.
18. The system of claim 17 , wherein determining the metadata further comprises determining at least one of: a size of the data; a number of rows of the data; and a number of columns of the received data.
19. The system of claim 17 , further comprising determining a timing of the data through the node and using the timing to determine if the transformed version of the received data is valid.