Service chain for complex configuration-driven data quality rationalization and data control
Techniques for data quality analysis may include the determination of data quality metrics using reconfigurable quality components. Access to the data may be based on the determined quality metrics. The configurable quality components may determine quality metrics for corresponding datasets. The quality components may be configured automatically based on quality configurations. The configuration of the quality components may be facilitated using a data orchestrator.
1. A computer-implemented method comprising:
receiving, by an orchestrator of a data processing pipeline, one or more quality configurations for one or more datasets, the one or more quality configurations defining one or more quality thresholds for the one or more datasets;
automatically configuring, by the orchestrator, one or more quality components based on the one or more quality configurations;
receiving, by the orchestrator from a data source, a first dataset of the one or more datasets, wherein the data source is associated with a first stage of the data processing pipeline;
determining, using the configured one or more quality components, a quality metric for the first dataset prior to advancing the first dataset to a second stage of the data processing pipeline;
determining whether the quality metric satisfies a threshold; and
upon determining that the quality metric satisfies the threshold, outputting the first dataset to a downstream processor, wherein the downstream processor is associated with the second stage of the data processing pipeline.
2. The computer-implemented method of claim 1 , further comprising:
receiving, by the orchestrator, a second dataset;
determining, using the configured one or more quality components, a second quality metric for the second dataset; and
upon determining that the second quality metric does not satisfy the threshold, generating a notification indicating one or more quality issues for the second dataset.
3. The computer-implemented method of claim 2 , further comprising providing the notification to a data owner to notify the data owner of the one or more quality issues for the second dataset.
4. The computer-implemented method of claim 2 , further comprising providing the notification to a data producer of the first dataset to notify the data producer of the one or more quality issues with the second dataset.
5. The computer-implemented method of claim 4 , wherein the notification is generated using lineage data that identifies at least one of:
an originator of the second dataset; or
one or more upstream processors having processed the second dataset.
6. The computer-implemented method of claim 1 , wherein the orchestrator is configured to reconfigure the one or more quality components based on one or more new quality configurations received by the orchestrator.
7. The computer-implemented method of claim 1 , wherein the configuring by the orchestrator comprises chaining two quality components of the one or more quality components together, wherein the quality metric is determined based on the two chained quality components.
8. The computer-implemented method of claim 1 , further comprising storing the quality metric in a data repository.
9. The computer-implemented method of claim 8 , further comprising querying the stored quality metric using an Application Programming Interface (API) configured to access the data repository.
10. The computer-implemented method of claim 1 , further comprising accessing the first dataset using a software development kit (SDK).
11. The computer-implemented method of claim 10 , wherein access to the first dataset is based on the determination that the quality metric satisfies the threshold.
12. The computer-implemented method of claim 1 , further comprising updating, using a machine learning algorithm, the one or more quality configurations, wherein the machine learning algorithm is trained based on the determination whether the quality metric satisfies the threshold.
13. The computer-implemented method of claim 1 , further comprising updating the one or more quality configurations based on a received update request.
14. A computing device comprising:
one or more processors; and
memory storing instructions that, when executed by the one or more processors, configure the computing device to:
automatically configure one or more quality components based on one or more quality configurations for one or more datasets received from a data source, the one or more quality configurations defining one or more quality thresholds for the one or more datasets, wherein the data source is associated with a first stage of a data processing pipeline of the one or more processors;
determine, using the configured one or more quality components, a quality metric for a first dataset of the one or more datasets prior to advancing the first dataset to a second stage of the data processing pipeline;
determine whether the quality metric satisfies a threshold;
based on a determination that the quality metric satisfies the threshold, output the first dataset to a downstream processor associated with the second stage of the data processing pipeline; and
based on a determination that the quality metric does not satisfy the threshold, generating a notification indicating one or more quality issues for the first dataset.
15. The computing device of claim 14 , wherein the quality metric is for one or more data records contained in the first dataset.
16. The computing device of claim 14 , wherein the first dataset comprises one or more data records and/or metadata.
17. The computing device of claim 14 , wherein the instructions, when executed by the one or more processors, configure the computing device to aggregate outputs of two quality components, of the one or more quality components, to determine the quality metric.
18. The computing device of claim 14 , wherein the downstream processor generates a report indicating the quality metric of the first dataset.
19. One or more non-transitory media storing instructions that, when executed, cause a computing device to:
receive, by an orchestrator of a data processing pipeline, one or more quality configurations for one or more datasets, the one or more quality configurations defining one or more quality thresholds for the one or more datasets;
automatically configure, by the orchestrator, one or more quality components based on the one or more quality configurations;
receive, by the orchestrator from a data source, a first dataset of the one or more datasets, wherein the data source is associated with a first stage of the data processing pipeline;
determine, using the configured one or more quality components, a quality metric for the first dataset prior to advancing the first dataset to a second stage of the data processing pipeline;
store the quality metric in a data repository;
determine whether the quality metric satisfies a threshold;
determine a query result based on a received query request for the stored quality metric, the query request using an Application Programming Interface (API) configured to access the data repository; and
based on a determination that the quality metric satisfies the threshold, output the first dataset to a downstream processor, wherein the downstream processor is associated with the second stage of the data processing pipeline.
20. The one or more non-transitory media of claim 19 , wherein the instructions, when executed, cause the computing device to provide access to the first dataset based on a request received via a software development kit (SDK).