Service chain for complex configuration-driven data quality rationalization and data control
Techniques for data quality analysis may include the determination of data quality metrics using reconfigurable quality components. Access to the data may be based on the determined quality metrics. The configurable quality components may determine quality metrics for corresponding datasets. The quality components may be configured automatically based on quality configurations. The configuration of the quality components may be facilitated using a data orchestrator.
1 . A computer-implemented method comprising:
automatically configuring, by an orchestrator of a data processing pipeline, one or more quality components based on one or more quality configurations for one or more datasets;
determining, using the configured one or more quality components, a quality metric for a first dataset of the one or more datasets associated with a first stage of the data processing pipeline;
determining whether the quality metric satisfies one or more parameters of the one or more quality configurations; and
upon determining that the quality metric satisfies the one or more parameters, outputting the first dataset to a downstream processor, wherein the downstream processor is associated with a second stage of the data processing pipeline.
2 . The computer-implemented method of claim 1 , wherein the quality metric for the first dataset is determined prior to advancing the first dataset to the second stage of the data processing pipeline.
3 . The computer-implemented method of claim 1 , wherein the one or more parameters comprise one or more quality thresholds for the one or more datasets.
4 . The computer-implemented method of claim 1 , further comprising receiving, by the orchestrator from a data source, the first dataset, wherein the data source is associated with the first stage of the data processing pipeline.
5 . The computer-implemented method of claim 1 , further comprising:
receiving, by the orchestrator, a second dataset;
determining, using the configured one or more quality components, a second quality metric for the second dataset; and
upon determining that the second quality metric does not satisfy the one or more parameters, generating a notification indicating one or more quality issues for the second dataset.
6 . The computer-implemented method of claim 5 , further comprising providing the notification to a data owner to notify the data owner of the one or more quality issues for the second dataset.
7 . The computer-implemented method of claim 5 , further comprising providing the notification to a data producer of the first dataset to notify the data producer of the one or more quality issues with the second dataset.
8 . The computer-implemented method of claim 7 , wherein the notification is generated using lineage data that identifies at least one of:
an originator of the second dataset; or
one or more upstream processors having processed the second dataset.
9 . The computer-implemented method of claim 1 , wherein the orchestrator is configured to reconfigure the one or more quality components based on one or more new quality configurations received by the orchestrator.
10 . The computer-implemented method of claim 1 , wherein the configuring by the orchestrator comprises chaining two quality components of the one or more quality components together, wherein the quality metric is determined based on the two chained quality components.
11 . The computer-implemented method of claim 1 , further comprising storing the quality metric in a data repository.
12 . The computer-implemented method of claim 11 , further comprising querying the stored quality metric using an Application Programming Interface (API) configured to access the data repository.
13 . The computer-implemented method of claim 1 , further comprising accessing the first dataset using a software development kit (SDK) based on the quality metric and the one or more parameters.
14 . The computer-implemented method of claim 1 , further comprising updating, using a machine learning algorithm, the one or more quality configurations, wherein the machine learning algorithm is trained based on the determination whether the quality metric satisfies the one or more parameters.
15 . The computer-implemented method of claim 1 , further comprising updating the one or more quality configurations based on a received update request.
16 . A computing device comprising:
one or more processors; and
memory storing instructions that, when executed by the one or more processors, configure the computing device to:
automatically configure one or more quality components based on one or more quality configurations for one or more datasets received from a data source;
determine, using the configured one or more quality components, a quality metric for a first dataset of the one or more datasets associated with a first stage of a data processing pipeline prior to advancing the first dataset to a second stage of the data processing pipeline;
determine whether the quality metric satisfies one or more parameters of the one or more quality configurations;
based on a determination that the quality metric satisfies the one or more parameters, output the first dataset to a downstream processor associated with the second stage of the data processing pipeline; and
based on a determination that the quality metric does not satisfy the one or more parameters, generating a notification indicating one or more quality issues for the first dataset.
17 . The computing device of claim 16 , wherein the quality metric is for one or more data records and/or metadata contained in the first dataset.
18 . The computing device of claim 16 , wherein the instructions, when executed by the one or more processors, configure the computing device to aggregate outputs of two quality components, of the one or more quality components, to determine the quality metric.
19 . The computing device of claim 16 , wherein the downstream processor is adapted to generate a report indicating the quality metric of the first dataset.
20 . One or more non-transitory media storing instructions that, when executed, cause a computing device to:
automatically configure, by an orchestrator of a data processing pipeline, one or more quality components based on one or more quality configurations for one or more datasets;
determine, using the configured one or more quality components, a quality metric for a first dataset of the one or more datasets associated with a first stage of the data processing pipeline; and
output, based on the quality metric and one or more parameters of the one or more quality configurations, the first dataset to a downstream processor, wherein the downstream processor is associated with a second stage of the data processing pipeline.