Data stream quality management for analytic environments
View Patent ↗According to one aspect of the present disclosure, a system and technique for data quality management is disclosed. The system includes a processor and an ingress quality specification (IQS) module executable by the processor in a runtime environment with a data stream analytic module. The IQS module is configured to: receive the data stream; analyze a subset of data of the data stream to determine if the subset of data meets a quality expectation of the analytic module; annotate the subset of data to indicate a quality status based on whether the subset of data meets the quality expectation of the analytic module; and output the data stream to the analytic module.
1. A system, comprising:
an analytic module configured to analyze a data stream output by an object and output an analysis of the object, the analytic module having a data quality expectation of data of the data stream;
a memory storing a plurality of ingress quality specification (IQS) modules each corresponding to a different quality characteristic and each associated with the analytic module; and
an interface configured to enable a selection of at least one IQS module from the plurality of IQS modules to deploy with the analytic module to:
receive the data stream from the object;
analyze a subset of data of the data stream to determine if the subset of data meets the quality expectation of the analytic module;
modify the subset of data by annotating the subset of data to indicate a quality status based on whether the subset of data meets the quality expectation of the analytic module; and
output the data stream to the analytic module; and
wherein the analytic module is configured to receive the data stream from the IQS module, identify data not meeting the quality expectation based on the annotations, and omit from its analysis of the object the data not meeting the quality expectation; and
wherein the selected IQS module is configured to determine whether a selected subset of data of the data stream includes a minimum quantity of data samples based on the quality expectation of the analytic module.
2. The system of claim 1 , wherein the selected IQS module is configured to:
apply a predicate to the subset of data.
3. The system of claim 2 , wherein the selected IQS module is configured to flag data of the subset passing the predicate.
4. The system of claim 3 , wherein the selected IQS module is configured to flag data of the subset failing the predicate.
5. The system of claim 1 , wherein the selected IQS module is configured to annotate a field of the data indicating the quality status.
6. The system of claim 1 , wherein the selected IQS module includes:
a selector module configured to select the subset of the data stream; and
a predicate module configured to apply a predicate to the selected subset of data to determine whether the selected subset of data meets the quality expectation of the analytic module.
7. The system of claim 6 , wherein the selector module is configured to select the subset of data based on a predetermined time period.