System for continuous monitoring of data quality in a dynamic feed environment
A system for providing continuous monitoring of data quality in a dynamic feed environment is disclosed. In particular, the system utilizes a feed inspection tool to detect anomalies in data gathering detected from feed metadata and anomalies in data measurement detected based on file contents. In order to do so, the feed inspection tool may aggregate, for a plurality of aggregation intervals, data feeds and associated metadata feeds. Once the data feeds and metadata feeds are aggregated, the feed inspection tool may generate, for a baseline model feed, baseline statistical models by utilizing historical data of the aggregated feeds in sliding windows of different lengths. The feed inspection tool may then identify, for a plurality of monitoring time delays, data outliers by comparing the aggregated feeds with the baseline model feed. A data quality feed based on the data outliers identified may then be generated and published.
1. A system comprising:
a memory that stores instructions; and
a processor that executes the instructions to perform operations, the operations comprising:
passively monitoring, by utilizing a feed inspection tool of a computing device, data feeds and metadata feeds associated with the data feeds;
aggregating, for a plurality of aggregation intervals, the data feeds and the metadata feeds by combining the data feeds and the metadata feeds into an aggregation feed;
generating, for a baseline model feed and by utilizing model parameters provided by the feed inspection tool, baseline models at multiple scales, wherein the baseline models are generated by utilizing the feed inspection tool of the computing device;
identifying, for a plurality of monitoring time delays, data outliers by comparing the aggregation feed for a current aggregation interval to the baseline model feed; and
generating a data quality feed based on the data outliers identified, wherein a configurable subset of entire files are sampled by utilizing a longitudinal sample that is selected when a full feed rate for delivering the entire files is greater than a number of files a subscriber of the data feeds and the data quality feed is capable of handling, wherein a set of random files across the data feeds are also sampled when generating the data quality feed so as to capture correlations among the data feeds that are not capturable using the longitudinal sample.
2. The system of claim 1 , wherein the operations further comprise deriving group-by variables and quantitative variables from the data feeds and the metadata feeds.
3. The system of claim 2 , wherein the operations further comprise computing a statistical summary for the quantitative variables for each combination of the group-by variables.
4. The system of claim 1 , wherein the operations further comprise receiving the data feeds and the metadata feeds from a data feed management system.
5. The system of claim 1 , wherein generating the data quality feed further comprises generating the data quality feed based on sampling the configurable subset of the entire files in the data feeds instead of sampling all the entire files in the data feeds.
6. The system of claim 1 , wherein the operations further comprise publishing the data quality feed so that the data quality feed is accessible to the subscriber of the data quality feed.
7. The system of claim 1 , wherein the operations further comprise subscribing to the data feeds and the metadata feeds.
8. The system of claim 1 , wherein generating of the baseline models at the multiple scales further comprises generating the baseline models by utilizing historical data associated with the aggregation feed in sliding windows of different lengths.
9. The system of claim 1 , wherein the operations further comprise facilitating access to the data quality feed for additional subscribers.
10. The system of claim 1 , wherein the operations further comprise generating a report or graph that visually identifies the data outliers.
11. The system of claim 1 , wherein the operations further comprise receiving a request to access a logical alert feed of the data quality feed, wherein the logical alert feed includes alerts computed for a specified time delay.
12. The system of claim 1 , wherein the operations further comprise detecting an error in processes associated with the data feeds.
13. The system of claim 1 , wherein the operations further comprise generating a time series plot that includes the data outliers.
14. A method comprising:
monitoring, by utilizing a feed inspection tool of a computing device, data feeds and metadata feeds associated with the data feeds;
aggregating, for a plurality of aggregation intervals, the data feeds and the metadata feeds by combining the data feeds and the metadata feeds into an aggregation feed;
generating, for a baseline model feed and by utilizing model parameters provided by the feed inspection tool, baseline models at multiple scales, wherein the baseline models are generated by utilizing the feed inspection tool of the computing device;
determining, for a plurality of monitoring time delays, data outliers by comparing the aggregation feed for a current aggregation interval to the baseline model feed; and
creating, by utilizing instructions from a memory that are executed by a processor, a data quality feed based on the data outliers identified, wherein a configurable subset of entire files are sampled by utilizing a longitudinal sample that is selected when a full feed rate for delivering the entire files is greater than a number of files a subscriber of the data feeds and the data quality feed is capable of handling, wherein a set of random files across the data feeds are also sampled when generating the data quality feed so as to capture correlations among the data feeds that are not capturable using the longitudinal sample.
15. The method of claim 14 , further comprising detecting a change in a distribution for records in the data feeds by analyzing contents of the data feeds.
16. The method of claim 14 , further comprising outputting an alert based on the data outliers.
17. The method of claim 14 , further comprising generating a report or graph that visually identifies the data outliers.
18. The method of claim 14 , further comprising subscribing to an unmatched data feed, wherein the unmatched data feed does not match any logical feeds defined by the subscriber.
19. The method of claim 18 , further comprising labeling unmatched files of the unmatched data feed to a data feed of the data feeds that shares a similarity to the unmatched files.
20. A computer-readable device comprising instructions, which when executed by a processor, cause the processor to perform operations comprising:
analyzing, by utilizing a feed inspection tool of a computing device, data feeds and metadata feeds associated with the data feeds;
aggregating, for a plurality of aggregation intervals, the data feeds and the metadata feeds by combining the data feeds and the metadata feeds into an aggregation feed;
generating, for a baseline model feed and by utilizing model parameters provided by the feed inspection tool, baseline models at multiple scales, wherein the baseline models are generated by utilizing the feed inspection tool of the computing device;
determining, for a plurality of monitoring time delays, data outliers by comparing the aggregation feed for a current aggregation interval to the baseline model feed; and
generating a data quality feed based on the data outliers identified, wherein a configurable subset of entire files are sampled by utilizing a longitudinal sample that is selected when a full feed rate for delivering the entire files is greater than a number of files a subscriber of the data feeds and the data quality feed is capable of handling, wherein a set of random files across the data feeds are also sampled when generating the data quality feed so as to capture correlations among the data feeds that are not capturable using the longitudinal sample.