Automated data observability system
An automated data observation system, platform, and corresponding methods and non-transitory computer readable medium include periodically obtaining metadata, automatically generating data quality metrics, obtaining predicted values for the data quality metrics, determining a first anomaly, generating a data pipeline graph, and estimating a root cause relating to the first anomaly by traversing the data pipeline graph. Obtaining predicted values for the data quality metrics includes executing a plurality of candidate machine learning models over time series data relating to the data quality metrics to generate candidate predicted values and selecting a machine learning model for a given data quality metric based on the candidate predicted values and measured values.
1 . An automated data observation system, comprising a processor and a memory storing instructions that when executed by the processor cause the processor to:
periodically obtain metadata relating to respective nodes of a plurality of nodes;
automatically generate data quality metrics for respective nodes of the plurality of nodes based on the metadata;
obtain predicted values for the data quality metrics including by:
executing a plurality of candidate machine learning models over time series data including measured values corresponding to respective ones of the data quality metrics to generate candidate predicted values for respective ones of the data quality metrics,
selecting a selected machine learning model from the plurality of candidate machine learning models for respective ones of the data quality metrics based on a comparison between at least some of the measured values and at least some of the predicted values, and
obtaining the predicted values based on the selected machine learning model for respective ones of the data quality metrics;
determine a first anomaly relating to a first one of the data quality metrics for a first node of the plurality of nodes including by comparing a first measured value at a first time relating to the first one of the data quality metrics and a first predicted value at the first time relating to the first one of the data quality metrics;
generate a data pipeline graph corresponding to the at least some of the plurality of nodes including the first node by parsing logs including information relating to usage of at least some of the plurality of nodes to identify relationships between at least some of the plurality of nodes, including creating a directional edge from a first table to a second table based on an identification of a pattern of inserts into the second table based on data selected from the first table, wherein the data pipeline graph includes information indicative of an importance of the directional edge based on the pattern of inserts;
estimate a root cause relating to the first anomaly by traversing the data pipeline graph from the first node;
determine a priority of the estimated root cause based on information indicative of the importance of directional edges between a second node associated with the root cause and the first node, wherein the priority is further determined based on an identity of users accessing the first node, second node, or nodes between the first node and second node and a frequency of users accessing the first node, second node, or nodes between the first node and second node; and
transmit, to a client device, information relating to the estimated root cause for display by the client device based on the priority,
wherein the nodes of the plurality of nodes are tables in a data warehouse system, the logs are query logs including queries executed by the data warehouse system.
2 . The system of claim 1 , wherein the plurality of nodes includes all tables in a data warehouse system, data quality metrics are generated for all of the plurality of nodes, and predicted values are obtained for all of the data quality metrics.
3 . The system of claim 1 , wherein the instructions to estimate the root cause includes comparing the information indicative of the importance to a threshold.
4 . The system of claim 1 , wherein the memory further comprises instructions that when executed by the processor cause the processor to:
detect that the first node has been deleted and that a second node has been created to replace the first node; and
in response to the detection, associate data quality metrics and time series data of the first node with the second node.
5 . The system of claim 1 , wherein the memory further comprises instructions that when executed by the processor cause the processor to:
detect that the first node has not been updated for a time period exceeding a time period threshold; and
based on the detection, suspend obtaining predicted values for data quality metrics associated with the first node.
6 . The system of claim 1 , wherein the data quality metrics include a last modification metric, a quantity of updates per time period metric, a volume of data per time period metric, and a node size metric.
7 . The system of claim 1 , wherein a subset of the candidate machine learning models of the plurality of candidate machine learning models are removed over time by eliminating candidate machine learning models that are measured to have a higher error over time.
8 . The system of claim 7 , wherein the plurality of candidate machine learning models includes a minimum of three candidate machine learning models.
9 . The system of claim 7 , wherein, after a reset time period has elapsed, candidate machine learning models are re-added to the plurality of candidate machine learning models.
10 . A method comprising:
periodically obtaining metadata relating to respective nodes of a plurality of nodes;
automatically generating data quality metrics for respective nodes of the plurality of nodes based on the metadata;
obtaining predicted values for the data quality metrics including by:
executing a plurality of candidate machine learning models over time series data including measured values corresponding to respective ones of the data quality metrics to generate candidate predicted values for respective ones of the data quality metrics,
selecting a selected machine learning model from the plurality of candidate machine learning models for respective ones of the data quality metrics based on a comparison between at least some of the measured values and at least some of the predicted values, and
obtaining the predicted values based on the selected machine learning model for respective ones of the data quality metrics;
determining a first anomaly relating to a first one of the data quality metrics for a first node of the plurality of nodes including by comparing a first measured value at a first time relating to the first one of the data quality metrics and a first predicted value at the first time relating to the first one of the data quality metrics;
generating a data pipeline graph corresponding to the at least some of the plurality of nodes including the first node by parsing logs including information relating to usage of at least some of the plurality of nodes to identify relationships between at least some of the plurality of nodes, including creating a directional edge from a first table to a second table based on an identification of a pattern of inserts into the second table based on data selected from the first table, wherein the data pipeline graph includes information indicative of an importance of the directional edge based on the pattern of inserts;
estimating a root cause relating to the first anomaly by traversing the data pipeline graph from the first node;
determining a priority of the estimated root cause based on information indicative of the importance of directional edges between a second node associated with the root cause and the first node, wherein the priority is further determined based on an identity of users accessing the first node, second node, or nodes between the first node and second node and a frequency of users accessing the first node, second node, or nodes between the first node and second node; and
transmitting information relating to the relating to the estimated root cause to a client device for display on the client device based on the priority,
wherein the nodes of the plurality of nodes are tables in a data warehouse system, the logs are query logs including queries executed by the data warehouse system.
11 . The method of claim 10 , further comprising:
generating a data pipeline graph corresponding to at least some of the plurality of nodes by parsing logs including information relating to usage of at least some of the plurality of nodes to identify relationships between at least some of the plurality of nodes; and
estimating a root cause relating to the first anomaly by traversing the data pipeline graph from the first node.
12 . The method of claim 11 , wherein candidate machine learning models are removed from the plurality of candidate machine learning models over time provided that the plurality of candidate machine learning models includes at least three candidate machine learning models, and wherein removed candidate machine learning models are periodically re-added to the plurality of candidate machine learning models.
13 . The method of claim 12 , wherein the data quality metrics include a last modification metric, a quantity of updates per time period metric, a volume of data per time period metric, and a node size metric, and obtaining predicted values for the data quality metrics relating to the first node is suspended based on a determination that measured values relating to the last modification metric indicates that the first node has not been modified for a time period exceeding a time period threshold.
14 . A non-transitory computer readable medium storing instructions that when executed by a processor cause the processor to:
periodically obtain metadata relating to respective nodes of a plurality of nodes;
automatically generate data quality metrics for respective nodes of the plurality of nodes based on the metadata;
obtain predicted values for the data quality metrics including by:
executing a plurality of candidate machine learning models over time series data including measured values corresponding to respective ones of the data quality metrics to generate candidate predicted values for respective ones of the data quality metrics,
selecting a selected machine learning model from the plurality of candidate machine learning models for respective ones of the data quality metrics based on a comparison between at least some of the measured values and at least some of the predicted values, and
obtaining the predicted values based on the selected machine learning model for respective ones of the data quality metrics;
determine a first anomaly relating to a first one of the data quality metrics for a first node of the plurality of nodes including by comparing a first measured value at a first time relating to the first one of the data quality metrics and a first predicted value at the first time relating to the first one of the data quality metrics;
generate a data pipeline graph corresponding to at least some of the plurality of nodes by parsing logs including information relating to usage of at least some of the plurality of nodes to identify relationships between at least some of the plurality of nodes, including creating a directional edge from a first table to a second table based on an identification of a pattern of inserts into the second table based on data selected from the first table, wherein the data pipeline graph includes information indicative of an importance of the directional edge based on the pattern of inserts;
estimate a root cause relating to the first anomaly by traversing the data pipeline graph from the first node;
determine a priority of the estimated root cause based on information indicative of the importance of directional edges between a second node associated with the root cause and the first node, wherein the priority is further determined based on an identity of users accessing the first node, second node, or nodes between the first node and second node and a frequency of users accessing the first node, second node, or nodes between the first node and second node; and
transmit, to a client device, information relating to the estimated root cause for display by the client device based on the priority,
wherein the nodes of the plurality of nodes are tables in a data warehouse system, the logs are query logs including queries executed by the data warehouse system.
15 . The non-transitory computer readable medium of claim 14 , wherein an importance of the estimated root cause is determined based on the information indicative of importance of the nodes and edges traversed when estimating the root cause.