Data visibility and quality management platform
Embodiments described herein comprise an advanced Software as a Service (SaaS) platform addressing challenges including trust, governance, quality, data accuracy, and transparency, ensuring full visibility of enterprise data warehouse data pipelines. These embodiments seamlessly integrate with cloud-based and legacy data platforms, enhancing visibility through monitoring data quality, tracking metadata changes, and generating exceptions for review. Data quality management allows for custom script creation to validate pipelines and business rules. Cloud-native connectors facilitate integration with one or more cloud-based services streamlining data delivery processes. Personalized notifications aid administrators in responsive action, while data profiling capabilities identify quality issues and provide distribution insights. Actionable data intelligence for privacy, security, and governance is enabled through data classification. Automation features encompass data quality checks, metadata monitoring, profiling, and classification, orchestrated by a flexible scheduler. The architecture leverages a scalable serverless design with independent microservices, dynamically scaling based on demand.
1 . A device, comprising:
a processor;
a memory communicatively coupled to the processor;
a data quality management logic when executed by the processor to:
select a target enterprise data warehouse (EDW) platform;
monitor query processing on the target EDW platform;
produce metrics on queries including query execution times, resource utilization, throughput, and error conditions;
analyze the produced metrics to identify peak demand hour data based on periods of highest aggregate resource utilization or query activity;
determine, based on the produced metrics, whether analysis will proceed using an automated analysis or a manual analysis;
scaling available EDW computing power up or down in response to the manual analysis;
generate, in response to the automated analysis, a plurality of suggested actions including compute-scaling recommendations and configuration changes derived from the thresholds or machine-learning outputs;
automatically select one of the suggested actions; and
dynamically apply the selected suggested action.
2 . The device of claim 1 , wherein the data quality management logic is further configured to send a notification in response to the dynamically applied selected suggested action.
3 . The device of claim 1 , wherein the suggested action is associated with computing power.
4 . The device of claim 3 , wherein the computing power is related to the EDW.
5 . The device of claim 4 , wherein the suggested action is to increase the overall available computing power of the EDW.
6 . The device of claim 1 , wherein the monitoring of the target EDW platform comprises at least monitoring the queries associated with the EDW.
7 . The device of claim 6 , wherein the monitoring further comprises determining a delay time associated with the queries.
8 . The device of claim 7 , wherein data quality management logic is further configured to produce a plurality of metrics based on the monitoring of the target EDW platform.
9 . The device of claim 8 , wherein the metrics are based on the delay time and the produced peak demand hour data.
10 . The device of claim 9 , wherein the analysis of the target EDW platform is at least based on the produced plurality of metrics.
11 . A method of dynamically monitoring enterprise data warehouse (EDW) platforms, comprising:
selecting, via a data quality management logic when executed by a processor, a target EDW platform;
monitoring query processing on the target EDW platform;
producing metrics on queries including query execution times, resource utilization, throughput, and error conditions;
analyzing the produced metrics to identify peak demand hour data based on periods of highest aggregate resource utilization or query activity;
determining, based on the produced metrics, whether analysis will proceed using an automated analysis or a manual analysis;
scaling available EDW computing power up or down in response to the manual analysis;
generating, in response to the automated analysis, a plurality of suggested actions including compute-scaling recommendations and configuration changes derived from the thresholds or machine-learning outputs;
automatically selecting one of the suggested actions; and
dynamically applying the selected suggested action.
12 . The method of claim 11 , wherein the data quality management logic is further configured to send a notification in response to the dynamically applied selected suggested action.
13 . The method of claim 11 , wherein the suggested action is associated with computing power.
14 . The method of claim 13 , wherein the computing power is related to the EDW.
15 . The method of claim 14 , wherein the suggested action is to increase the overall available computing power of the EDW.
16 . The method of claim 11 , wherein the monitoring of the target EDW platform comprises at least monitoring the queries associated with the EDW.
17 . The method of claim 16 , wherein the monitoring further comprises determining a delay time associated with the queries.
18 . The method of claim 17 , wherein data quality management logic is further configured to produce a plurality of metrics based on the monitoring of the target EDW platform.
19 . The method of claim 18 , wherein the metrics are based on the delay time and the produced peak demand hour data.
20 . The method of claim 19 , wherein the analysis of the EDW platform is at least based on the produced plurality of metrics.