Built-in data quality monitoring
Described herein are techniques for data quality monitoring in a network-based data system. A data metric function used to evaluate data quality can be stored, where the data metric function is defined as schema level object. The data metric function can be attached to a table associated with an account and is evaluated on data associates with the table to generate evaluation results. The evaluation results can be stored in an account-specific central database, from which access is provided to the evaluation results to a user for the account.
1 . A method comprising:
providing a base table within an account in network-based data system;
storing a data metric function to evaluate data quality in the network-based data system, the data metric function being defined as a schema level object;
providing a dynamic table within the account storing incremental change row information of the base table, the dynamic table being a nested child object of the base table;
attaching the data metric function to the base table associated in the network-based data system;
refreshing the dynamic table to load new changed row information of the base table since a last refresh of the dynamic table;
evaluating, by at least one hardware processor, the data metric function against the refreshed dynamic table to generate evaluation results, an amount of data in the dynamic table being less than an amount of data in the base table;
storing the evaluation results in an account-specific central location in the network-based data system; and
providing access to the evaluation results in the account-specific central location to a user of the account.
2 . The method of claim 1 , further comprising:
setting a schedule for data metric evaluation for the table, wherein evaluating the data metric function is performed based on the schedule.
3 . The method of claim 2 , further comprising:
executing the data metric function at a later time than a scheduled time using a stored version of the table corresponding to the scheduled time.
4 . The method of claim 1 , wherein the refreshing of the dynamic table is performed based on a scheduled time associated with the data metric function.
5 . The method of claim 1 , wherein the data metric function is a first data metric function, the method further comprising:
attaching a second data metric function to the table.
6 . The method of claim 1 , wherein the table is a first table and the account is a first account, and wherein the data metric function is attached to a second table associated with a second account in the network-based data system.
7 . A machine-storage medium embodying instructions that, when executed by a machine, cause the machine to perform operations comprising:
providing a base table within an account in network-based data system;
storing a data metric function to evaluate data quality in the network-based data system, the data metric function being defined as a schema level object;
providing a dynamic table within the account storing incremental change row information of the base table, the dynamic table being a nested child object of the base table;
attaching the data metric function to the base table associated in the network-based data system;
refreshing the dynamic table to load new changed row information of the base table since a last refresh of the dynamic table;
evaluating the data metric function against the refreshed dynamic table to generate evaluation results, an amount of data in the dynamic table being less than an amount of data in the base table;
storing the evaluation results in an account-specific central location in the network-based data system; and
providing access to the evaluation results in the account-specific central location to a user of the account.
8 . The machine-storage medium of claim 7 , further comprising:
setting a schedule for data metric evaluation for the table, wherein evaluating the data metric function is performed based on the schedule.
9 . The machine-storage medium of claim 8 , further comprising:
executing the data metric function at a later time than a scheduled time using a stored version of the table corresponding to the scheduled time.
10 . The machine-storage medium of claim 7 , wherein the refreshing of the dynamic table is performed based on a scheduled time associated with the data metric function.
11 . The machine-storage medium of claim 7 , wherein the data metric function is a first data metric function, the machine-storage medium further comprising:
attaching a second data metric function to the table.
12 . The machine-storage medium of claim 7 , wherein the table is a first table and the account is a first account, and wherein the data metric function is attached to a second table associated with a second account in the network-based data system.
13 . A system comprising:
at least one hardware processor; and
at least one memory storing instructions that, when executed by the at least one hardware processor, cause the at least one hardware processor to perform operations comprising:
providing a base table within an account in network-based data system;
storing a data metric function to evaluate data quality in the network-based data system, the data metric function being defined as a schema level object;
providing a dynamic table within the account storing incremental change row information of the base table, the dynamic table being a nested child object of the base table;
attaching the data metric function to the base table associated in the network-based data system;
refreshing the dynamic table to load new changed row information of the base table since a last refresh of the dynamic table;
evaluating the data metric function against the refreshed dynamic table to generate evaluation results, an amount of data in the dynamic table being less than an amount of data in the base table;
storing the evaluation results in an account-specific central location in the network-based data system; and
providing access to the evaluation results in the account-specific central location to a user of the account.
14 . The system of claim 13 , the operations further comprising:
setting a schedule for data metric evaluation for the table, wherein evaluating the data metric function is performed based on the schedule.
15 . The system of claim 14 , the operations further comprising:
executing the data metric function at a later time than a scheduled time using a stored version of the table corresponding to the scheduled time.
16 . The system of claim 15 , the operations further comprising:
wherein the refreshing of the dynamic table is performed based on a scheduled time associated with the data metric function.
17 . The system of claim 13 , wherein the data metric function is a first data metric function, the operations further comprising:
attaching a second data metric function to the table.
18 . The system of claim 13 , wherein the table is a first table and the account is a first account, and wherein the data metric function is attached to a second table associated with a second account in the network-based data system.