Functional metadata system and method for data quality inspection
View Patent ↗The present disclosure provides a functional metadata method for data quality inspection, which includes steps as follows. Based on historical data, the functional metadata is established, where the functional metadata includes an acceptable pattern, a data distribution and a data definition; according to the functional metadata, at least one of a detection of the acceptable pattern, a detection of the data distribution and a detection of the data definition is performed on a data content, so as to obtain a detection result.
1 . A functional metadata system for a data quality inspection, comprising:
a storage device configured to store a data content; and
a processor electrically connected to the storage device, the processor configured to perform the data quality inspection on the data content to obtain a detection result, and the data quality inspection selected from a group consisting of a detection of an acceptable pattern, a detection of a data distribution and a detection of a data definition, wherein the data content comprises a string content, wherein the processor detects whether a format of the string content matches the acceptable pattern, wherein when the format of the string content does not match the acceptable pattern, the processor uses an artificial intelligence model to determine whether the format of the string content conforms to a default standard answer of the data definition, as a basis of confirming a correctness of the string content.
2 . The functional metadata system for the data quality inspection of claim 1 , wherein the data content comprises a numeric content, and the processor performs the data distribution based on a historical data so as to define a default reasonable range for the numeric content.
3 . The functional metadata system for the data quality inspection of claim 1 , wherein the data content comprises the string content, and the processor determines whether the string content matches a predefined attribute in the data definition.
4 . The functional metadata system for the data quality inspection of claim 1 , wherein the data content comprises a numeric content, and the processor determines whether the numeric content is abnormal based on a default common sense standard.
5 . A functional metadata method for a data quality inspection, comprising steps of:
creating a functional metadata based on a historical data, the functional metadata comprises an acceptable pattern, a data distribution and a data definition; and
performing at least one of a detection of the acceptable pattern, a detection of the data distribution and a detection of the data definition on a data content according to the functional metadata, so as to obtain a detection result, wherein the data content comprises a string content, where the step of performing the detection of the acceptable pattern on the data content comprises:
analyzing a format of the string content through an artificial intelligence model, and then detecting whether the format of the string content matches the acceptable pattern; and
when the format of the string content does not match the acceptable pattern, using the artificial intelligence model to determine whether the format of the string content conforms to a default standard answer of the data definition, as a basis of confirming a correctness of the string content.
6 . The functional metadata method for the data quality inspection of claim 5 , wherein the data content comprises a numeric content, and the step of performing the detection of the data distribution on the data content comprises:
performing a data fitting on the historical data in advance to obtain a probability distribution of the historical data so as to analyze whether the numeric content is an abnormal value.
7 . The functional metadata method for the data quality inspection of claim 5 , wherein the data content comprises a numeric content, and the step of performing the detection of the data definition on the data content comprises:
defining a predetermined numerical boundary of a default category based on a default common sense standard; and
when the numeric content belongs to the default category, determining whether the numeric content exceeds the predetermined numerical boundary.