Smart data quality protocols

1. A system for smart generation of data warehouse quality protocols, the system comprising:

a data source comprising a set of source attributes having a first data structure;

a data warehouse comprising target data obtained from one or more data sources, the target data comprising a set of target attributes, the target attributes having a second data structure and comprising a transformation of the source attributes;

a processor comprising an artificial intelligence (“AI”) model and configured to:

generate a source vector from each source attribute in the set of source attributes;

generate a target vector from each target attribute in the set of target attributes; and

determine a probability of correspondence between the source vector and each of the target vectors for input to the AI model;

using the AI model:

determine whether one of the target attributes is a transformation of a source attribute based at least in part on a probability of correspondence between the source vector generated from the source attribute and one of the target vector generated from the target attribute;

output a data quality metric for the source attribute based at least in part on a first parameter of the transformation;

output a data quality metric for the target attribute based at least in part on a second parameter of the transformation; and

validate the source attribute and the target attribute using the data quality metrics, wherein the data quality metrics comprise a ratio of data to errors and a number of empty values.

2. The system of claim 1 , wherein the processor is further configured to:

determine a cardinality for an attribute;

when the cardinality of the attribute meets a predetermined threshold, use an n-gram algorithm to generate a vector for the attribute; and

when the cardinality of the attribute does not meet the predetermined threshold, use a direct word comparison algorithm to generate a vector for the attribute.

3. The system of claim 1 , wherein the processor is further configured to:

determine a term frequency for an attribute;

determine an inverse document frequency for an attribute; and

determine an attribute vector based at least in part on the term frequency and the inverse document frequency.

4. The system of claim 1 , further comprising determining whether a source attribute is used or unused, the determination based on a relationship between the target vector and the source vector, wherein a source attribute is unused when a probability of correspondence between the source vector generated from the source attribute and each of the target vectors is below a predetermined threshold.

5. The system of claim 1 further comprising a user interface, the user interface configured to receive input of a source attribute data set and a target attribute data set.

6. The system of claim 5 , wherein:

the AI model outputs the data quality metric based on one or more predefined rules for an attribute classification; and

the user interface is further configured to receive input modifying a predefined rule.

7. One or more non-transitory computer-readable media storing computer-executable instructions which, when executed by a processor on a computer system, perform a method for artificial intelligence (“AI”) based generation of data warehouse quality protocols, the method comprising:

generating a source vector from each source attribute in a set of source attributes, the set of source attributes obtained from one or more data sources and comprising a first data structure;

generating a target vector from each target attribute in a set of target attributes, the target attributes comprising target data obtained from one or more data sources and integrated into a data warehouse, and comprising a second data structure;

determining a probability of correspondence between the source vector and each of the target vectors;

identifying a transformation from the first data structure to the second data structure based at least in part on a probability of correspondence between the source vector generated from the source attribute and one of the target vectors, and classifying the source attribute;

generating a data quality metric for the source attribute based at least in part on the source attribute classification;

identifying a transformation from the first data structure to the second data structure based at least in part on a probability of correspondence between the source vector and the target vector, and classifying the target attribute;

generating a data quality metric for the target attribute based at least in part on the target attribute classification; and

validating the source attribute and the target attribute using the data quality metrics, wherein the data quality metrics comprise a ratio of data to errors and a number of empty values.

8. The media of claim 7 , wherein classifying an attribute further comprises:

determining a term frequency for an attribute;

determining an inverse document frequency for an attribute; and

determining an attribute vector based at least in part on the term frequency and the inverse document frequency.

9. The media of claim 7 , wherein classifying the target attribute comprises determining whether the target attribute comprises a transformation of the source attribute, the determination based on a relationship between the target vector and the source vector.

10. The media of claim 7 , wherein classifying the source attribute comprises determining whether the source attribute is used or unused, the determination based on a relationship between the target vector and the source vector, wherein a source attribute is unused when a source vector generated from the source attribute has no probability of correspondence to any of the target vectors.

11. The media of claim 7 , further comprising receiving selection of a source attribute data set and a target attribute data set via a user interface.

12. The media of claim 11 , further comprising:

generating the data quality metric based on one or more predefined rules for an attribute classification; and

receiving a modification of a predefined rule via the user interface.

13. The media of claim 7 , wherein the source attribute comprises semi-structured or unstructured data and the target attribute comprises structured data.

14. A method for artificial intelligence (“AI”) based generation of data warehouse quality protocols, the method comprising:

generating a source vector from each source attribute in a set of source attributes, the set of source attributes obtained from one or more data sources;

generating a target vector from each target attribute in a set of target attributes, the target attributes comprising target data obtained from one or more data sources and integrated into a data warehouse;

determining a probability of correspondence between the source vector and each of the target vectors;

using an AI model:

determining that a target attribute is a transformation of a source attribute based at least in part on a probability of correspondence between the source vector generated from the source attribute and one of the target vectors, and classifying the source attribute based on the transformation;

outputting a data quality metric for the source attribute based at least in part on the source attribute classification;

determining that a target attribute is a transformation of a source attribute based at least in part on a probability of correspondence between the source vector and the target vector, and classifying the target attribute based on the transformation;

outputting a data quality metric for the target attribute based at least in part on the target attribute classification; and

validating the source data set and the target data set using the data quality metrics, wherein the data quality metrics comprise a ratio of data to errors and a number of empty values.

15. The method of claim 14 , further comprising:

determining a term frequency for an attribute;

determining an inverse document frequency for an attribute; and

determining an attribute vector based at least in part on the term frequency and the inverse document frequency.

16. The method of claim 14 , further comprising receiving a selection of a source attribute data set and a target attribute data set via a user interface.

17. The method of claim 16 , further comprising:

generating the data quality metric based on one or more predefined rules for an attribute classification; and

receiving a modification of one or more of the rules via the user interface.

18. The method of claim 14 , wherein the source attribute comprises semi-structured or unstructured data and the target attribute comprises structured data.