Data preparation and modeling using artificial intelligence

1 . A method performed by one or more computers, the method comprising:

providing, by the one or more computers and for presentation at a user device, data for a user interface to create or edit a data model;

receiving, by the one or more computers, user input through the user interface that indicates one or more data sets;

in response to the receiving of the user input indicating the one or more data sets:

accessing, by the one or more computers, a data model corresponding to the one or more data sets that are indicated by the user input, wherein the data model is accessed from one or more data repositories comprising data models for different data sets; and

generating, by the one or more computers, a set of recommendations for data modeling, data preparation, or data enrichment for the one or more data sets,

wherein at least one recommendation in the set of recommendations is generated based on a set of stored rules that specify criteria for generating recommendations, wherein the at least one recommendation is determined to be appropriate for the one or more data sets based on the stored rules, wherein the at least one recommendation comprises at least one of: duplicate row removal, standardizing a temporal data format, enriching data, or filling in one or more missing values, and

wherein at least one recommendation in the set of recommendations is generated using one or more artificial intelligence and/or machine learning (AI/ML) models;

prioritizing, by the one or more computers, the recommendations in the set of recommendations for presentation in the user interface based on user acceptance or user dismissal of previous recommendations by one or more users;

providing, by the one or more computers, the set of prioritized recommendations for display in the user interface in association with one or more interactive controls to accept or dismiss the recommendations;

in response to receiving user input accepting one or more of the prioritized recommendations in the set of recommendations, updating, by the one or more computers, the data model and the one or more data sets to apply one or more updates corresponding to the accepted recommendations; and

providing, by the one or more computers, the updated one or more data sets and the updated data model to a chatbot or other application, wherein the updated data model indicates column names for columns in the updated one or more data sets and the updated data model includes data describing metrics and attributes of the updated one or more data sets.

2 . The method of claim 1 , wherein the one or more AI/ML models comprises a large language model (LLM).

3 . The method of claim 1 , comprising repeatedly providing additional recommendations for display in the user interface as the data model is being created.

4 . The method of claim 1 , comprising learning from user input that accepts or dismisses recommendations for data modeling, data preparation, or data enrichment, to alter which recommendations are presented for future creation or editing of data models.

5 . The method of claim 1 , comprising:

searching through existing data repositories using to find data sets and models; and

providing a recommendation that indicates a data source to add to the data model.

6 . The method of claim 1 , comprising storing information about attributes and metrics from data sets in a vector database.

7 . The method of claim 6 , comprising using the one or more AI/ML models and the vector database to determine whether portions of the one or more data sets represent attributes or metrics.

8 . The method of claim 6 , comprising:

providing a list of column names to the one or more AI/ML models along with a description of metrics and attributes; and

receiving, from the one or more AI/ML models, an indication of column names with a respective classification.

9 . The method of claim 6 , comprising:

determining a vector representation for each column of one or more columns of data of the one or more data sets;

calculating a distance between the vector representations of the one or more columns and vector representations from the vector database; and

based on the calculated distances, determining, for each column of the one or more columns of data, at least one of:

a type of data object corresponding to the column,

a category or dimension represented by the column, or

a semantic meaning of data in the column.

10 . The method of claim 6 , comprising using the vector database to infer hierarchy relationships among columns or data objects of the one or more data sets, based on similarity to other data sets described by the vector database.

11 . The method of claim 1 , obtaining, from the one or more AI/ML models, inferred information for a portion of the one or more data sets including a semantic role, a data type, a data format, a delimiter, or default action to perform when data is missing.

12 . The method of claim 1 , wherein the set of recommendations comprises a recommendation for an aggregation level for data summarization determined based on characteristics of the one or more data sets.

13 . The method of claim 1 , comprising performing automatic relationship detection and automatic relationship validation for relationships among data objects in the one or more data sets.

14 . The method of claim 1 , wherein the set of recommendations comprises a data modelling recommendation comprising at least one of:

creating a hierarchy or link between multiple data sets,

creating a new relationship between multiple data objects, or

associating data from the one or more data sets with an attribute or metric.

15 . A system comprising:

one or more computers; and

one or more non-transitory computer-readable media storing instructions that are operable, when executed by the one or more computers, to cause the system to perform operations comprising:

providing, by the one or more computers and for presentation at a user device, data for a user interface to create or edit a data model;

receiving, by the one or more computers, user input through the user interface that indicates one or more data sets;

in response to the receiving of the user input indicating the one or more data sets:

accessing, by the one or more computers, a data model corresponding to the one or more data sets that are indicated by the user input, wherein the data model is accessed from one or more data repositories comprising data models for different data sets; and

generating, by the one or more computers, a set of recommendations for data modeling, data preparation, or data enrichment for the one or more data sets,

wherein at least one recommendation in the set of recommendations is generated based on a set of stored rules that specify criteria for generating recommendations, wherein the at least one recommendation is determined to be appropriate for the one or more data sets based on the stored rules, wherein the at least one recommendation comprises at least one of: duplicate row removal, standardizing a temporal data format, enriching data, or filling in one or more missing values, and

wherein at least one recommendation in the set of recommendations is generated using one or more artificial intelligence and/or machine learning (AI/ML) models;

prioritizing, by the one or more computers, the recommendations in the set of recommendations for presentation in the user interface based on user acceptance or user dismissal of previous recommendations by one or more users;

providing, by the one or more computers, the set of prioritized recommendations for display in the user interface in association with one or more interactive controls to accept or dismiss the recommendations;

in response to receiving user input accepting one or more of the prioritized recommendations in the set of recommendations, updating, by the one or more computers, the data model and the one or more data sets to apply one or more updates corresponding to the accepted recommendations; and

providing, by the one or more computers, the updated one or more data sets and the updated data model to a chatbot or other application, wherein the updated data model indicates column names for columns in the updated one or more data sets and the updated data model includes data describing metrics and attributes of the updated one or more data sets.

16 . The system of claim 15 , comprising wherein the operations comprise using a vector database to infer hierarchy relationships among columns or data objects of the one or more data sets, based on similarity to other data sets described by the vector database.

17 . The system of claim 15 , wherein the operations comprise learning from user input that accepts or dismisses recommendations for data modeling, data preparation, or data enrichment, to alter which recommendations are presented for future creation or editing of data models.

18 . One or more non-transitory computer-readable media storing instructions that are operable, when executed by one or more computers, to cause the one or more computers to perform operations comprising:

providing, by the one or more computers and for presentation at a user device, data for a user interface to create or edit a data model;

receiving, by the one or more computers, user input through the user interface that indicates one or more data sets;

in response to the receiving of the user input indicating the one or more data sets:

accessing, by the one or more computers, a data model corresponding to the one or more data sets that are indicated by the user input, wherein the data model is accessed from one or more data repositories comprising data models for different data sets; and

generating, by the one or more computers, a set of recommendations for data modeling, data preparation, or data enrichment for the one or more data sets,

wherein at least one recommendation in the set of recommendations is generated based on a set of stored rules that specify criteria for generating recommendations, wherein the at least one recommendation is determined to be appropriate for the one or more data sets based on the stored rules, wherein the at least one recommendation comprises at least one of: duplicate row removal, standardizing a temporal data format, enriching data, or filling in one or more missing values, and

wherein at least one recommendation in the set of recommendations is generated using one or more artificial intelligence and/or machine learning (AI/ML) models;

prioritizing, by the one or more computers, the recommendations in the set of recommendations for presentation in the user interface based on user acceptance or user dismissal of previous recommendations by one or more users;

providing, by the one or more computers, the set of prioritized recommendations for display in the user interface in association with one or more interactive controls to accept or dismiss the recommendations;

in response to receiving user input accepting one or more of the prioritized recommendations in the set of recommendations, updating, by the one or more computers, the data model and the one or more data sets to apply one or more updates corresponding to the accepted recommendations; and

providing, by the one or more computers, the updated one or more data sets and the updated data model to a chatbot or other application, wherein the updated data model indicates column names for columns in the updated one or more data sets and the updated data model includes data describing metrics and attributes of the updated one or more data sets.