AUTOMATED DATABASE SCHEMA MATCHING

1 . A method for automatically ingesting data from disparate data sources having respective data schemas into a target database having a target data schema, comprising:

receiving, from a user, a data file comprising a plurality of source data columns formatted according to a data schema of a data source and comprising a data dictionary comprising information describing the plurality of source data columns;

generating a plurality of count data for each cell of a plurality of cells selected from a source data column of the plurality of source data columns, each count datum comprising a number of occurrences of a characteristic detected in each cell;

selecting one or more target data columns from a plurality of target data columns specified in the target data schema as being semantically related to the source data column based on the plurality of count data for each cell, a column header of the source data column, and the data dictionary;

displaying, to the user, one or more graphical representations corresponding to the one or more target data columns selected for the source data column;

receiving an input from the user that selects a graphical representation corresponding to a first target data column from the one or more selected target data columns for the source data column;

storing the plurality of source data columns in a data table generated to transform the data schema of the data file into the target data schema of the target database, wherein the source data column corresponds to the first target data column; and

loading the generated data table into the target database.

2 . The method of claim 1 , wherein selecting one or more target data columns from the plurality of target data columns specified in the target data schema comprises:

determining one or more clusters from a plurality of clusters associated with the target data schema based on the plurality of count data for each cell; and

selecting a first set of data columns from the plurality of target data columns based on the one or more determined clusters, wherein the first set of data columns comprises the one or more target data columns.

3 . The method of claim 2 , comprising:

generating a second plurality of count data for each cell of a plurality of cells selected from each target data column of the plurality of target data columns of the target database;

clustering the plurality of cells for each target data column into the plurality of clusters based on the second plurality of count data for each cell; and

associating a set of target data columns from the plurality of target data columns with each cluster based on a number of cells from the set of target data columns being grouped into the cluster.

4 . The method of claim 2 , wherein determining the one or more clusters comprises:

executing a machine learning algorithm configured to assign a cluster from the plurality of clusters to each cell based on the plurality of count data for each cell.

5 . The method of claim 2 , wherein selecting one or more target data columns from the plurality of target data columns specified in the target data schema comprises:

selecting a second set of data columns from the first set of data columns based on header comparisons between the source data column and each target data column of the first set of data columns, wherein the second set of data columns comprises the one or more target data columns.

6 . The method of claim 5 , wherein selecting the second set of data columns comprises:

for each header comparison between the source data column and each target data column of the first set of data columns, determining a number of string operations to convert the source data column into the target data column; and

selecting the second set of data columns from the first set of data columns based on the number of string operations determined for each target data column of the first set of data columns.

7 . The method of claim 6 , wherein each of the string operations comprises deleting a character, adding a character, or substituting a character.

8 . The method of claim 5 , wherein selecting one or more target data columns from the plurality of target data columns specified in the target data schema comprises:

selecting a third set of data columns from the second set of data columns based on data description comparisons between the source data column and each target data column of the second set of data columns, wherein the third set of data columns comprises the one or more target data columns, and wherein the data dictionary comprises the data description for the source data column.

9 . The method of claim 8 , wherein selecting the third set of data columns from the second set of data columns comprises:

vectorizing a data description of the source data column to generate a source vector;

vectorizing the second set of data columns to generate a set of corresponding target vectors;

calculating a set of distances corresponding to distances between the source vector and each target vector of the set of target vectors; and

selecting the third set of data columns from the second set of data columns based on the set of calculated distances.

10 . The method of claim 9 , wherein the third set of data columns correspond to the one or more selected target data columns, comprising:

calculating one or more scores corresponding to the one or more selected target data columns based on the set of distances calculated for the third set of data columns; and

displaying the one or more scores proximate to the one or more graphical representations corresponding to the one or more selected target data columns.

11 . The method of claim 1 , wherein generating the plurality of count data comprises:

randomly selecting a predetermined number of cells from the source data column, wherein the predetermined number of cells corresponds to the plurality of selected cells.

12 . The method of claim 1 , wherein the number of occurrences of a characteristic detected in each cell comprises: a number of alphabetical characters in the cell, a number of digits in the cell, a number of white spaces in the cell, a number of special characters in the cell, a number of total characters in the cell, a number of people names identified in the cell, a number of location names identified in the cell, a number of nouns identified in the cell, or a number of verbs identified in the cell.

13 . The method of claim 1 , wherein displaying the one or more graphical representations corresponding to the one or more selected target data columns comprises:

displaying a graphical icon next to the first target data column from the one or more selected target data columns indicating that the first target data column was previously selected by the user for the source data column.

14 . A system for automatically ingesting data from disparate data sources having respective data schemas into a target database having a target data schema, comprising:

one or more processors; and

memory storing one or more programs that when executed by the one or more processors cause the one or more processors to:

receive, from a user, a data file comprising a plurality of source data columns formatted according to a data schema of a data source and comprising a data dictionary comprising information describing the plurality of source data columns;

select a plurality of cells from a source data column of the plurality of source data columns;

generate a plurality of count data for each cell of the plurality of cells, each count datum comprising a number of occurrences of a characteristic detected in each cell;

select one or more target data columns from a plurality of target data columns specified in the target data schema as being semantically related to the source data column based on the plurality of count data for each cell, a column header of the source data column, and the data dictionary;

display, to the user, one or more graphical representations corresponding to the one or more selected target data columns for the source data column;

receive an input from the user that selects a graphical representation corresponding to a first target data column from the one or more selected target data columns for the source data column; and

store the plurality of source data columns in a data table generated to transform the data schema of the data file into the target data schema of the target database, wherein the source data column corresponds to the first target data column; and

load the generated data table into the target database.

15 . The system of claim 14 , wherein to select one or more target data columns from the plurality of target data columns specified in the target data schema, wherein the one or more processors are caused to:

determine one or more clusters from a plurality of clusters associated with the target data schema based on the plurality of count data for each cell; and

select a first set of data columns from the plurality of target data columns based on the one or more determined clusters, wherein the first set of data columns comprises the one or more target data columns.

16 . The system of claim 15 , wherein the one or more processors are caused to:

generate a second plurality of count data for each cell of a plurality of cells selected from each target data column of the plurality of target data columns of the target database;

cluster the plurality of cells for each target data column into the plurality of clusters based on the second plurality of count data for each cell; and

associate a set of target data columns from the plurality of target data columns with each cluster based on a number of cells from the set of target data columns being grouped into the cluster.

17 . The system of claim 15 , wherein to determine the one or more clusters, the one or more processors are caused to:

execute a machine learning algorithm configured to assign a cluster from the plurality of clusters to each cell based on the plurality of count data for each cell.

18 . The system of claim 15 , wherein to select one or more target data columns from the plurality of target data columns specified in the target data schema, the one or more processors are caused to:

select a second set of data columns from the first set of data columns based on header comparisons between the source data column and each target data column of the first set of data columns, wherein the second set of data columns comprises the one or more target data columns.

19 . The system of claim 18 , wherein to select the second set of data columns, the one or more processors are caused to:

for each header comparison between the source data column and each target data column of the first set of data columns, determine a number of string operations to convert the source data column into the target data column; and

select the second set of data columns from the first set of data columns based on the number of string operations determined for each target data column of the first set of data columns.

20 . The system of claim 19 , wherein each of the string operations comprises deleting a character, adding a character, or substituting a character.

21 . The system of claim 18 , wherein to select one or more target data columns from the plurality of target data columns specified in the target data schema, the one or more processors are caused to:

select a third set of data columns from the second set of data columns based on data description comparisons between the source data column and each target data column of the second set of data columns, wherein the third set of data columns comprises the one or more target data columns, and wherein the data dictionary comprises the data description for the source data column.

22 . The system of claim 21 , wherein to select the third set of data columns from the second set of data columns, the one or more processors are caused to:

vectorize a data description of the source data column to generate a source vector;

vectorize the second set of data columns to generate a set of corresponding target vectors;

calculate a set of distances corresponding to distances between the source vector and each target vector of the set of target vectors; and

select the third set of data columns from the second set of data columns based on the set of calculated distances.

23 . The system of claim 22 , wherein the third set of data columns correspond to the one or more selected target data columns, and the one or more processors are caused to:

calculate one or more scores corresponding to the one or more selected target data columns based on the set of distances calculated for the third set of data columns; and

display the one or more scores proximate to the one or more graphical representations corresponding to the one or more selected target data columns.

24 . The system of claim 14 , wherein to generate the plurality of count data. the one or more processors are caused to:

randomly select a predetermined number of cells from the source data column, wherein the predetermined number of cells corresponds to the plurality of selected cells.

25 . The system of claim 14 , wherein the number of occurrences of a characteristic detected in each cell comprises: a number of alphabetical characters in the cell, a number of digits in the cell, a number of white spaces in the cell, a number of special characters in the cell, a number of total characters in the cell, a number of people names identified in the cell, a number of location names identified in the cell, a number of nouns identified in the cell, or a number of verbs identified in the cell.

26 . The system of claim 1 , wherein to display the one or more graphical representations corresponding to the one or more selected target data columns, the one or more processors are caused to:

display a graphical icon next to the first target data column from the one or more selected target data columns indicating that the first target data column was previously selected by the user for the source data column.

27 . A non-transitory computer-readable storage medium comprising instructions for ingesting data from disparate data sources having respective data schemas into a target database having a target data schema, wherein the instructions, when executed by one or more processors, cause the one or more processors to perform instructions comprising:

receiving, from a user, a data file comprising a plurality of source data columns formatted according to a data schema of a data source and comprising a data dictionary comprising information describing the plurality of source data columns;

generating a plurality of count data for each cell of a plurality of cells selected from a source data column of the plurality of source data columns, each count datum comprising a number of occurrences of a characteristic detected in each cell;

selecting one or more target data columns from a plurality of target data columns specified in the target data schema as being semantically related to the source data column based on the plurality of count data for each cell, a column header of the source data column, and the data dictionary;

displaying, to the user, one or more graphical representations corresponding to the one or more target data columns selected for the source data column;

receiving an input from the user that selects a graphical representation corresponding to a first target data column from the one or more selected target data columns for the source data column;

storing the plurality of source data columns in a data table generated to transform the data schema of the data file into the target data schema of the target database, wherein the source data column corresponds to the first target data column; and

loading the generated data table into the target database.