Systems and methods for annotating datasets
Disclosed herein are systems and methods for joining datasets. The system may include one or more processors and a memory storing instructions that, when executed by the one or more processors. The processor may cause the system to perform determining at least a first database table to be annotated, the first database table including a set of columns and rows of a dataset. In some embodiments, the system may include determining at least one typeclass that applies to one or more columns included in the first database table, wherein the typeclass describes values stored in the one or more columns and annotating the one or more columns, wherein the annotated columns are associated with the typeclass.
1. A system comprising:
one or more processors; and
a memory storing instructions that, when executed by the one or more processors, cause the system to perform:
determining a column of a first database table to be annotated, the first database table including a set of columns and rows of a dataset;
using a trained machine learning model:
determining a typeclass that applies to the column of the first database table, wherein the typeclass describes values stored in the column;
annotating the column based on the determined typeclass;
determining that the typeclass also applies to a second column included in a second database table in response to a probability of a correspondence between respective data values and data formats in the column and the second column satisfying a threshold;
determining, to join the column and the second column in response to:
the determination that the typeclass also applies to the second column; and
a historical frequency of joining the first column and the second column exceeding a second threshold, the historical frequency comprising a degree of recency of a previous join operation between the column and the second column; and
joining, in response to the determination to join the column and the second column, the column and the second column.
2. The system of claim 1 , wherein the typeclass describes metadata information for the one or more columns.
3. The system of claim 2 , wherein the metadata information describes a data type corresponding to data values in the one or more columns.
4. The system of claim 2 , wherein the metadata information describes a data format corresponding to data values in the one or more columns.
5. The system of claim 1 , wherein the typeclass is associated with one or more data validations.
6. The system of claim 5 , wherein the data validations are automatically applied to validate values stored in the one or more columns.
7. The system of claim 1 , wherein the instructions further cause the system to perform:
joining the first database table with at least one second database table based at least in part on the one or more annotated columns.
8. The system of claim 1 , wherein the probability of the correspondence between respective data values or data formats in the column and the second column is determined based on patterns of data entries in the column and the second column.
9. The system of claim 8 , wherein the patterns include relative positions of signs in the data entries.
10. The system of claim 1 , wherein the determination to join the column to the second column is further based on an aggregated edit distance between entries of the column and second entries of the second column.
11. A method being implemented by a computing system including one or more physical processors and storage media storing machine-readable instructions, the method comprising:
determining a column of a first database table to be annotated, the first database table including a set of columns and rows of a dataset;
using a trained machine learning model:
determining a typeclass that applies to the column of the first database table, wherein the typeclass describes values stored in the column;
annotating the column based on the determined typeclass;
determining that the typeclass also applies to a second column included in a second database table in response to a probability of a correspondence between respective data values and data formats in the column and the second column satisfying a threshold;
determining to join the column and the second column in response to a historical frequency of joining the first column and the second column exceeding a second threshold, the historical frequency comprising a degree of recency of a previous join operation between the column and the second column; and
joining, in response to the determination to join the column and the second column, the column and the second column.
12. The method of claim 11 , wherein the typeclass describes metadata information for the one or more columns.
13. The method of claim 11 , wherein the typeclass is associated with one or more data validations.
14. The method of claim 13 , wherein the data validations are automatically applied to validate values stored in the one or more columns.
15. The method of claim 11 , wherein the instructions further cause the processors to perform:
joining the first database table with at least one second database table based at least in part on the one or more annotated columns.
16. A non-transitory computer readable medium comprising instructions that, when executed, cause one or more processors to perform:
determining a column of a first database table to be annotated, the first database table including a set of columns and rows of a dataset;
using a trained machine learning model:
determining a typeclass that applies to the column of the first database table, wherein the typeclass describes values stored in the column;
annotating the column based on the determined typeclass;
determining that the typeclass also applies to a second column included in a second database table in response to a probability of a correspondence between respective values and data formats in the column and the second column satisfying a threshold;
determining to join the column and the second column in response to a historical frequency of joining the first column and the second column exceeding a second threshold, the historical frequency comprising a degree of recency of a previous join operation between the column and the second column; and
joining, in response to the determination to join the column and the second column, the column and the second column.
17. The non-transitory computer readable medium of claim 16 , wherein the typeclass describes metadata information for the one or more columns.
18. The non-transitory computer readable medium of claim 17 , wherein the metadata information describes a data type corresponding to data values in the one or more columns.
19. The non-transitory computer readable medium of claim 17 , wherein the metadata information describes a data format corresponding to data values in the one or more columns.