Schema matching of disparate data sources using embedding-based similarity scores
Embodiments provide for schema matching between multiple disparate data sources using multi-dimensional matrices, external embeddings, transformation operations, and similarity metrics.
1. A system comprising one or more processors and at least one memory storing processor executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
generating, for a first data structure associated with a first data set and based at least in part on an external embeddings data structure comprising a lookup table, a first embeddings set by appending, for each (1) first code of the first data structure or (2) first portion of the first data structure and to the first data structure, a corresponding embedding of one or more first corresponding embeddings in the lookup table for the first code of the first data structure or the first portion of the first data structure;
generating, for a second data structure associated with a second data set and based at least in part on the lookup table of the external embeddings data structure, a second embeddings set by appending, for each (1) second code of the second data structure or (2) second portion of the second data structure and to the second data structure, a corresponding embedding of the one or more second corresponding embeddings in the lookup table for the second code of the second data structure or the second portion of the second data structure;
generating a plurality of first row representations for the first embeddings set by aggregating the one or more first corresponding embeddings in the first embeddings set such that each row in the first data structure is aggregated into a first single row representation;
generating a plurality of second row representations for the second embeddings set by aggregating the one or more second corresponding embeddings in the second embeddings set such that each row in the second data structure is aggregated into a second single row representation;
generating a first column representation for the first embeddings set by aggregating the plurality of first row representations;
generating a second column representation for the second embeddings set by aggregating the plurality of second row representations;
generating a similarity metric based on the first column representation and the second column representation;
based on the similarity metric, generating a similarity indication element associated with the first data structure and the second data structure; and
providing the similarity indication element associated with the first data structure and the second data structure while restricting access to raw data contained in the first data structure and the second data structure.
2. The system of claim 1 , wherein the operations further comprise:
receiving the first data structure from a first data source; and
receiving the second data structure from a second data source.
3. The system of claim 1 , wherein the first data structure comprises a first plurality of first records arranged according to a first matrix and the second data structure comprises a second plurality of second records arranged according to a second matrix.
4. The system of claim 3 , wherein the first data structure is associated with a first unknown schema.
5. The system of claim 4 , wherein the second data structure is associated with a second unknown schema.
6. The system of claim 1 , wherein the similarity metric is generated based on a cosine similarity metric, a Euclidean distance, or a Jaccard similarity metric.
7. The system of claim 1 , wherein the lookup table contains a first column representing terms or codes and a second column representing corresponding embeddings associated with the terms or codes.
8. The system of claim 1 , wherein the external embeddings data structure is generated based on a trained embeddings machine learning model.
9. The system of claim 1 , wherein:
the first code of the first data structure or the first portion of the first data structure includes a first medical code;
the second code of the second data structure or the second portion of the second data structure includes a second medical code; and
the first medical code and the second medical code are either (1) codes of a same diagnosis but from different levels in a taxonomy or (2) a same code but one of the first medical code or the second medical code has a missing value relative to the other.
10. A computer-implemented method, comprising:
generating, by one or more processors and for a first data structure associated with a first data set and based at least in part on an external embeddings data structure comprising a lookup table, a first embeddings set by appending, for each (1) first code of the first data structure or (2) first portion of the first data structure and to the first data structure, a corresponding embedding of one or more first corresponding embeddings in the lookup table for the first code of the first data structure or the first portion of the first data structure;
generating, by the one or more processors and for a second data structure associated with a second data set and based at least in part on the lookup table of the external embeddings data structure, a second embeddings set by appending, for each (1) second code of the second data structure or (2) second portion of the second data structure and to the second data structure, a corresponding embedding of the one or more second corresponding embeddings in the lookup table for the second code of the second data structure or the second portion of the second data structure;
generating, by the one or more processors, a plurality of first row representations for the first embeddings set by aggregating the one or more first corresponding embeddings in the first embeddings set such that each row in the first data structure is aggregated into a first single row representation;
generating, by the one or more processors, a plurality of second row representations for the second embeddings set by aggregating the one or more second corresponding embeddings in the second embeddings set such that each row in the second data structure is aggregated into a second single row representation;
generating, by the one or more processors, a first column representation for the first embeddings set by aggregating the plurality of first row representations;
generating, by the one or more processors, a second column representation for the second embeddings set by aggregating the plurality of second row representations;
generating, by the one or more processors, a similarity metric based on the first column representation and the second column representation;
based on the similarity metric, generating, by the one or more processors, a similarity indication element associated with the first data structure and the second data structure; and
providing, by the one or more processors, the similarity indication element associated with the first data structure and the second data structure while restricting access to raw data contained in the first data structure and the second data structure.
11. The computer-implemented method of claim 10 , further comprising:
receiving the first data structure from a first data source; and
receiving the second data structure from a second data source.
12. The computer-implemented method of claim 10 , wherein the first data structure comprises a first plurality of first records arranged according to a first matrix and the second data structure comprises a second plurality of second records arranged according to a second matrix.
13. The computer-implemented method of claim 12 , wherein the first data structure is associated with a first unknown schema and wherein the second data structure is associated with a second unknown schema.
14. The computer-implemented method of claim 10 , wherein the similarity metric is generated based on a cosine similarity metric, a Euclidean distance, or a Jaccard similarity metric.
15. The computer-implemented method of claim 10 , wherein the lookup table contains a first column representing terms or codes and a second column representing corresponding embeddings associated with the terms or codes.
16. The computer-implemented method of claim 10 , wherein the external embeddings data structure is generated based on a trained embeddings machine learning model.
17. One or more non-transitory computer-readable storage media storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
generating, for a first data structure associated with a first data set and based at least in part on an external embeddings data structure comprising a lookup table, a first embeddings set by appending, for each (1) first code of the first data structure or (2) first portion of the first data structure and to the first data structure, a corresponding embedding of one or more first corresponding embeddings in the lookup table for the first code of the first data structure or the first portion of the first data structure;
generating, for a second data structure associated with a second data set and based at least in part on the lookup table of the external embeddings data structure, a second embeddings set by appending, for each (1) second code of the second data structure or (2) second portion of the second data structure and to the second data structure, a corresponding embedding of the one or more second corresponding embeddings in the lookup table for the second code of the second data structure or the second portion of the second data structure;
generating a plurality of first row representations for the first embeddings set by aggregating the one or more first corresponding embeddings in the first embeddings set such that each row in the first data structure is aggregated into a first single row representation;
generating a plurality of second row representations for the second embeddings set by aggregating the one or more second corresponding embeddings in the second embeddings set such that each row in the second data structure is aggregated into a second single row representation;
generating a first column representation for the first embeddings set by aggregating the plurality of first row representations;
generating a second column representation for the second embeddings set by aggregating the plurality of second row representations;
generating a similarity metric based on the first column representation and the second column representation;
based on the similarity metric, generating a similarity indication element associated with the first data structure and the second data structure; and
providing the similarity indication element associated with the first data structure and the second data structure while restricting access to raw data contained in the first data structure and the second data structure.
18. The one or more non-transitory computer-readable storage media of claim 17 , wherein the operations further comprise:
receiving the first data structure from a first data source; and
receiving the second data structure from a second data source.
19. The one or more non-transitory computer-readable storage media of claim 17 , wherein the first data structure comprises a first plurality of first records arranged according to a first matrix and the second data structure comprises a second plurality of second records arranged according to a second matrix.
20. The one or more non-transitory computer-readable storage media of claim 19 , wherein the first data structure is associated with a first unknown schema and wherein the second data structure is associated with a second unknown schema.