Systems and methods for data entity matching between disparate datasets
Embodiments of the present disclosure provide systems and methods for matching data entities across disparate datasets. One method may include generating a reference numeric vector for a reference data object and generating a plurality of match candidate numeric vectors for a plurality of match candidate data objects. The method may also include identifying a subset of match candidate data objects from the plurality of match candidate data objects and generating a plurality of data features based on a comparison between one or more reference object attributes corresponding to the reference data object and one or more matching object attributes corresponding to the subset of match candidate data objects. The method may also include identifying a match between the reference data object and a match candidate data object of the subset of match candidate data objects based on the plurality of data features.
1 . A computer-implemented method comprising:
receiving, by one or more processors, from a first data source, a reference data object comprising a plurality of data elements;
generating, by the one or more processors, a text representation for the reference data object according to a predefined sequence, wherein the text representation comprises a string value comprising at least two data elements, of the plurality of data elements, positioned in accordance with the predefined sequence;
generating, by the one or more processors, a reference numeric vector for the reference data object based on the text representation;
generating, by the one or more processors, a plurality of match candidate numeric vectors for a plurality of match candidate data objects from a second data source based on a plurality of text representations associated with the plurality of match candidate data objects;
identifying, by the one or more processors, a subset of match candidate data objects from the plurality of match candidate data objects based on a comparison between the reference numeric vector and the plurality of match candidate numeric vectors;
generating, by the one or more processors, a plurality of data features based on a comparison between (a) a plurality of reference object attributes corresponding to the reference data object and (b) a plurality of matching object attributes corresponding to the subset of match candidate data objects;
providing, by the one or more processors, the plurality of data features to an entity matching machine learning model to generate a plurality of match scores, wherein a match score of the plurality of match scores comprises a probabilistic value of a match likelihood between one of the subset of match candidate data objects and the reference data object; and
identifying, by the one or more processors, a match between the reference data object and a match candidate data object of the subset of match candidate data objects based on the plurality of match scores and a threshold.
2 . The computer-implemented method of claim 1 , wherein the plurality of data features comprises a string matching feature corresponding to a text attribute pair, wherein the text attribute pair comprises:
(i) a reference object attribute of the plurality of reference object attributes that comprises a first text string corresponding to the reference data object, and
(ii) a matching object attribute of the plurality of matching object attributes that comprises a second text string corresponding to the match candidate data object of the subset of match candidate data objects.
3 . The computer-implemented method of claim 2 , wherein the string matching feature comprises a Levenshtein ratio between the first text string and the second text string.
4 . The computer-implemented method of claim 1 , wherein the plurality of data features comprises a geographic distance feature corresponding to a region attribute pair, wherein the region attribute pair comprises:
(i) a reference object attribute of the plurality of reference object attributes that comprises a first region identifier corresponding to the reference data object, and
(ii) a matching object attribute of the plurality of matching object attributes that comprises a second region identifier corresponding to the match candidate data object of the subset of match candidate data objects.
5 . The computer-implemented method of claim 4 , wherein the geographic distance feature identifies a geographic distance between the reference data object and the match candidate data object.
6 . The computer-implemented method of claim 4 , wherein the geographic distance feature comprises a Haversine distance between the first region identifier and the second region identifier.
7 . The computer-implemented method of claim 1 , wherein the plurality of data features comprises a vector similarity score corresponding to a vector attribute pair, wherein the vector attribute pair comprises:
(i) the reference numeric vector corresponding to the reference data object, and
(ii) a respective candidate numeric vector of the plurality of match candidate numeric vectors that corresponds to the match candidate data object of the subset of match candidate data objects.
8 . The computer-implemented method of claim 1 , wherein the reference data object is associated with a ground truth data object and the computer-implemented method further comprises:
modifying a parameter of the entity matching machine learning model based on a comparison between the ground truth data object and the match candidate data object.
9 . The computer-implemented method of claim 1 , wherein identifying the subset of match candidate data objects from the plurality of match candidate data objects comprises:
generating a vector similarity score between the reference numeric vector and a respective match candidate numeric vector of the plurality of match candidate numeric vectors; and
identifying the subset of match candidate data objects from the plurality of match candidate data objects based on the vector similarity score.
10 . The computer-implemented method of claim 9 , wherein identifying the subset of match candidate data objects from the plurality of match candidate data objects comprises:
selecting the subset of match candidate data objects from the plurality of match candidate data objects based on a comparison between the vector similarity score and a threshold similarity score.
11 . The computer-implemented method of claim 9 , wherein identifying the subset of match candidate data objects from the plurality of match candidate data objects comprises:
selecting the subset of match candidate data objects from the plurality of match candidate data objects based on the vector similarity score, a plurality of vector similarity scores corresponding to the plurality of match candidate data objects, and a threshold number of candidate data objects.
12 . A computing system comprising memory and one or more processors communicatively coupled to the memory, the one or more processors configured to:
receive from a first data source, a reference data object comprising a plurality of data elements;
generate a text representation for the reference data object according to a predefined sequence, wherein the text representation comprises a string value comprising at least two data elements, of the plurality of data elements, positioned in accordance with the predefined sequence;
generate a reference numeric vector for a reference data object from a first data source based on a text representation associated with the reference data object;
generate a plurality of match candidate numeric vectors for a plurality of match candidate data objects from a second data source based on a plurality of text representations associated with the plurality of match candidate data objects;
identify a subset of match candidate data objects from the plurality of match candidate data objects based on a comparison between the reference numeric vector and the plurality of match candidate numeric vectors;
generate a plurality of data features based on a comparison between (a) a plurality of reference object attributes corresponding to the reference data object and (b) a plurality of matching object attributes corresponding to the subset of match candidate data objects;
provide the plurality of data features to an entity matching machine learning model to generate a plurality of match scores, wherein a match score of the plurality of match scores comprises a probabilistic value of a match likelihood between one of the subset of match candidate data objects and the reference data object; and
identify a match between the reference data object and a match candidate data object of the subset of match candidate data objects based on the plurality of match scores and a threshold.
13 . The computing system of claim 12 , wherein the plurality of data features comprises a string matching feature corresponding to a text attribute pair, wherein the text attribute pair comprises:
(i) a reference object attribute of the plurality of reference object attributes that comprises a first text string corresponding to the reference data object, and
(ii) a matching object attribute of the plurality of matching object attributes that comprises a second text string corresponding to the match candidate data object of the subset of match candidate data objects.
14 . The computing system of claim 13 , wherein the string matching feature comprises a Levenshtein ratio between the first text string and the second text string.
15 . The computing system of claim 12 , wherein the plurality of data features comprises a geographic distance feature corresponding to a region attribute pair, wherein the region attribute pair comprises:
(i) a reference object attribute of the plurality of reference object attributes that comprises a first region identifier corresponding to the reference data object, and
(ii) a matching object attribute of the plurality of matching object attributes that comprises a second region identifier corresponding to the match candidate data object of the subset of match candidate data objects.
16 . The computing system of claim 15 , wherein the geographic distance feature identifies a geographic distance between the reference data object and the match candidate data object.
17 . The computing system of claim 15 , wherein the geographic distance feature comprises a Haversine distance between the first region identifier and the second region identifier.
18 . The computing system of claim 12 , wherein the plurality of data features comprises a vector similarity score corresponding to a vector attribute pair, wherein the vector attribute pair comprises:
(i) the reference numeric vector corresponding to the reference data object, and
(ii) a respective candidate numeric vector of the plurality of match candidate numeric vectors that corresponds to the match candidate data object of the subset of match candidate data objects.
19 . The computing system of claim 12 , wherein the reference data object is associated with a ground truth data object and the one or more processors are further configured to:
modify a parameter of the entity matching machine learning model based on a comparison between the ground truth data object and the match candidate data object.
20 . One or more non-transitory computer-readable storage media including instructions that, when executed by one or more processors, cause the one or more processors to:
receive from a first data source, a reference data object comprising a plurality of data elements;
generate a text representation for the reference data object according to a predefined sequence, wherein the text representation comprises a string value comprising at least two data elements, of the plurality of data elements, positioned in accordance with the predefined sequence;
generate a reference numeric vector for a reference data object from a first data source based on a text representation associated with the reference data object;
generate a plurality of match candidate numeric vectors for a plurality of match candidate data objects from a second data source based on a plurality of text representations associated with the plurality of match candidate data objects;
identify a subset of match candidate data objects from the plurality of match candidate data objects based on a comparison between the reference numeric vector and the plurality of match candidate numeric vectors;
generate a plurality of data features based on a comparison between (a) a plurality of reference object attributes corresponding to the reference data object and (b) a plurality of matching object attributes corresponding to the subset of match candidate data objects;
provide the plurality of data features to an entity matching machine learning model to generate a plurality of match scores, wherein a match score of the plurality of match scores comprises a probabilistic value of a match likelihood between one of the subset of match candidate data objects and the reference data object; and
identify a match between the reference data object and a match candidate data object of the subset of match candidate data objects based on the plurality of data features.