Flexible entity resolution networks
Among other techniques, techniques for machine learning-based entity resolution are described. An example method includes receiving an entity resolution request, the entity resolution request indicating a first entity and a second entity; identifying a plurality of first attributes in a data model; identifying a plurality of second attributes in the data model; creating a first string based on the plurality of first attributes of the data model; creating a second string based on the plurality of second attributes of the data model; generating a first prompt based on the first string; generating a second prompt based on the second string; providing the first prompt to a domain-agnostic large language model; generating, by the domain-agnostic large language model using the first prompt, a first domain-agnostic large language model result; clipping the first domain-agnostic large language model result; providing the second prompt to the domain-agnostic large language model; generating, by the domain-agnostic large language model using the second prompt, a second domain-agnostic large language model result; clipping the second domain-agnostic large language model result; generating, by a downstream neural network classifier, a machine learning final result based on the clipped first and second domain-agnostic large language model result; and merging, based on the machine learning final result, the first entity and the second entity.
1 . A system comprising:
one or more processors; and
memory storing instructions that, when executed by the one or more processors, cause the system to perform:
receiving a plurality of different machine learning models;
testing each of the plurality of different machine learning models, the testing including simulating a respective match performance of each of the plurality of different machine learning models;
selecting, based on the testing and the simulating, a domain-agnostic large language model of the plurality of different machine learning models;
receiving an entity resolution request, the entity resolution request indicating a first entity and a second entity;
identifying a plurality of first attributes in a data model;
identifying a plurality of second attributes in the data model;
creating a first string based on the plurality of first attributes in the data model;
creating a second string based on the plurality of second attributes in the data model;
generating a first prompt based on the first string;
generating a second prompt based on the second string;
providing the first prompt to the domain-agnostic large language model;
generating, by the domain-agnostic large language model using the first prompt, a first domain-agnostic large language model result;
clipping the first domain-agnostic large language model result;
providing the second prompt to the domain-agnostic large language model;
generating, by the domain-agnostic large language model using the second prompt, a second domain-agnostic large language model result;
clipping the second domain-agnostic large language model result;
generating, by a downstream neural network classifier, a machine learning final result based on the clipped first and second domain-agnostic large language model results;
merging, based on the machine learning final result, the first entity and the second entity.
2 . The system of claim 1 , wherein the plurality of different machine learning models are received from a user.
3 . The system of claim 1 , wherein the instructions, when executed by the one or more processors, cause the system to perform tuning the domain-agnostic large language model based on user feedback received through a graphical user interface.
4 . A method comprising:
receiving a plurality of different machine learning models;
testing each of the plurality of different machine learning models, the testing including simulating a respective match performance of each of the plurality of different machine learning models;
selecting, based on the testing and the simulating, a domain-agnostic large language model of the plurality of different machine learning models;
receiving an entity resolution request, the entity resolution request indicating a first entity and a second entity;
identifying a plurality of first attributes in a data model;
identifying a plurality of second attributes in the data model;
creating a first string based on the plurality of first attributes in the data model;
creating a second string based on the plurality of second attributes in the data model;
generating a first prompt based on the first string;
generating a second prompt based on the second string;
providing the first prompt to the domain-agnostic large language model;
generating, by the domain-agnostic large language model using the first prompt, a first domain-agnostic large language model result;
clipping the first domain-agnostic large language model result;
providing the second prompt to the domain-agnostic large language model;
generating, by the domain-agnostic large language model using the second prompt, a second domain-agnostic large language model result;
clipping the second domain-agnostic large language model result;
generating, by a downstream neural network classifier, a machine learning final result based on the clipped first and second domain-agnostic large language model results;
merging, based on the machine learning final result, the first entity and the second entity.
5 . The method of claim 4 , wherein the plurality of different machine learning models are received from a user.
6 . A system comprising:
one or more processors; and
memory storing instructions that, when executed by the one or more processors, cause the system to perform:
receiving a plurality of different machine learning models;
testing each of the plurality of different machine learning models, the testing including simulating a respective match performance of each of the plurality of different machine learning models;
selecting, based on the testing and the simulating, a domain-agnostic large language model of the plurality of different machine learning models;
swapping the domain-agnostic large language model with an approximator network, wherein the approximator network approximates outputs of the domain-agnostic large language model;
receiving an entity resolution request, the entity resolution request indicating a first entity and a second entity;
identifying a plurality of first attributes in a data model;
identifying a plurality of second attributes in the data model;
creating a first string based on the plurality of first attributes in the data model;
creating a second string based on the plurality of second attributes in the data model;
generating a first prompt based on the first string;
generating a second prompt based on the second string;
providing the first prompt to the approximator network;
generating, by the approximator network using the first prompt, a first approximator network result;
clipping the first approximator network result;
providing the second prompt to another approximator network;
generating, by the other approximator network using the second prompt, a second approximator network result;
clipping the second approximator network result;
generating, by a downstream neural network classifier, a machine learning final result based on the clipped first and second approximator network results;
merging, based on the machine learning final result, the first entity and the second entity.
7 . The system of claim 6 , wherein the plurality of different machine learning models are received from a user.
8 . The system of claim 6 , wherein the swapping is performed pre-runtime.
9 . The system of claim 6 , wherein the swapping is performed during runtime.
10 . The system of claim 6 , wherein the swapping is performed post-runtime.
11 . The system of claim 6 , wherein the approximator network is smaller than the domain-agnostic large language model.
12 . A method comprising:
receiving a plurality of different machine learning models;
testing each of the plurality of different machine learning models, the testing including simulating a respective match performance of each of the plurality of different machine learning models;
selecting, based on the testing and the simulating, a domain-agnostic large language model of the plurality of different machine learning models;
swapping the domain-agnostic large language model with an approximator network, wherein the approximator network approximates outputs of the domain-agnostic large language model;
receiving an entity resolution request, the entity resolution request indicating a first entity and a second entity;
identifying a plurality of first attributes in a data model;
identifying a plurality of second attributes in the data model;
creating a first string based on the plurality of first attributes in the data model;
creating a second string based on the plurality of second attributes in the data model;
generating a first prompt based on the first string;
generating a second prompt based on the second string;
providing the first prompt to the approximator network;
generating, by the approximator network using the first prompt, a first approximator network result;
clipping the first approximator network result;
providing the second prompt to another approximator network;
generating, by the other approximator network using the second prompt, a second approximator network result;
clipping the second approximator network result;
generating, by a downstream neural network classifier, a machine learning final result based on the clipped first and second approximator network results;
merging, based on the machine learning final result, the first entity and the second entity.
13 . The method of claim 12 , wherein the plurality of different machine learning models are received from a user.
14 . The method of claim 12 , wherein the swapping is performed pre-runtime.
15 . The method of claim 12 , wherein the swapping is performed during runtime.
16 . The method of claim 12 , wherein the swapping is performed post-runtime.
17 . The method of claim 12 , wherein the approximator network is smaller than the domain-agnostic large language model.