Entity resolution framework for data matching
Systems and methods are described for matching a corrupted database record with a record of a validated database. The system receives a corrupted record from a first database. The corrupted record is vectorized to create an input data vector. A denoised data vector is generated by applying a denoising autoencoder to the input data vector, where the denoising autoencoder is specific to the first database. The system compares the denoised data vector with each of a plurality of validated data vectors generated based on records of the validated database to determine that a first denoised data vector matches a matching vector. In response, the system trains the denoising autoencoder using a data pair that includes the input data vector and the matching vector. The system also outputs the validated record that was used to generate the first matching vector.
1. A method comprising:
receiving a corrupted record from a first database;
generating a first input data vector based on the corrupted record using a denoising autoencoder, wherein the denoising autoencoder was trained using a plurality of records from the first database;
generating a second data vector based on a validated record from a validated database; and
in response to determining a match based on comparing the first input data vector with the second data vector:
transmitting the validated record to the first database.
2. The method of claim 1 , further comprising, in response to determining that a match does not exist, outputting an indication of a match failure.
3. The method of claim 1 , further comprising, in response to determining the match:
determining that the corrupted record is incomplete, wherein the corrupted record comprises an incomplete textual pair of data that consists of a label and a value; and
in response to determining that the corrupted record is incomplete:
transmitting from the first database a validated label and value to cause the first database to complete the corrupted record by inclusion of the validated label and value.
4. The method of claim 1 , further comprising:
using an additional training example data pair as an input to a neural network;
generating an output based on the additional training example data pair as the input; and
evaluating the neural network's performance based on the output.
5. The method of claim 4 , wherein training the denoising autoencoder using the plurality of records from the first database comprises adjusting neuron connections between neurons of a neural network based on the example data pairs.
6. The method of claim 1 , further comprising:
accessing a plurality of denoising autoencoders;
selecting a denoising autoencoder, from the plurality of autoencoders, that is specific to the first database; and
using the selected autoencoder to generate the first input data vector based on the corrupted record.
7. The method of claim 6 , wherein the denoising autoencoder specific to the first database further comprises a neural network that comprises an input layer, a hidden layer, and an output layer, and wherein the hidden layer includes fewer neurons than the input layer.
8. The method of claim 1 , further comprising, providing a data pair comprising the first input data vector and the second data vector as a training example data pair to the denoising autoencoder to train the denoising autoencoder to denoise input from the first database.
9. The method of claim 1 , further comprising:
comparing the first input data vector with a plurality of validated data vectors, wherein the comparison includes performing binary pairwise comparisons between elements of the first input data vector and elements of the plurality of validated data vectors; and
calculating a percentage of pairwise comparisons that resulted in a match.
10. The method of claim 9 , further comprising, continuously updating the denoising autoencoder when a match is made between the first input data vector and one or more of the plurality of validated data vectors.
11. A system comprising:
communications circuitry of a device configured to receive a corrupted record from a first database; and
control circuitry of the device configured to:
generate a first input data vector based on the corrupted record using a denoising autoencoder, wherein the denoising autoencoder was trained using a plurality of records from the first database;
generate a second data vector based on a validated record from a validated database; and
in response to determining a match based on comparing the first input data vector with the second data vector:
transmit, using the communications circuitry, the validated record to the first database.
12. The system of claim 11 , further comprising, in response to determining that a match does not exist, the control circuitry configured to output an indication of a match failure.
13. The system of claim 11 , further comprising, in response to determining the match, the control circuitry configured to:
determine that the corrupted record is incomplete, wherein the corrupted record comprises an incomplete textual pair of data that consists of a label and a value; and
in response to determining that the corrupted record is incomplete:
transmit from the first database a validated label and value to cause the first database to complete the corrupted record by inclusion of the validated label and value.
14. The system of claim 11 , wherein the control circuitry is further configured to:
use an additional training example data pair as an input to a neural network;
generate an output based on the additional training example data pair as the input; and
evaluate the neural network's performance based on the output.
15. The system of claim 14 , wherein training the denoising autoencoder using the plurality of records from the first database comprises the control circuitry is configured to adjust neuron connections between neurons of a neural network based on the example data pairs.
16. The system of claim 11 , wherein the control circuitry is further configured to:
access a plurality of denoising autoencoders;
select a denoising autoencoder, from the plurality of autoencoders, that is specific to the first database; and
use the selected autoencoder to generate the first input data vector based on the corrupted record.
17. The system of claim 16 , wherein the denoising autoencoder specific to the first database further comprises a neural network that comprises an input layer, a hidden layer, and an output layer, and wherein the hidden layer includes fewer neurons than the input layer.
18. The system of claim 11 , wherein the control circuitry is further configured to provide a data pair comprising the first input data vector and the second data vector as a training example data pair to the denoising autoencoder to train the denoising autoencoder to denoise input from the first database.
19. The system of claim 11 , wherein the control circuitry is further configured to:
compare the first input data vector with a plurality of validated data vectors, wherein the comparison includes performing binary pairwise comparisons between elements of the first input data vector and elements of the plurality of validated data vectors; and
calculate a percentage of pairwise comparisons that resulted in a match.
20. The system of claim 19 , wherein the control circuitry is further configured to continuously update the denoising autoencoder when a match is made between the first input data vector and one or more of the plurality of validated data vectors.