Systems and methods for record linkage and paraphrase generation using surrogate learning
A method of using unlabeled data to train a classifier is disclosed. In one embodiment related to record linkage, the method entails retrieving a set of candidate data records from a master database based on a least one update record. Next, a surrogate learning technique is used to identify one of the candidate data records as a match for the one update record. Lastly, the exemplary method links or merges the update record and the identified one of the candidate data records.
1. A method of using a processor and a memory for classifying data associated with a feature space X to a set of classes y={0,1}, wherein features defining the feature space X are partitioned into X=X 1 ×X 2 , a random feature vector xεX is denoted correspondingly as x=(x 1 , x 2 ), and feature x 1 is a binary random variable, the method comprising:
estimating P(x 1 |x 2 ) from a set of unlabeled data;
estimating P(x 1 =0|x 2 ) from a set of labeled data;
determining whether to classify a portion of the data to y=0 or y=1 based on the estimated P(x 1 =0|x 2 ); and
logically associating the portion of the data in the memory with the class y=0 or the class y=1 based on the determination.
2. The method of claim 1 , wherein the unlabeled data is a set of identity records, and the binary random variable indicates whether a field of one of the identity records matches a corresponding field in another identity record.
3. The method of claim 1 , wherein the feature x 1 is a total recall feature for the class y=1.
4. The method of claim 1 , wherein P(x 1 ,x 2 |y=0)=P(x 1 |y=0) P(x 2 |y=0).
5. The method of claim 1 , wherein the data is a segment of text from a new article.
6. A system having a processor and a memory for classifying data associated with a feature space X to a set of classes y={0,1}, wherein features defining the feature space X are partitioned into X=X 1 ×X 2 , a random feature vector xεX is denoted correspondingly as x=(x 1 , x 2 ), and feature x 1 is a binary random variable, the system further comprising:
means for estimating P(x 1 |x 2 ) from a set of unlabeled data;
means for estimating P(x 1 =0|x 2 ) from a set of labeled data;
means for determining whether to classify a portion of the data to y=0 or y=1 based on the estimated P(x 1 =0\x 2 ); and
means, responsive to the determination, for logically associating the portion of the data in the memory with the class y=0 or the class y=1.
7. The system of claim 6 , wherein the unlabeled data is a set of identity records, and the binary random variable indicates whether a field of one of the identity records matches a corresponding field in another identity record.
8. The system of claim 6 , wherein the feature x 1 is a total recall feature for the class y=1.
9. The system of claim 6 , wherein P(x 1 ,x 2 |y=0)=P(x 1 |y=0) P(x 2 |y=0).
10. A method of using a processor and a memory for linking or merging update records with a master database of data records, the method comprising:
retrieving a set of candidate data records from the master database based on a least one update record;
using surrogate learning to identify one of the candidate data records as a match for the one update record; and
linking or merging the update record and the identified one of the candidate data records.