Scalable binning for big data deduplication
Fast record deduplication is accomplished by providing as an input, data records having multiple attributes, and local similarity functions of individual attributes with local similarity thresholds. Bin IDs are then generated based on the local similarity functions and the local similarity thresholds. The Bin IDs are unique identifiers of a respective bin of records, and the bin of records is a set of records that are possibly pairwise similar. Local candidate pairs are identified based on data records that share Bin IDs. The local candidate pairs are aggregated to produce a set of global candidate pairs. The set of global candidate pairs are filtered by deciding whether a pair of data records represents a duplicate.
1. A computer implemented method for use in fast record deduplication comprising:
(a) inputting, into software running on one or more computer processors, data records having multiple attributes;
(b) inputting, into the software, local similarity functions of individual attributes with local similarity thresholds;
(c) generating, by the software, Bin IDs based on the local similarity functions and the local similarity thresholds, wherein the Bin IDs are unique identifiers of a respective bin of records, and wherein all similar record pairs appear in the same bin;
(d) identifying local candidate pairs based on data records that share Bin IDs;
(e) aggregating, by the software, the local candidate pairs to produce a set of global candidate pairs; and
(f) filtering, by the software, the set of global candidate pairs by deciding whether a pair of data records represents a duplicate.
2. The method of claim 1 wherein the candidate pairs are identified by a Cartesian product of all data records sharing Bin IDs.
3. A system for performing fast record deduplication comprising at least one non-transitory computer-readable medium containing computer program instructions that when executed by at least one computer processor causes the at least one computer processor to perform the steps of:
(a) inputting data records having multiple attributes;
(b) inputting local similarity functions of individual attributes with local similarity thresholds;
(c) generating Bin IDs based on the local similarity functions and the local similarity thresholds, wherein the Bin IDs are unique identifiers of a respective bin of records, and wherein all similar record pairs appear in the same bin;
(d) identifying local candidate pairs based on data records that share Bin IDs;
(e) aggregating the local candidate pairs to produce a set of global candidate pairs; and
(f) filtering the set of global candidate pairs by deciding whether a pair of data records represents a duplicate.
4. The system of claim 3 wherein the candidate pairs are identified by a Cartesian product of all data records sharing Bin IDs.