Method and system for large scale data curation
An end-to-end data curation system and the various methods used in linking, matching, and cleaning large-scale data sources. The goal of this system is to provide scalable and efficient record deduplication. The system uses a crowd of experts to train the system. The system operator can optionally provide a set of hints to reduce the number of questions sent to the experts. The system solves the problem of schema mapping and record deduplication in a holistic way by unifying these problems into a unified linkage problem.
1. A computer implemented method for integrating one or more database storage sources by performing object linkage in computer memory on object pairs from the one or more database storage sources, in order to separate the object pairs into linked object pairs and non-linked object pairs, comprising:
constructing an initial linkage model configured to generate and classify candidate object pairs from the one or more database storage sources, wherein the candidate object pairs include both schema mappings and record duplicates;
generating candidate object pairs;
classifying the candidate object pairs as linked object pairs or non-linked object pairs using the initial linkage model;
selecting a subset of the candidate object pairs using stratified sampling to send to data experts for labeling;
constructing an enhanced linkage model based on the candidate object pairs labeled by data experts;
classifying the candidate object pairs as linked object pairs or non-linked object pairs using the enhanced linkage model;
producing a global schema for an integrated data source based on the linked object pairs that are schema mappings, wherein each cluster of linked objects in schema mapping linked object pairs represents one attribute within the global schema; and
deduplicating by coalescing linked object pairs that are record duplicates, wherein each cluster of linked objects in record duplicate linked object pairs represents one record within the integrated data source.
2. The method of claim 1 , wherein each of the database storage sources are relational databases.
3. The method of claim 1 wherein the classifying is performed through a Naive Bayes classifier.