Method and system for linking heterogeneous data sources
A method for linking records (related to an entity) from separate databases may include extracting a first record from a first database as a first vector, extracting a second record from a second database as a second vector, generating first and second sub-vectors for the first and second vectors, where each sub-vector includes quality features from the respective vector, pre-processing the first and second sub-vectors using domain knowledge, calculating a distance assessment classifier based on the first and second sub-vectors, and determining whether the distance represented by the distance assessment classifier is greater than a threshold. If the distance is greater than the threshold, the records may be linked; if not, the method extracts additional records and repeats after generating first and second sub-vectors until the distance is greater than the threshold. A system for linking records is also disclosed.
1. A system for linking records from separate clinical trial databases, comprising:
an enterprise information integration subsystem for receiving data from two separate clinical trial databases;
a data cleaner for parsing and cleaning the data to generate cleansed data;
a data normalizer and labeler for normalizing and labeling the cleansed data by standardizing lexical variations and ontological concepts in the cleansed data;
a feature vector builder for building vectors that include features that map the normalized and labeled data in a finite dimensional space for comparison;
an entity classifier configured to resolve entities in the finite dimensional space using probabilistic matching;
an entity clusterer for grouping data based on similarity in the finite dimensional space;
an application programming interface to interact with the entities resolved by the entity classifier and the data grouped by the entity clusterer; and
a linked database for storing resolved entities in a harmonized schema having records linked based on the resolved entities.
2. The system of claim 1 , wherein the enterprise information integration subsystem provides a database connectivity programming interface to one or more databases.
3. The system of claim 1 , wherein the enterprise information integration subsystem supports automatic optimization of SQL queries.
4. The system of claim 1 , wherein the application programming interface (API) is a REST-ful API.
5. The system of claim 1 , wherein the probabilistic matching comprises calculating a distance between two records.
6. The system of claim 1 , wherein the enterprise information integration subsystem integrates several database schemas into one federated schema.
7. The system of claim 1 , wherein cleaning the data comprises at least one of making all letters uppercase, removing trailing spaces, removing hyphens or dashes, converting literals to numbers, and converting numbers to literals.