System and method for comparing heterogeneous data sources
The present invention is a computer-implemented system and method that allows data in different databases to be shared without requiring the data to be remodeled to fit an existing data convention. The invention allows the comparison of two database structures, and facilitates the searching of information from one database to other databases, or from an external set of search criterion against a plurality of databases. The invention allows information to be exchanged from heterogeneous data sources, or database query sources without having to alter the structure of the data sources that are being searched. A number of techniques may be used to do this comparison and facilitate the cross database searching. Tree comparison methods, user defined mapping methods, the use of similarity comparisons to determine similar database structures and data are used to facilitate the cross database searching.
1. A method for matching data contained in a source data structure to data contained in a target data structure, comprising:
selecting a set of one or more comparison methods;
comparing each node of the source data structure with each node in the target data structure using the selected comparison methods;
determining a measure of similarity between each node of the source data structure and each node of the target data structure;
automatically mapping data from a node of the source data structure to a node of the target data structure if the measure of similarity between the source data structure node and the target data structure node exceed a predetermined threshold value; and
notifying a user of the mapped data results.
2. The method of claim 1 , wherein the one or more comparison methods are selected from the group consisting of exact string match, similarity string comparison, data type lineage and inheritance, similar child structure and synonym table lookup.
3. The method of claim 1 , wherein each data node comprises an element name, an element data type attribute, and an attribute description value.
4. The method of claim 1 , further comprising a strategy list whereby selected comparison methods are assigned to each data node element name, each data node element attribute value.
5. The method of claim 4 , wherein the data node element attribute value is selected form the group consisting of attribute data type value and attribute description value.
6. The method of claim 1 , wherein the measure of similarity is based on a percentage value of similarity.
7. The method of claim 1 , further comprising manually defining a mapping between selected nodes of the source and target data structures prior to the steps of selecting, comparing, and determining.
8. The method of claim 1 , further comprising manually defining a mapping between selected nodes of the source and target data structures after the automatically mapping step.
9. The method of claim 1 , wherein the mapping process comprises storing data from the source data structure into the target data structure.
10. The method of claim 1 , wherein the mapping process comprises storing indices of mapped data for linking data between the source data structure and the target data structure.
11. The method of claim 1 , wherein the automatic mapping step is selected from the group consisting of many to one element data transformation, one to many element data transformation where a number of source tokens equals a number of target elements, one to many element data transformation where a number of source tokens id greater than a number of target elements, one to many element data transformation where a number of source tokens is less than a number of target elements, many to many element data transformation where a number of source elements equals a number of target elements, many to many element data transformation where a number of source elements is less than a number of target elements, and many to many element data transformation where a number of source elements is greater than a number of target elements.
12. The method of claim 1 , further comprising submitting and executing a search request in the target data structure based on elements in the source data structure.
13. The method of claim 11 , further comprising returning a search result containing data indices and data.
14. The method of claim 1 , further comprising representing each node in a data structure in a language selected from the group consisting of HTML, XML, and SGML.
15. The method of claim 1 , further comprising selecting another set of comparison methods and recursively repeating the steps of comparing and determining a measure of similarity.
16. A system for matching data contained in a source data structure to data contained in a target data structure, comprising:
a strategy list for selecting a set of one or more comparison methods;
means for comparing each node of the source data structure with each node in the target data structure using the selected comparison methods;
means for determining a measure of similarity between each node of the source data structure and each node of the target data structure;
a search engine component for automatically mapping data from a node of the source data structure to a node in the target data structure if the measure of similarity between the source data structure node and the target data structure node exceed a predetermined threshold value;
indices for designating a mapping between similar nodes of the source and target data structure; and
means for notifying a user of the mapped data results.
17. The system of claim 16 , wherein a search request from a user application designates the set of one or more comparison methods.
18. The system of claim 16 , wherein the means for comparing nodes of the source and target data structure is selected from the group consisting of exact string match, similarity string comparison, data type lineage and inheritance, similar child structure and synonym table lookup.
19. The system of claim 16 , wherein each node of the source and target data structures comprises an element name, an element data type attribute and an attribute description value.
20. The system of claim 16 , wherein the means for comparing nodes and determining a measure of similarity between nodes of the source and target data structure comprises a search engine component.
21. The system of claim 16 , further comprising search index databases for storing the mapping indices.
22. The system of claim 16 , further comprising database management systems for storing the target and source databases.
23. The system of claim 16 , further comprising a data gateway component for accepting a search request from a user application, issuing search commands to a search engine component and a data management component, and sending a result set to the requesting user application.
24. The system of claim 16 , wherein the strategy list comprises a matrix of comparison methods for each data structure node comprising element name, attribute data type value, and attribute description value.
25. The system of claim 16 , wherein the mapping comprises source node data being added to target node data.
26. The system of claim 16 , further comprising means for enabling a user to manually enter mapping data.
27. A computer-readable media containing instructions for controlling a computer system to implement the method of claim 1 .
28. A computer-readable media containing instructions for controlling a computer system to implement the method of claim 15 .