System and method for ontology induction through statistical profiling and reference schema matching
In accordance with various embodiments, described herein is a system (Data Artificial Intelligence system, Data AI system), for use with a data integration or other computing environment, that leverages machine learning (ML, DataFlow Machine Learning, DFML), for use in managing a flow of data (dataflow, DF), and building complex dataflow software applications (dataflow applications, pipelines). In accordance with an embodiment, the system can perform an ontology analysis of a schema definition, to determine the types of data, and datasets or entities, associated with that schema; and generate, or update, a model from a reference schema that includes an ontology defined based on relationships between datasets or entities, and their attributes. A reference HUB including one or more schemas can be used to analyze data flows, and further classify or make recommendations such as, for example, transformations enrichments, filtering, or cross-entity data fusion of an input data.
1. A method for use with a data integration or other computing environment comprising:
receiving input defining one or more schemas;
accessing the one or more schemas to obtain one or more entity definitions associated with entities provided by the reference of one or more schemas;
generating a sample data for the one or more entities from the one or more schemas;
profiling the sample data to determine one or more metrics associated with the sample data;
generating one or more rules based on the entity definitions; and
generating a functional type system based on the generated one or more rules, for use in processing a data input.
2. The method of claim 1 , wherein the one or more rules includes data rules that are defined in terms of profiled data attribute, or composite value metrics; relationship rules that define associations across entities and attribute vectors; and complex rules that can be derived through a combination of data and relationship rules.
3. The method of claim 1 , wherein the one or more schemas are provided in a reference HUB.
4. The method of claim 1 , wherein the functional type system is persisted to a knowledge source.
5. The method of claim 4 , wherein the knowledge source is a system HUB.
6. The method of claim 1 , wherein the one or more schemas operate as a reference ontology, for use in type-tagging, comparing, classifying, or otherwise evaluating a metadata schema or ontology provided by registered HUBs.
7. The method of claim 1 , wherein the method is performed in a cloud or cloud-based computing environment.
8. A system for ontology analysis of a schema definition for use with a data integration or other computing environment, comprising:
one or more processors operable to:
receiving input defining one or more schemas;
accessing the one or more schemas to obtain one or more entity definitions associated with entities provided by the reference of one or more schemas;
generating a sample data for the one or more entities from the one or more schemas;
profiling the sample data to determine one or more metrics associated with the sample data;
generating one or more rules based on the entity definitions; and
generating a functional type system based on the generated one or more rules, for use in processing a data input.
9. The system of claim 8 , wherein the one or more rules includes data rules that are defined in terms of profiled data attribute, or composite value metrics; relationship rules that define associations across entities and attribute vectors; and complex rules that can be derived through a combination of data and relationship rules.
10. The system of claim 8 , wherein the one or more schemas are provided in a reference HUB.
11. The system of claim 8 , wherein the functional type system is persisted to a knowledge source.
12. The system of claim 11 , wherein the knowledge source is a system HUB.
13. The system of claim 8 , wherein the one or more schemas operate as a reference ontology, for use in type-tagging, comparing, classifying, or otherwise evaluating a metadata schema or ontology provided by registered HUBs.
14. The system of claim 8 , wherein the system is provided in a cloud or cloud-based computing environment.
15. A non-transitory computer readable storage medium, including instructions stored thereon which when read and executed by one or more computers cause the one or more computers to perform a method comprising:
receiving input defining one or more schemas;
accessing the one or more schemas to obtain one or more entity definitions associated with entities provided by the reference of one or more schemas;
generating a sample data for the one or more entities from the one or more schemas;
profiling the sample data to determine one or more metrics associated with the sample data;
generating one or more rules based on the entity definitions; and
generating a functional type system based on the generated one or more rules, for use in processing a data input.
16. The non-transitory computer readable storage medium of claim 15 , wherein the one or more rules includes data rules that are defined in terms of profiled data attribute, or composite value metrics; relationship rules that define associations across entities and attribute vectors; and complex rules that can be derived through a combination of data and relationship rules.
17. The non-transitory computer readable storage medium of claim 15 , wherein the one or more schemas are provided in a reference HUB.
18. The non-transitory computer readable storage medium of claim 15 , wherein the functional type system is persisted to a knowledge source.
19. The non-transitory computer readable storage medium of claim 18 , wherein the knowledge source is a system HUB.
20. The non-transitory computer readable storage medium of claim 15 , wherein the one or more schemas operate as a reference ontology, for use in type-tagging, comparing, classifying, or otherwise evaluating a metadata schema or ontology provided by registered HUBs.