SCHEMA ABSTRACTION IN DATA ECOSYSTEMS
The disclosed embodiments provide a system for performing data management. During operation, the system obtains a first schema with a first syntax for describing a first data set and a second schema with a second syntax for describing a second data set. Next, the system converts the first schema into a first standardized form with a standardized syntax and the second schema into a second standardized form with the standardized syntax. The system then outputs the first and second standardized forms for use in accessing the first and second data sets.
1 . A method, comprising:
obtaining a first schema with a first syntax for describing a first data set and a second schema with a second syntax for describing a second data set;
converting, by a computer system, the first schema into a first standardized form with a standardized syntax and the second schema into a second standardized form with the standardized syntax; and
outputting the first and second standardized forms for use in accessing the first and second data sets.
2 . The method of claim 1 , further comprising:
converting a set of data types in the first schema into a set of abstract types; and
including the abstract types in the first standardized form.
3 . The method of claim 2 , wherein converting the set of data types into the set of abstract types comprises at least one of:
converting a primitive type in the first schema into a generic type; and
including a use case associated with a data type in a corresponding abstract type based on a data pattern associated with the data type.
4 . The method of claim 3 , wherein the use case comprises at least one of:
a domain-specific type; and
a system-specific type.
5 . The method of claim 1 , further comprising:
generating a comparison of the first and second schemas using the first and second standardized forms.
6 . The method of claim 1 , further comprising:
outputting the first and second standardized forms in response to a query comprising a term that is common to the first and second schemas.
7 . The method of claim 1 , further comprising:
converting the first standardized form into a third schema with a third syntax for describing the first data set.
8 . The method of claim 1 , further comprising:
including, in the first standardized form, a profiling attribute for use in data profiling of the first data set.
9 . The method of claim 1 , wherein the first standardized form stores a structure associated with the first schema in a flattened structure comprising a set of field names and a set of paths associated with the field names.
10 . The method of claim 1 , wherein the first standardized form stores a structure associated with the first schema in a standardized nested structure.
11 . The method of claim 1 , wherein the first and second schemas are obtained from at least one of:
a relational database;
a graph database;
an in-memory database;
a data warehouse;
a distributed data store;
an application;
an analytics platform;
a machine learning platform; and
a runtime environment.
12 . An apparatus, comprising:
one or more processors; and
memory storing instructions that, when executed by the one or more processors, cause the apparatus to:
obtain a first schema with a first syntax for describing a first data set and a second schema with a second syntax for describing a second data set;
convert the first schema into a first standardized form with a standardized syntax and the second schema into a second standardized form with the standardized syntax; and
output the first and second standardized forms for use in accessing the first and second data sets.
13 . The apparatus of claim 12 , wherein the memory further stores instructions that, when executed by the one or more processors, cause the apparatus to:
convert a set of data types in the first schema into a set of abstract types; and
include the abstract types in the first standardized form.
14 . The apparatus of claim 12 , wherein converting the set of data types into the set of abstract types comprises at least one of:
converting a primitive type in the first schema into a generic type; and
including a use case associated with a data type in a corresponding abstract type based on a data pattern associated with the data type.
15 . The apparatus of claim 14 , wherein the use case comprises at least one of:
a domain-specific type; and
a system-specific type.
16 . The apparatus of claim 12 , wherein the memory further stores instructions that, when executed by the one or more processors, cause the apparatus to:
output the first and second standardized forms in response to a query comprising a term that is common to the first and second schemas.
17 . The apparatus of claim 12 , wherein the memory further stores instructions that, when executed by the one or more processors, cause the apparatus to:
generate a comparison of the first and second schemas using the first and second standardized forms.
18 . The apparatus of claim 12 , wherein the first standardized form stores a structure associated with the first schema in a flattened structure comprising a set of field names and a set of paths associated with the field names.
19 . A system, comprising:
an processing module comprising a non-transitory computer-readable medium comprising instructions that, when executed, cause the system to:
obtain a first schema with a first syntax for describing a first data set and a second schema with a second syntax for describing a second data set; and
convert the first schema into a first standardized form with a standardized syntax and the second schema into a second standardized form with the standardized syntax; and
a presentation module comprising a non-transitory computer-readable medium comprising instructions that, when executed, cause the system to output the first and second standardized forms for use in accessing the first and second data sets.
20 . The system of claim 19 , wherein the non-transitory computer-readable medium of the processing module further comprises instructions that, when executed, cause the system to:
convert a set of data types in the first schema into a set of abstract types; and
include the abstract types in the first standardized form.