Method for large-scale data schema analysis and quality assurance
View Patent ↗A system and method for large-scale data schema analysis and quality assurance is disclosed. A data schema may be received and compiled into an internal representation for analysis. The schema's information may also be exposed via an application programming interface. Using the schema's application programming interface along with a provided requirement, the schema may be analyzed and/or corrected for quality, consistency and interoperability.
1. A method for large-scale data schema analysis and quality assurance comprising:
receiving a data schema;
exposing schema information including schema fields of the received data schema via an application programming interface;
receiving a user-defined requirement specifying an analysis to perform on the received data schema, wherein receiving the user-defined requirement comprises:
receiving a user-written code indicating a requirement specification; and
compiling the requirement specification from the user-written code into an executable binary;
providing at least one additional analysis application programming interface required to perform the schema analysis specified by the user-defined requirement; and
using the schema's application programming interface along with the provided at least one additional analysis application programming interface to analyze the data schema, wherein the additional analysis application programming interface provides methods that analyze the data schema by traversing the exposed schema fields or locating changes within the exposed schema fields.
2. The method of claim 1 further comprising logging information related to analyzing the schema.
3. A system for large-scale data schema analysis and quality assurance comprising:
one or more processing devices and
one or more storage devices storing instructions that, when executed by the one or more processing devices, cause the one or more processing devices to:
receive a data schema;
expose schema information including schema fields of the received data schema via an application programming interface;
receive a user-defined requirement specifying an analysis to perform on the received data schema, wherein the one or more processing devices receive the user-defined requirement by:
receiving a user-written code indicating a requirement specification; and
compiling the requirement specification from the user-written code into an executable binary;
provide at least one additional analysis application programming interface interfaces for required to perform the schema analysis specified by the user-defined requirement; and
use the schema's application programming interface along with the provided additional analysis application programming interface interfaces to analyze the data schema,
wherein the additional analysis application programming interface provides methods that analyze the data schema by traversing the exposed schema fields or locating changes within the exposed schema fields.
4. The system of claim 3 further comprising logging information related to analyzing the schema.
5. The system of claim 3 further comprising rewriting the schema in response to the results of the schema analysis.