Systems and methods for self-service data structure discovery
Systems and methods for self-service data structure discovery are disclosed. In one embodiment, a method may include: (1) retrieving, by a data crawler, data from one or more data sources; (2) storing, by the data crawler, the data to a data catalog; (3) receiving, at a search dashboard, a query from a user, the query comprising a search field; (4) submitting, by the search dashboard and using a query tool, the query to a data catalog, wherein the query tool searches the search field by data structure, datatype, field journey, primary key details, and/or multi-value column detail across layers, data lineage controls in the data catalog; (5) receiving, at the search dashboard, a result of the query from the data catalog; and (6) presenting, by the search dashboard, the results.
1. A method, comprising:
automatically crawling, by a data crawler, a plurality of data sources comprising a schema registry, a database, and an API store, wherein the crawling identifies data in each of the plurality of data sources, and wherein the crawling is performed periodically;
pulling, by the data crawler, data from the plurality of data sources using a plugin for each of the plurality of data sources that provides an interface for REST API calls;
storing, by the data crawler, the data to a data catalog;
receiving, at a search dashboard, a query from a user, the query comprising a search field;
submitting, by the search dashboard and using a query tool, the query to a data catalog, wherein the query tool searches the search field by data structure, datatype, field journey, primary key details, and/or multi-value column detail across layers, data lineage controls in the data catalog;
receiving, at the search dashboard, a result of the query from the data catalog; and
presenting, by the search dashboard, the results.
2. The method of claim 1 , wherein the data comprises at least one of event schema, database schema details, database tables, metadata, and API specifications.
3. The method of claim 2 , further comprising:
identifying, by the data crawler or the query tool, data naming conventions based on relationships among the event schema, the database schema details, and the API specifications.
4. The method of claim 2 , further comprising:
identifying, by the data crawler or the query tool, data type inconsistencies based on relationships among the event schema, the database schema details, and the API specifications.
5. The method of claim 1 , wherein the query comprises at least an application identifier, an event identifier, an API identifier, and a data identifier.
6. A system, comprising:
a plurality of data sources comprising a schema registry, a database, and an API store;
a data crawler computer program executed by a computer processor that is configured to automatically craw the plurality of data sources, to pull data from the plurality of data sources using a plugin for each of the plurality of data sources that provides an interface for REST API calls and to store the data to a data catalog, wherein the crawling identifies data in each of the plurality of data sources, and wherein the crawling is performed periodically; and
automatically crawling, by a data crawler, a plurality of data sources comprising a schema registry, a database, and an API store;
an electronic device executing a search dashboard computer program and a query tool computer program, wherein the search dashboard computer program is configured to receive a query from a user, the query comprising a search field, to submit, using the query tool computer program, the query to the data catalog, wherein the query tool computer program is configured to search the search field by data structure, datatype, field journey, primary key details, and/or multi-value column detail across layers, data lineage controls in the data catalog, to receive a result of the query from the data catalog, and to present the result.
7. The system of claim 6 , wherein the data comprises at least one of event schema, database schema details, database tables, metadata, and API specifications.
8. The system of claim 7 , wherein the data crawler computer program or the query tool computer program is further configured to identify data naming conventions based on relationships among the event schema, the database schema details, and the API specifications.
9. The system of claim 7 , wherein the data crawler computer program or the query tool computer program is further configured to identify data type inconsistencies based on relationships among the event schema, the database schema details, and the API specifications.
10. The system of claim 6 , wherein the query comprises at least an application identifier, an event identifier, an API identifier, and a data identifier.
11. A non-transitory computer readable storage medium, including instructions stored thereon, which when read and executed by one or more computer processors, cause the one or more computer processors to perform steps comprising:
automatically crawling a plurality of data sources comprising a schema registry, a database, and an API store, wherein the crawling identifies data in each of the plurality of data sources, and wherein the crawling is performed periodically;
pulling data from the plurality of data sources using a plugin for each of the plurality of data sources that provides an interface for REST API calls;
storing the data to a data catalog;
receiving a query from a user, the query comprising a search field, wherein the query comprises at least an application identifier, an event identifier, an API identifier, and a data identifier;
submitting the query to a data catalog;
searching the data catalog for the search field by data structure, datatype, field journey, primary key details, and/or multi-value column detail across layers, data lineage controls;
receiving a result of the query from the data catalog; and
presenting the results.
12. The non-transitory computer readable storage medium of claim 11 , wherein the data comprises at least one of event schema, database schema details, database tables, metadata, and API specifications.
13. The non-transitory computer readable storage medium of claim 12 , further including instructions stored thereon, which when read and executed by one or more computer processors, cause the one or more computer processors to perform steps comprising identifying data naming conventions based on relationships among the event schema, the database schema details, and the API specifications.
14. The non-transitory computer readable storage medium of claim 12 , further including instructions stored thereon, which when read and executed by one or more computer processors, cause the one or more computer processors to perform steps comprising identifying data type inconsistencies based on relationships among the event schema, the database schema details, and the API specifications.