Method for connecting a relational data store's meta data with hadoop
A system for sharing a metadata store between a relational database and an unstructured data source is disclosed. The unstructured data source may comprise a Hadoop system with a Hadoop Distributed Files System.
1. A method, comprising:
receiving, by one or more processors, a query;
determining, by the one or more processors, that information to be used in connection with performing the query is stored at one or more of a plurality of data sources, including a first data source and a second data source,
wherein a shared catalog service manages metadata pertaining to the first data source and the second data source,
wherein the first data source comprises unstructured data, and the second data source comprises structured data; and
generating a response to the query using the unstructured data of the first data source and the structured data of the second data source based on database schemas obtained from the shared catalog service,
wherein the first data source is in communication with the shared catalog service via an adaptor that receives one or more calls for database schema, the one or more calls being made using a protocol that is native to the first data source, and the adaptor converts the one or more calls to a protocol associated with the shared catalog service.
2. The method of claim 1 , wherein the unstructured data is stored in a distributed file system.
3. The method of claim 1 , wherein the second data source comprises a relational database.
4. The method of claim 1 , wherein the metadata pertaining to the first data source and the second data source comprises corresponding database schemas.
5. The method of claim 4 , wherein the database schema comprises database table definitions.
6. The method of claim 1 , wherein the first data source applies a first database schema to unstructured data, the first database schema being obtained from the shared catalog service.
7. The method of claim 1 , wherein schemas for the first data source and the second data source are stored in a relational database.
8. The method of claim 1 , wherein in the event that a first database schema and a second database schema are the same and that the first database schema is applied to the unstructured data of the first data source and the second database schema is applied to the structured data of the second data source, a client device connected to a system comprising the first data source, the second data source, and the shared catalog service can interface with the unstructured data of the first data source and the structured data of the second data source.
9. The method of claim 1 , wherein a third data source communicates with the shared catalog service to retrieve a third database schema corresponding to a configuration of data stored at the third data source from among a plurality of database schemas.
10. The method of claim 1 , wherein application of a first database schema to the unstructured data comprised in the first data source enables a Relationship Database Management System (RDBMS) command to be run against the unstructured data.
11. The method of claim 1 , wherein the shared catalog service accesses the database schema using a Relationship Database Management System (RDBMS) query.
12. The method of claim 1 , wherein the first data source retrieves a first database schema based on querying the shared catalog service for the first database schema, and the second data source retrieves a second database schema based at least in part on querying the shared catalog service for the second database schema.
13. The method of claim 12 , wherein the generating the response to the query using the unstructured data of the first data source and the structured data of the second data source comprises performing a JOIN operation between the unstructured data and the structured data and returning a result of the JOIN operation.
14. The method of claim 1 , wherein the adapter establishes a Java Database Connectivity (JDBC) connection with the shared catalog service.
15. A system, comprising:
a shared catalog service configured to store database metadata and to provide catalog service to one or more data sources;
a first data source, of the one or more data sources, with access to the shared catalog service, wherein the first data source stores unstructured data; and
a second data source, of the one or more data sources, with access to the shared catalog service, wherein the second data source stores structured data,
wherein the shared catalog service manages metadata pertaining to the first data source and the second data source, and
wherein in response to receiving a query, information to be used in connection with performing the query is determined to correspond to information stored at one or more of the first data source and the second data source, and a response to the query is generated using the unstructured data of the first data source and the structured data of the second data source based on database schemas obtained from the shared catalog service,
wherein the first data source is in communication with the shared catalog service via an adaptor that receives one or more calls for database schema, the one or more calls being made using a protocol that is native to the first data source, and the adaptor converts the one or more calls to a protocol associated with the shared catalog service.
16. The system of claim 15 , wherein the adapter establishes a Java Database Connectivity (JDBC) connection with the shared catalog service.
17. A computer program product, comprising a non-transitory computer readable medium having program instructions embodied therein for:
receiving, by one or more processors, a query;
determining, by the one or more processors, that information to be used in connection with performing the query is stored at one or more of a plurality of data sources, including a first data source and a second data source,
wherein a shared catalog service manages metadata pertaining to the first data source and the second data source,
wherein the first data source comprises unstructured data, and the second data source comprises structured data; and
generating a response to the query using the unstructured data of the first data source and the structured data of the second data source based on database schemas obtained from the shared catalog service,
wherein the first data source is in communication with the shared catalog service via an adaptor that receives one or more calls for database schema, the one or more calls being made using a protocol that is native to the first data source, and the adaptor converts the one or more calls to a protocol associated with the shared catalog service.
18. The computer program product of claim 17 , wherein the adapter establishes a Java Database Connectivity (JDBC) connection with the shared catalog service.