Pluggable storage system for parallel query engines
A method, article of manufacture, and apparatus for managing data. In some embodiments, this includes, receiving a query from a client, based on the received query, analyzing a catalog for location information, based on the analysis, determining a first storage system, an associated first file system and an associated first protocol, using the associated first protocol to communicate with the first storage system, and performing at least a portion of the query on the first storage system.
1. A method for managing data, comprising:
receiving, by a universal namenode, a query from a client;
based at least in part on the received query, accessing a catalog service and searching a catalog provided by the catalog service for location information of one or more files responsive to the query;
based at least in part on a result of the search of the catalog, determining to move at least one of one or more files from a second storage system to a first storage system, and determining to communicate with the first storage system in connection with the one or more files responsive to the queries;
in response to a determination to move the at least one of the one or more files from the second storage system to the first storage system, moving the at least one of the one or more of files to the first storage system;
performing at least a portion of the query on the first storage system;
providing, to the client, results of the query such that in response to a determination that various portions of the results correspond to query results stored on a set of a plurality of storage systems, the results of the query are presented in a unified view across the set of the plurality of storage systems and appear, from a perspective of the client, to exist from a single namespace, wherein the plurality of storage systems comprises at least the first storage system and the second storage system; and
in response to a determination that performing the query has finished, moving the at least one of the one or more files from the first storage system to the second storage system.
2. The method of claim 1 , further comprising determining a first protocol for communication with the storage system, wherein the universal namenode communicates with the first storage system using the first protocol.
3. The method of claim 1 , wherein the catalog stores a mapping of a plurality of files stored on a plurality of storage systems to a location at which the plurality of files are respectively stored on the corresponding plurality of storage systems.
4. The method of claim 1 , wherein a first protocol for communication with the first storage system is determined, based at least in part, on a first file system associated with the first storage system.
5. The method of claim 1 , further comprising:
determining whether to move a first file that is stored on the first storage system to another storage system based at least in part on a frequency with which the first file is accessed;
in response to a determination that the first file is determined to be moved from the first storage system to the other storage system, moving the first file to the other storage system; and
updating location information associated with the first file stored in the catalog to point from a location at which the first file was stored in the first storage system to a location at which the first file is stored in the other storage system, wherein, from the perspective of the client, the a location of the first file is the same in relation to the singe namespace regardless of whether the first file is moved from the first storage system to the other storage system.
6. The method of claim 5 , wherein the updating of the location information comprises updating the mapping of the one or more files stored on the plurality of storage systems to the location at which the one or more files so as to update a record corresponding to the first file.
7. The method of claim 1 , wherein the query received from the client is a query to identify one or more files stored across a distributed storage system comprising the plurality of storage systems.
8. The method of claim 1 , wherein the determining to move the at least one of the one or more files from the second storage system to the first storage system based at least in part on the search of the catalog comprises: determining to move the at least one of the one or more files in response to the receiving the query from the client.
9. The method of claim 1 , further comprising:
tracking, by a universal job tracker of the universal namenode, a status of one or more portions of the query being respectively performed by corresponding ones of the plurality of storage systems.
10. The method of claim 1 , further comprising the first storage system reporting back the results of at least a portion of the query.
11. The method of claim 1 , further comprising adding a new storage system.
12. The method of claim 11 , further comprising determining a new protocol for communication with the new storage system.
13. The method of claim 12 , wherein a new file system associated with the new storage system is the same as a first file system associated with the first storage system.
14. The method of claim 12 , wherein a new file system associated with the new storage system is not the same as a first file system associated with the first storage system.
15. The method of claim 12 , further comprising updating the catalog based on data stored in the new storage system.
16. The method of claim 1 , further comprising determining to communicate with the second storage system in connection with the one or more files responsive to the queries, and determining a second protocol for communication with the second storage system.
17. The method of claim 1 , wherein the determining to move at least one of the one or more files is further based at least in part on information pertaining to a historical usage of files stored among the plurality of storage systems.
18. The method of claim 1 , wherein the universal namenode has one or more protocols configured to connect with the plurality of storage systems.
19. The method of claim 1 , wherein the moving the at least the one of the one or more files from the second storage system to the first storage system is transparent to the client.
20. A system for managing data, comprising one or more processors configured to:
receive a query from a client;
based at least in part on the received query, accessing a catalog service and searching a catalog provided by the catalog service for location information of one or more files responsive to the query;
based at least in part on a result of the search of the catalog, determine to move at least one of the one or more files from a second storage system to a first storage system, and determine to communicate with the first storage system in connection with the one or more files responsive to the queries;
in response to a determination to move the at least one of the one or more files from the second storage system to the first storage system, move the at least one of the one or more files to the first storage system;
perform at least a portion of the query on the first storage system;
provide, to the client, results of the query such that in response to a determination that various portions of the results correspond to query results stored on a set of a plurality of storage systems, the results of the query are presented in a unified view across the set of the plurality of storage systems and appear, from a perspective of the client, to exist from a single namespace, wherein the plurality of storage systems comprises at least the first storage system and the second storage system; and
in response to a determination that performing the query has finished, move the at least one of the one or more files from the first storage system to the second storage system.
21. A computer program product for processing data, comprising a non-transitory computer readable medium having program instructions embodied therein for:
receiving a query from a client;
based at least in part on the received query, accessing a catalog service and searching a catalog provided by the catalog service for location information of one or more files responsive to the query;
based at least in part on a result of the search of the catalog, determining to move at least one of the one or more files from a second storage system to a first storage system, and determining to communicate with the first storage system in connection with the one or more files responsive to the queries;
in response to a determination to move the at least one of the one or more files from the second storage system to the first storage system, moving the at least one of the one or more files to the first storage system;
performing at least a portion of the query on the first storage system;
providing, to the client, results of the query such that in response to a determination that various portions of the results correspond to query results stored on a set of a plurality of storage systems, the results of the query are presented in a unified view across the set of the plurality of storage systems and appear, from a perspective of the client, to exist from a single namespace, wherein the plurality of storage systems comprises at least the first storage system and the second storage system; and
in response to a determination that performing the query has finished, moving the at least one of the one or more files from the first storage system to the second storage system.