Background format optimization for enhanced queries in a distributed computing cluster
A format conversion engine for Apache Hadoop that converts data from its original format to a database-like format at certain time points for use by a low latency (LL) query engine. The format conversion engine comprises a daemon that is installed on each data node in a Hadoop cluster. The daemon comprises a scheduler and a converter. The scheduler determines when to perform the format conversion and notifies the converter when the time comes. The converter converts data on the data node from its original format to a database-like format for use by the low latency (LL) query engine.
1. A method for performing queries on stored data in a distributed computing cluster, the method comprising:
creating, by a query engine at a first data node in a plurality of data nodes forming a peer-to-peer network, a query fragment based on an availability of converted data at the first data node that interacts with other data nodes, the converted data corresponding to data associated with a query received from a client, wherein the converted data is the data associated with the query converted, by the first data node, from an original format into a target format that is specified by a schema;
causing, by the query engine at the first data node, execution of the query fragment on data that corresponds to a format for which the query fragment is created, based on the schema;
obtaining, by the query engine at the first data node, an intermediate result based on the execution of the query fragment; and
aggregating, by the query engine at the first data node for the client, the intermediate result with other intermediate results executed at query engines at the other data nodes in the peer-to-peer network.
2. The method of claim 1 , wherein the target format is a columnar format.
3. The method of claim 1 , wherein the target format is configured for relational database processing.
4. The method of claim 1 , wherein, when the converted data is available, the query fragment is created for the target format.
5. The method of claim 1 , wherein, when the converted data is not available, the query fragment is created for the original format.
6. The method of claim 1 , further comprising:
parsing the query from the client;
wherein the query fragment is created in response to parsing the query from the client.
7. The method of claim 1 , further comprising:
forming, based on the plurality of data nodes, the peer-to-peer network;
wherein each data node functions as a peer in the peer-to-peer network.
8. The method of claim 1 , further comprising:
distributing, by the first data node, the query fragment to a second data node of the plurality of data nodes.
9. The method of claim 1 , wherein the distributed computing cluster is a Hadoop™ cluster.
10. A computer system for performing queries on stored data in a distributed computing cluster, the computer system comprising:
one or more processors; and
one or more memory modules having instructions stored thereon, which when executed by the one or more processors, cause the computer system to:
create, by a query engine at a first data node in a plurality of data nodes forming a peer-to-peer network, a query fragment based on an availability of converted data at the first data node that interacts with other data nodes, the converted data corresponding to data associated with a query received from a client, wherein the converted data is the data associated with the query converted, by the first data node, from an original format into a target format that is specified by a schema;
cause, by the query engine at the first data node, execution of the query fragment on data that corresponds to a format for which the query fragment is created, based on the schema;
obtain, by the query engine at the first data node, an intermediate result based on the execution of the query fragment; and
aggregate, by the query engine at the first data node for the client, the intermediate result with other intermediate results from the other data nodes in the distributed computing cluster.
11. The computer system of claim 10 , wherein the target format is a columnar format.
12. The computer system of claim 10 , wherein the target format is configured for relational database processing.
13. The computer system of claim 10 , wherein, when the converted data is available, the query fragment is created for the target format.
14. The computer system of claim 10 , wherein, when the converted data is not available, the query fragment is created for the original format.
15. A non-transitory machine-readable storage medium having stored thereon instructions which, when executed by one or more processors, cause the one or more processors to perform a method comprising:
creating, by a query engine at a first data node in a plurality of data nodes forming a peer-to-peer network, a query fragment based on an availability of converted data at the first data node that interacts with other data nodes, the converted data corresponding to data associated with a query received from a client, wherein the converted data is the data associated with the query converted, by the first data node, from an original format into a target format that is specified by a schema;
causing, by the query engine at the first data node, execution of the query fragment on data that corresponds to a format for which the query fragment is created, based on the schema;
obtaining, by the query engine at the first data node, an intermediate result based on the execution of the query fragment; and
aggregating, by the query engine at the first data node for the client, the intermediate result with other intermediate results from the other data nodes in a distributed computing cluster.
16. The non-transitory machine-readable storage medium of claim 15 , wherein the target format is a columnar format.
17. The non-transitory machine-readable storage medium of claim 15 , wherein the target format is configured for relational database processing.
18. The non-transitory machine-readable storage medium of claim 15 , wherein, when the converted data is available, the query fragment is created for the target format.
19. The non-transitory machine-readable storage medium of claim 15 , wherein, when the converted data is not available, the query fragment is created for the original format.