Accessing electronic databases
Examples disclosed herein relate to accessing electronic databases. Some examples disclosed herein may include partitioning a computation task into subtasks. A processing node of a computation engine may generate a database query for retrieving an electronic data segment associated with at least one of the subtasks from a database. The database query may include pre-processing instructions for a database management system (DBMS) associated with the database to pre-process the electronic data segment before providing the electronic data segment to the processing node. The pre-processing instructions may include at least one of: filtering, projection, join, aggregation, count, and user-defined instructions. The generated query may be provided to the DBMS.
1. A method for accessing electronic databases, the method comprising:
partitioning, by a processor of a computation engine, a computation task into a plurality of subtasks to be processed by a plurality of processing nodes of the computation engine;
generating, by each processing node of the plurality of processing nodes that are to process a subtask of the plurality of subtasks, a database query for retrieving an electronic data segment associated with the subtask from a database node of a plurality of database nodes of a database system, wherein the database query generated by each processing node of the plurality of processing nodes includes instructions that instruct a database management system (DBMS) associated with the database system to:
retrieve the electronic data segment from the database node,
perform a specific pre-processing operation, including one of a filtering operation, a projection operation, a join operation, an aggregation operation, or a count operation, on the retrieved electronic data segment and
after performing the specific pre-processing operation on the retrieved electronic data segment, transmit the retrieved electronic data segment to the processing node that generated the database query; and
transmitting, by each processing node of the plurality of processing nodes, the generated database query to the DBMS via a connection.
2. The method of claim 1 , wherein the instructions in the database query include user-defined instructions.
3. The method of claim 1 , wherein the instructions in the database query are implemented using Structured Query Language (SQL) programming commands.
4. The method of claim 1 , wherein transmitting the generated database query to the DBMS includes:
transmitting, by each of the plurality of the processing nodes, the generated database query to the DBMS via processing pipelines.
5. The method of claim 4 , wherein the processing pipelines are connected via an application programming interface (API).
6. The method of claim 1 , further comprising:
obtaining, from the DBMS, metadata associated with the database system, wherein the metadata includes an Internet Protocol (IP) address of the database node storing the electronic data segment and
causing each processing node of the plurality of processing nodes to generate the database query based on the metadata.
7. A system, comprising:
a processor to:
receive a computation task; and
partition the computation task into a plurality of subtasks to be processed by a plurality of processing nodes of a computation engine; and
the plurality of processing nodes that are to process the plurality of subtasks to:
generate database queries for retrieving electronic data segments associated with respective ones of the plurality of subtasks from a plurality of database nodes of a database system,
wherein each database query of the database queries generated by one processing node of the plurality of processing nodes includes instructions that instruct a database management system (DBMS) associated with the database system to:
retrieve an electronic data segment of the electronic data segments from a database node of the plurality of database nodes,
perform a specific pre-processing operation, including one of a filtering operation, a projection operation, a join operation, an aggregation operation, or a count operation, on the retrieved electronic data segment; and
after performing the specific pre-processing operation on the retrieved electronic data segment, transmit the retrieved electronic data segment to the processing node that generated the database query; and
provide the generated database queries to the DBMS via connections in parallel.
8. The system of claim 7 , further comprising:
the DBMS, wherein the DBMS is to:
receive the database queries from the plurality of processing nodes;
retrieve the electronic data segments associated with the plurality of subtasks from the plurality of database nodes;
perform the specific pre-processing operation on the retrieved electronic data segments according to the instructions in the database queries; and
after performing the specific pre-processing operation on the electronic data segments, transmit the retrieved electronic data segments to the plurality of processing nodes via processing pipelines of the plurality of processing nodes in the computation engine and processing pipelines of the plurality of database nodes in the database system.
9. The system of claim 8 , wherein:
the computation engine is to establish connections between the processing pipelines of the plurality of processing nodes and the processing pipelines of the plurality of database nodes; and
the DBMS is to transmit the retrieved electronic data segments from each of the plurality of database nodes to an associated one of the plurality of processing nodes via the established connections.
10. The system of claim 9 , wherein, after partitioning the computation task into the plurality of subtasks, the processor assigns the plurality of subtasks to each processing node of the plurality of processing nodes.
11. The system of claim 7 , wherein, after partitioning the computation task into the plurality of subtasks, the processor:
obtains, from the DBMS, metadata associated with the database system, wherein the metadata includes Internet Protocol (IP) addresses of the plurality of database nodes storing the electronic data segments corresponding to the computation task; and
causes the plurality of processing nodes to generate the database queries based on the metadata.
12. The system of claim 7 , wherein the database queries for retrieving the electronic data segments from the plurality of database nodes include instructions to retrieve the electronic data segments stored entirely on a single database node among the plurality of database nodes.
13. The system of claim 7 , wherein the instructions in each database query of the database queries are implemented using a database query programming language.
14. The system of claim 7 , wherein the computation engine receives the computation task via a Java Database Connectivity (JDBC) connector or an Open Database Connectivity (ODBC) connector.
15. A non-transitory machine-readable storage medium storing instructions that, when executed by a processor, cause the processor to:
partition a computation task into a plurality of subtasks to be processed by a plurality of processing nodes of a computation engine;
assign the plurality of subtasks to the plurality of processing nodes; and
cause each processing node of the plurality of processing nodes that process a subtask of the plurality of subtasks to:
generate a database query for retrieving an electronic data segment associated with the subtask from a database node of a plurality of database nodes of a database system, wherein the database query generated by the processing node includes instructions that instruct a database management system (DBMS) associated with the database system to:
retrieve the electronic data segment from the database node,
perform a specific pre-processing operation, including one of a filtering operation, a projection operation, a join operation, an aggregation operation, or a count operation, on the retrieved electronic data segment, and
after performing the specific pre-processing operation on the retrieved electronic data segment, transmit the retrieved electronic data segment to the processing node that generated the database query;
transmit the database query to the DBMS via a connection; and
receive, from the DBMS, the retrieved electronic data segment associated with the subtask, wherein the specific pre-processing operation in the database query was performed on the retrieved electronic data segment by the DBMS.
16. The non-transitory machine-readable storage medium of claim 15 , wherein the instructions, when executed by the processor, cause the processor to:
obtain, from the DBMS, metadata associated with the database system in response to partitioning the computation task into the plurality of subtasks; and
cause each processing node of the plurality of processing nodes to generate the database query based on the metadata.
17. The non-transitory machine-readable storage medium of claim 16 , wherein the metadata includes hash values for boundaries of the plurality of database nodes storing electronic data segments corresponding to the subtasks.
18. The non-transitory machine-readable storage medium of claim 16 , wherein the metadata includes Internet Protocol (IP) addresses of the plurality of database nodes storing electronic data segments corresponding to the computation task.
19. The non-transitory machine-readable storage medium of claim 15 , wherein the connection is implemented as a Java database connectivity (JDBC) connection.
20. The non-transitory machine-readable storage medium of claim 15 , wherein the instructions in the database query include user-defined instructions.