Complex computation across heterogenous computer systems
A programmatic interface to allow external functions to be registered and called in parallel from within a database management system is described for coordinating a computation at multiple nodes. In one embodiment, each node executes a process configured for starting a program to perform the computation in response to a command received from a database system. In response to receiving a query at the database system, multiple commands are transmitted to the processes for concurrently performing the computation at each said corresponding process. Results are received from each of the processes and execution of the statement is completed based on the results received.
1. A method comprising performing a machine-executed operation involving instructions, wherein the machine-executed operation is at least one of:
A) sending said instructions over transmission media;
B) receiving said instructions over transmission media;
C) storing said instructions onto a machine-readable storage medium; and
D) executing the instructions;
wherein said instructions are instructions which, when executed by one or more processors, cause the one or more processors to perform the steps of:
at each node of a plurality of nodes, executing a corresponding process; and
at a database system, receiving a statement specifying an external routine for performing a computation and, in response to receiving the statement:
concurrently transmitting a plurality of commands for performing the computation to each said corresponding process;
receiving results from each said corresponding process; and
completing processing of the statement based on the results received from each said corresponding process.
2. The method of claim 1 , wherein the steps of concurrently transmitting, receiving results, and completing processing are performed by said database system.
3. The method of claim 1 , wherein at least one of said plurality of nodes is implemented using a different type of hardware, operation system software, or application software than said database system.
4. The method of claim 1 , wherein the corresponding process, executed on each node of said plurality of nodes, is configured to start a program to perform the computation in response to a command received from said database system.
5. The method of claim 1 , wherein each said corresponding process instructs a program to perform the computation upon a data container.
6. A method comprising performing a machine-executed operation involving instructions, wherein the machine-executed operation is at least one of:
A) sending said instructions over transmission media;
B) receiving said instructions over transmission media;
C) storing said instructions onto a machine-readable storage medium; and
D) executing the instructions;
wherein said instructions are instructions which, when executed by one or more processors, cause the one or more processors to perform the steps of:
receiving a statement, at a database system, specifying an external routine for performing the computation; and
in response to receiving the statement:
transmitting a plurality of commands for performing the computation to a plurality of respective processes;
receiving results from each said corresponding process; and
completing processing of the statement based on the results received from each said corresponding process.
7. A method according to claim 6 , wherein said instructions, when executed by the one or more processors, further cause the one or more processors to perform the step of:
determining a cohort of nodes from among a plurality of nodes capable of performing the computation,
wherein the plurality of the respective processes correspond to the cohort of the nodes.
8. A method according to claim 7 , wherein the plurality of nodes includes at least one node not included in the cohort of the nodes.
9. A method according to claim 7 , wherein said determining is based on a degree of parallelism supported by each of the nodes.
10. A method according to claim 7 , wherein said instructions, when executed by the one or more processors, further cause the one or more processors to perform the step of accessing a registry specifying an association between the programs and the data containers, wherein said determining is based on the association between the programs and the data containers.
11. A method according to claim 7 , wherein said instructions, when executed by the one or more processors, further cause the one or more processors to perform the step of accessing a registry specifying respective attributes for the data containers, wherein said determining is based on matching the respective attributes for the data containers with a parameter in the statement.
12. A method according to claim 7 , wherein said instructions, when executed by the one or more processors, further cause the one or more processors to perform the step of accessing a registry specifying a partitioning function associated with the programs and the data containers, wherein said determining is based on the results from executing the partitioning function associated with the programs and the data containers.
13. The method of claim 6 , wherein the steps of transmitting the plurality of commands, receiving results, and completing processing are performed by said database system.
14. The method of claim 6 , wherein each of the plurality of respective processes is executing on one of a plurality of nodes, and wherein at least one of said plurality of nodes is implemented using a different type of hardware, operation system software, or application software than said database system.
15. The method of claim 6 , wherein each of the respective processes is configured to start a program to perform the computation in response to said commands.
16. A method according to claim 15 , wherein at least some of the programs, started by each of the respective processes, execute in parallel.
17. The method of claim 6 , wherein each said corresponding process instructs a program to perform the computation upon a data container.