Dynamic self-reconfiguration of nodes in a processing pipeline

1. A computer-implemented method for processing a query, the method comprising:

generating a query plan for the query;

instantiating, by a data processing system, an execution pipeline for executing the query based upon the query plan, wherein the execution pipeline comprises a directed acyclic graph comprising a plurality of nodes,

determining, by at least a first node in the plurality of nodes, capabilities of one or more neighboring nodes in the plurality of nodes, wherein the determining comprises:

communicating, by the first node to the one or more neighboring nodes of the first node within the execution pipeline, information identifying a capability of the first node; and

receiving, by the first node from at least one neighboring node of the one or more neighboring nodes within the execution pipeline, information identifying a capability of the at least one neighboring node;

identifying, by the data processing system, based upon the capability of the first node, the capability of the at least one neighboring node, or a combination thereof, a change to be made to the execution pipeline, the change involving the first node, the at least one neighboring node, or a combination thereof;

applying the change to the execution pipeline to create a modified execution pipeline, wherein the applying comprises reconfiguring the first node, the at least one neighboring node, or a combination thereof, wherein the reconfiguring the first node comprises: (i) changing a type of an input or an output of the first node from a first type to a second type different from the first type, and (ii) reconfiguring a function performed by the first node from handling the first type to handling the second type, and wherein: (a) the first type is a variable length record type and the second type is a fixed length record type, or (b) the first type is a fixed length record type and the second type is a variable length record type; and

executing the query by executing the modified execution pipeline.

2. The method of claim 1 , wherein applying the change to the execution pipeline includes reconfiguring a position of the first node within the directed acyclic graph.

3. The method of claim 2 , wherein:

prior to the applying, the first node is positioned downstream in the directed acyclic graph from a second node in the plurality of nodes; and

reconfiguring the position of the first node within the directed acyclic graph comprises moving the first node to a new position within the directed acyclic graph wherein the first node is upstream from the second node in the modified execution pipeline.

4. The method of claim 3 , wherein the second distance is less than the first distance.

5. The method of claim 2 , wherein:

prior to the applying, the first node is at a first distance from a source root node in the directed acyclic graph; and

reconfiguring the position of the first node within the directed acyclic graph comprises changing the first node to a new position within the directed acyclic graph at a second distance from the source root node, the second distance being different from the first distance.

6. The method of claim 1 , further comprising receiving, by the first node from at least one neighboring node of the first node within the execution pipeline, information identifying a capability of a node in the plurality of nodes other than the first node and the at least one neighboring node of the first node.

7. A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform processing comprising:

generating a query plan for a query;

instantiating an execution pipeline for executing the query based upon the query plan, wherein the execution pipeline comprises a directed acyclic graph comprising a plurality of nodes,

determining, by at least a first node in the plurality of nodes, capabilities of one or more neighboring nodes in the plurality of nodes, wherein the determining comprises:

communicating, by the first node to the one or more neighboring nodes of the first node within the execution pipeline, information identifying a capability of the first node; and

receiving, by the first node from at least one neighboring node of the one or more neighboring nodes within the execution pipeline, information identifying a capability of the at least one neighboring node;

identifying, based upon the capability of the first node, the capability of the at least one neighboring node, or a combination thereof a change to be made to the execution pipeline, the change involving the first node, the at least one neighboring node, or a combination thereof;

applying the change to the execution pipeline to create a modified execution pipeline, wherein the applying comprises reconfiguring the first node, the at least one neighboring node, or a combination thereof within the directed acyclic graph, wherein the reconfiguring the first node comprises: (i) changing a type of an input or an output of the first node from a first type to a second type different from the first type, and (ii) reconfiguring a function performed by the first node from handling the first type to handling the second type, and wherein: (a) the first type is a variable length record type and the second type is a fixed length record type, or (b) the first type is a fixed length record type and the second type is a variable length record type; and

executing the query by executing the modified execution pipeline.

8. The non-transitory computer-readable medium of claim 7 , wherein applying the change to the execution pipeline includes changing a position of the first node within the directed acyclic graph.

9. A data processing system comprising:

one or more processors;

memory associated with the one or more processors, the memory storing instructions that when executed by the one or more processors cause the one or more processors to perform processing comprising:

generating a query plan for a query;

instantiating an execution pipeline for executing the query based upon the query plan, wherein the execution pipeline comprises a directed acyclic graph comprising a plurality of nodes,

determining, by at least a first node in the plurality of nodes, capabilities of one or more neighboring nodes in the plurality of nodes, wherein the determining comprises:

communicating, by the first node to the one or more neighboring nodes of the first node within the execution pipeline, information identifying a capability of the first node; and

receiving, by the first node from at least one neighboring node of the one or more neighboring nodes within the execution pipeline, information identifying a capability of the at least one neighboring node;

identifying, based upon the capability of the first node, the capability of the at least one neighboring node, or a combination thereof, a change to be made to the execution pipeline, the change involving the first node, the at least one neighboring node, or a combination thereof;

applying the change to the execution pipeline to create a modified execution pipeline, wherein the applying comprises reconfiguring the first node, the at least one neighboring node, or a combination thereof within the directed acyclic graph, wherein the reconfiguring the first node comprises: (i) changing a type of an input or an output of the first node from a first type to a second type different from the first type, and (ii) reconfiguring a function performed by the first node from handling the first type to handling the second type, and wherein: (a) the first type is a variable length record type and the second type is a fixed length record type, or (b) the first type is a fixed length record type and the second type is a variable length record type; and

executing the query by executing the modified execution pipeline.

10. The data processing system of claim 9 , wherein applying the change to the execution pipeline includes changing a position of the first node within the directed acyclic graph.