Systems and methods for processing using directed acyclic graphs
A computer system for executing a processing graph including: (A) accessing a graph definition of the processing graph from a nodes database, each node of the processing graph representing a component of computation performed by at least one program referenced by that node; (B) constructing the processing graph in the at least one memory device; (C) identifying at least one group of nodes in the processing graph; (D) traversing the at least one group of nodes; (E) upon visiting a node of the at least one group of nodes, accessing a node definition of the node from the nodes database, the node definition identifying a program referenced by the node; (F) executing the program to generate an output; and (G) passing the output to a parent node of the node, the parent node using the output during processing of the parent node.
1 . A computer system for executing computational components in accordance with a custom processing graph comprising a plurality of nodes, wherein each node of the plurality of nodes defines one of the computational components comprising at least one computer program referenced by that node, the computer system comprising at least one processor in communication with at least one memory device, the at least one processor programmed to:
access, from a nodes database, a graph definition to construct the custom processing graph, the graph definition identifying the plurality of nodes for the custom processing graph and one or more edges, each edge of the one or more edges connecting a pair of nodes of the identified plurality of nodes;
provide a graphical user interface configured to receive user input from a user;
construct a root node for the custom processing graph based upon the user input;
perform a lookup, within the nodes database, for the identified plurality of nodes;
construct the custom processing graph in the at least one memory device by:
in response to not finding an identified node of the identified plurality of nodes, (i) marking the identified node as a missing node and (ii) generating an error for the missing node;
correcting the error by performing at least one of (i) adding the missing node to the custom processing graph, (ii) editing a parent node of the missing node to avoid reliance on the missing node, or (iii) constructing a custom node to stand in for the missing node; and
creating, using the one mor more edges, dependencies and links between the root node and one or more identified nodes of the identified plurality of nodes found in the lookup, thereby enabling automatic detection and integration of dependent computational components defined by the identified plurality of nodes;
receive an input to generate a particular output for each of thousands of data record identifiers;
identify, using the received input, at least one group of the identified plurality of nodes in the custom processing graph, the identified at least one group associated with the thousands of data record identifiers;
traverse, for each of the thousands of data record identifiers, the identified at least one group of nodes by starting at the root node and visiting each identified node of the identified at least one group of nodes;
upon visiting each identified node, access a node definition of the identified node from the nodes database, the node definition identifying (i) a computer program referenced by the identified node and (ii) the particular output to be generated by executing the identified computer program;
in response to identifying the computer program referenced each identified node, execute the identified computer program to generate the particular output;
in response to generating the particular output, pass the particular output to an identified parent node of the identified node, the identified parent node using the particular output during processing of the identified parent node; and
in response to processing the identified parent node and the root node, generate a final output.
2 . The computer system of claim 1 , wherein identifying the at least one group of nodes in the custom processing graph comprises identifying a first group of nodes and second group of nodes in the custom processing graph.
3 . The computer system of claim 2 , wherein traversing the identified at least one group of nodes comprises traversing the first group of nodes in the custom processing graph to visit and process each node in the first group and traversing the second group of nodes in the custom processing graph to visit and process each node in the second group.
4 . The computer system of claim 3 , wherein the first group of nodes and the second group of nodes are traversed simultaneously.
5 . The computer system of claim 1 , wherein the at least one processor is further programmed to provide the graphical user interface configured to receive the user input from the user prior to execution of the custom processing graph, the user input modifying a virtual data structure of the custom processing graph by adding, editing, replacing, or deleting one or more of the identified plurality of nodes within the custom processing graph, thereby reducing memory consumption by loading only the modified virtual data structure and associated computations for executing the custom processing graph.
6 . The computer system of claim 1 , wherein the node includes a child node, wherein execution of the computer program further includes visiting the child node.
7 . The computer system of claim 6 , wherein visiting the child node includes passing input data from the node to the child node, and wherein visiting the child node further includes using the input data to execute another computer program to generate child output data.
8 . The computer system of claim 7 , wherein visiting the child node further includes passing the child output data from the child node to the node.
9 . The computer system of claim 7 , wherein executing the computer program further includes using the child output data to generate the particular output.
10 . The computer system of claim 1 , wherein executing the computer program includes executing the computer program based upon an embedded programming language provided on the computer system.
11 . A computer-implemented method for executing computational components in accordance with a custom processing graph including a plurality of nodes, wherein each node of the plurality of nodes defines one of the computational component including at least one computer program referenced by that node, the method implemented by a computer system including at least one processor in communication with at least one memory device, the method comprising:
accessing, from a nodes database, a graph definition to construct the custom processing graph from, the graph definition identifying the plurality of nodes for the custom processing graph and one or more edges, each edge of the one or more edges connecting a pair of nodes of the identified plurality of nodes;
providing a graphical user interface configured to receive user input from a user;
constructing a root node for the custom processing graph based upon the user input;
performing a lookup, within the nodes database, for the identified plurality of nodes;
constructing the custom processing graph in the at least one memory device by:
in response to not finding an identified node of the identified plurality of nodes, (i) marking the identified node as a missing node and (ii) generating an error for the missing node;
correcting the error by performing at least one of (i) adding the missing node to the custom processing graph, (ii) editing a parent node of the missing node to avoid reliance on the missing node, or (iii) constructing a custom node to stand in for the missing node; and
creating, using the one mor more edges, dependencies and links between the root node and one or more identified nodes of the identified plurality of nodes found in the lookup, thereby enabling automatic detection and integration of dependent computational components defined by the plurality of nodes;
receiving an input to generate a particular output for each of thousands of data record identifiers;
identifying, using the received input, at least one group of the identified plurality of nodes in the custom processing graph, the identified at least one group associated with the thousands of data record identifiers;
traversing, for each of the thousands of data record identifiers, the identified at least one group of nodes starting at the root node and visiting each identified node of the identified at least one group of nodes;
upon visiting identified node, accessing a node definition of the identified node from the nodes database, the node definition identifying (i) a computer program referenced by the identified node and (ii) the particular output to be generated by executing the identified computer program;
response to identifying the computer program referenced each identified node, executing the identified computer program to generate the particular output;
in response to generating the particular output, passing the particular output to an identified parent node of the identified node, the identified parent node using the particular output during processing of the identified parent node; and
in response to processing the identified parent node and the root node, generating a final output.
12 . The computer-implemented method of claim 11 , wherein identifying the at least one group of nodes in the custom processing graph comprises identifying a first group of nodes and second group of nodes in the custom processing graph.
13 . The computer-implemented method of claim 12 , wherein traversing the identified at least one group of nodes comprises traversing the first group of nodes in the custom processing graph to visit and process each node in the first group and traversing the second group of nodes in the custom processing graph to visit and process each node in the second group.
14 . The computer-implemented method of claim 13 , wherein the first group of nodes and the second group of nodes are traversed simultaneously.
15 . The computer-implemented method of claim 11 , wherein the at least one processor is further programmed to provide the graphical user interface configured to receive the user input from the user prior to execution of the custom processing graph, the user input modifying a virtual data structure of the custom processing graph by adding, editing, replacing, or deleting one or more of the identified plurality of nodes within the custom processing graph, thereby reducing memory consumption by loading only the modified virtual data structure and associated computations for executing the custom processing graph.
16 . The computer-implemented method of claim 11 , wherein the node includes a child node, wherein execution of the computer program further includes visiting the child node.
17 . The computer-implemented method of claim 16 , wherein visiting the child node includes passing input data from the node to the child node, wherein visiting the child node includes using the input data to execute another computer program to generate child output data.
18 . The computer-implemented method of claim 17 , wherein visiting the child node further includes passing the child output data from the child node to the node.
19 . The computer-implemented method of claim 17 , wherein executing the computer program further includes using the child output data to generate the particular output.
20 . The computer-implemented method of claim 11 , wherein executing the computer program includes executing the computer program based upon an embedded programming language provided on the computer system.