DATA FLOW GRAPH NODE PARALLEL UPDATE FOR MACHINE LEARNING
Techniques are disclosed for data flow graph node parallel update for machine learning. A first plurality of processing elements is configured to implement a portion of a data flow graph. The nodes include at least one variable node and implement part of a neural network. A second plurality of processing elements is configured to implement a second portion of the data flow graph. These nodes include at least one additional variable node and implement an additional part of the neural network. Training data is issued to the first plurality of processing elements. The training data is used to update variables within the at least one variable node. Additional variables are updated within the at least one additional variable node. The updating includes forwarding training data from the first plurality to the second plurality. The neural network is trained based on the variables that were updated and the additional variables.
1 . A processor-implemented method for data manipulation comprising:
configuring a first plurality of processing elements within a reconfigurable fabric to implement a first portion of a data flow graph, wherein nodes of the first portion of the data flow graph include at least one variable node, and wherein the first portion of the data flow graph implements part of a neural network;
configuring a second plurality of processing elements within the reconfigurable fabric to implement a second portion of the data flow graph, wherein the nodes of the second portion of the data flow graph include at least one additional variable node, and wherein the second portion of the data flow graph implements an additional part of the neural network;
issuing training data to the first plurality of processing elements, wherein the training data is used to update variables within the at least one variable node; and
updating additional variables within the at least one additional variable node, wherein the updating is based on forwarding the training data from the first plurality of processing elements to the second plurality of processing elements.
2 . The method of claim 1 further comprising training the neural network based on the variables within the at least one variable node that were updated and the additional variables within the at least one additional variable node.
3 . The method of claim 1 further comprising passing gradients from the second plurality of processing elements to the first plurality of processing elements, wherein the gradients are used to further update variables within the at least one variable node.
4 . The method of claim 3 further comprising forwarding additional training data from the first plurality of processing elements to the second plurality of processing elements, wherein the additional training data is based on data from the variables within the at least one variable node that were further updated.
5 . The method of claim 3 wherein the gradients are used as part of the updating of the additional variables.
6 . The method of claim 1 wherein the variables and the additional variables are updated concurrently.
7 . The method of claim 6 wherein the variables and the additional variables that are updated concurrently comprise parallel neural network training.
8 . The method of claim 1 wherein the forwarding the training data includes N copies of a variable contained within the at least one variable node, wherein the N copies are used for distribution within the data flow graph, and wherein N is an integer greater than or equal to 1 and less than or equal to a total number of nodes in the data flow graph.
9 . The method of claim 1 wherein the variables and the additional variables that were both updated include a mean of gradients.
10 . The method of claim 9 wherein the mean of gradients is sourced based on gradients from the first plurality of processing elements and the second plurality of processing elements.
11 . The method of claim 10 further comprising training the neural network, based on the mean of gradients.
12 . The method of claim 1 wherein the data flow graph comprises machine learning or deep learning.
13 . The method of claim 1 wherein the configuring is controlled by a session manager.
14 . The method of claim 1 wherein the processing elements are controlled by circular buffers.
15 . The method of claim 14 wherein the circular buffers are statically scheduled.
16 . The method of claim 1 wherein data flow graph is used to train a neural network.
17 - 18 . (canceled)
19 . The method of claim 1 wherein the reconfigurable fabric comprises processing elements.
20 . The method of claim 19 wherein the processing elements are controlled by circular buffers.
21 . (canceled)
22 . A computer program product embodied in a non-transitory computer readable medium for data manipulation, the computer program product comprising code which causes one or more processors to perform operations of:
configuring a first plurality of processing elements within a reconfigurable fabric to implement a first portion of a data flow graph, wherein nodes of the first portion of the data flow graph include at least one variable node, and wherein the first portion of the data flow graph implements part of a neural network;
configuring a second plurality of processing elements within the reconfigurable fabric to implement a second portion of the data flow graph, wherein the nodes of the second portion of the data flow graph include at least one additional variable node, and wherein the second portion of the data flow graph implements an additional part of the neural network;
issuing training data to the first plurality of processing elements, wherein the training data is used to update variables within the at least one variable node; and
updating additional variables within the at least one additional variable node, wherein the updating is based on forwarding the training data from the first plurality of processing elements to the second plurality of processing elements.
23 . A computer system for data manipulation comprising:
a memory which stores instructions;
one or more processors attached to the memory wherein the one or more processors, when executing the instructions which are stored, are configured to:
configure a first plurality of processing elements within a reconfigurable fabric to implement a first portion of a data flow graph, wherein nodes of the first portion of the data flow graph include at least one variable node, and wherein the first portion of the data flow graph implements part of a neural network;
configure a second plurality of processing elements within the reconfigurable fabric to implement a second portion of the data flow graph, wherein the nodes of the second portion of the data flow graph include at least one additional variable node, and wherein the second portion of the data flow graph implements an additional part of the neural network;
issue training data to the first plurality of processing elements, wherein the training data is used to update variables within the at least one variable node; and
update additional variables within the at least one additional variable node, wherein the updating is based on forwarding the training data from the first plurality of processing elements to the second plurality of processing elements.