DATA FLOW GRAPH NODE UPDATE FOR MACHINE LEARNING
Techniques are disclosed for data flow graph node update for machine learning. A plurality of processing elements is configured within a reconfigurable fabric to implement a data flow graph. The nodes of the data flow graph include one or more variable nodes, and the data flow graph implements a neural network. N copies of a variable contained in a variable node are issued, where the N copies are used for distribution within the data flow graph, and where N is an integer greater than or equal to one and less than or equal to the total number of nodes in the graph. The N copies of a variable are distributed within the data flow graph. The neural network is updated based on the N copies of a variable. Results from the distribution are averaged. The averaging includes parallel training of different data for machine learning.
1 . A processor-implemented method for data manipulation comprising:
configuring a plurality of processing elements within a reconfigurable fabric to implement a data flow graph, wherein nodes of the data flow graph include one or more variable nodes, and wherein the data flow graph implements a neural network;
issuing N copies of a variable contained in one of the one or more variable nodes, wherein the N copies are used for distribution within the data flow graph, and wherein N is an integer greater than or equal to 1;
distributing the N copies of a variable within the data flow graph; and
updating the neural network, based on the N copies of a variable.
2 . The method of claim 1 wherein the issuing N copies occurs before the one or more variable nodes are paused for updating.
3 . The method of claim 1 wherein the distributing within the data flow graph includes propagating the N copies to other nodes within the data flow graph.
4 . The method of claim 3 wherein the other nodes include non-variable nodes.
5 . The method of claim 4 wherein the non-variable nodes further distribute the N copies to still other nodes within the data flow graph.
6 . The method of claim 1 wherein N is less than or equal to a total number of nodes in the graph.
7 . The method of claim 1 further comprising averaging updates resulting from the distributing the N copies of a variable.
8 . The method of claim 7 further comprising training the neural network, based on the averaging.
9 . The method of claim 8 wherein the training comprises distributed neural network training.
10 . The method of claim 1 further comprising updating based on a running average of copies of the variable within the data flow graph.
11 . The method of claim 1 wherein the variable nodes contain weights for deep learning.
12 . The method of claim 1 wherein the data flow graph comprises machine learning or deep learning.
13 . The method of claim 1 wherein the configuring is controlled by a session manager.
14 . The method of claim 1 further comprising pausing the data flow graph.
15 . The method of claim 14 wherein the pausing is accomplished by loading invalid data.
16 . The method of claim 15 wherein the pausing is controlled by an execution manager.
17 . The method of claim 14 wherein the pausing is accomplished by withholding new data from entering the data flow graph.
18 . The method of claim 17 wherein the pausing is controlled by an execution manager.
19 . The method of claim 1 further comprising issuing two or more sets of N copies of the variable for distribution within the data flow graph.
20 . The method of claim 19 further comprising averaging two or more sets of updates resulting from the distributing the two or more sets of N copies.
21 . The method of claim 20 wherein the averaging two or more sets of updates comprises parallel training of different data for machine learning.
22 . The method of claim 1 wherein the processing elements are controlled by circular buffers.
23 . (canceled)
24 . The method of claim 1 wherein data flow graph is used to train a neural network.
25 - 26 . (canceled)
27 . A computer program product embodied in a non-transitory computer readable medium for data manipulation, the computer program product comprising code which causes one or more processors to perform operations of:
configuring a plurality of processing elements within a reconfigurable fabric to implement a data flow graph, wherein nodes of the data flow graph include one or more variable nodes, and wherein the data flow graph implements a neural network;
issuing N copies of a variable contained in one of the one or more variable nodes, wherein the N copies are used for distribution within the data flow graph, and wherein N is an integer greater than or equal to 1;
distributing the N copies of a variable within the data flow graph; and
updating the neural network, based on the N copies of a variable.
28 . A computer system for data manipulation comprising:
a memory which stores instructions;
one or more processors attached to the memory wherein the one or more processors, when executing the instructions which are stored, are configured to:
configure a plurality of processing elements within a reconfigurable fabric to implement a data flow graph, wherein nodes of the data flow graph include one or more variable nodes, and wherein the data flow graph implements a neural network;
issue N copies of a variable contained in one of the one or more variable nodes, wherein the N copies are used for distribution within the data flow graph, and wherein N is an integer greater than or equal to 1;
distribute the N copies of a variable within the data flow graph; and
update the neural network, based on the N copies of a variable.