Partial-activation of neural network based on heat-map of neural network activity
A device, system, and method for training or prediction of a neural network. A current value may be stored for each of a plurality of synapses or filters in the neural network. A historical metric of activity may be independently determined for each individual or group of the synapses or filters during one or more past iterations. A plurality of partial activations of the neural network may be iteratively executed. Each partial-activation iteration may activate a subset of the plurality of synapses or filters in the neural network. Each individual or group of synapses or filters may be activated in a portion of a total number of iterations proportional to the historical metric of activity independently determined for that individual or group of synapses or filters. Training or prediction of the neural network may be performed based on the plurality of partial activations of the neural network.
1. A method for training or prediction of a neural network, the method comprising:
storing a current value of each of a plurality of synapses or filters in a neural network;
generating a heat map comprising a historical metric of activity for each individual or group of one or more of the synapses or filters during one or more past iterations;
iteratively executing a plurality of partial activations of the neural network, wherein each partial-activation iteration activates a subset of the plurality of synapses or filters in the neural network based on the heat map of historical metrics of activity of the synapses or filters, wherein the synapses or filters in hot spots of the heat map that were activated with relatively high weights or frequencies during the one or more past iterations are activated more often in the partial-activation iterations than synapses or filters activated in cold spots of the heat map that were activated with relatively low weights or frequencies during the one or more past iterations; and
performing training or prediction of the neural network based on the plurality of partial activations of the neural network.
2. The method of claim 1 , wherein the historical metric of activity is selected from the group consisting of: a frequency that each individual or group of synapses or filters was activated in one or more past iterations, a magnitude of the synapse or filter weights in one or more past iterations, a magnitude of a change in the synapse or filter weights in one or more past error correction iterations, an average median or standard deviation of any of those measures, and a value derived from any individual or combination thereof.
3. The method of claim 1 comprising dynamically updating the historical metrics of activity periodically or each iteration or time an individual or group of synapses or filters is activated.
4. The method of claim 3 comprising displaying a moving image of the heat map comprising a sequence of frames visualizing the dynamically updated historical metrics of activity over a sequence of one or more respective iterations.
5. The method of claim 1 comprising, during training, performing a plurality of backward pass error correction iterations to correct each individual or group of activated synapses or filters.
6. The method of claim 1 comprising, during training or prediction:
performing a plurality of forward pass inference iterations to generate a plurality of partial outputs, respectively, in which each individual or group of synapses or filters is computed in the portion of the total number of inference iterations; and
aggregating the plurality of partial outputs to generate an aggregated output approximating an output generated by fully-activating the neural network by executing a single instance of all synapses or filters of the neural network.
7. The method of claim 6 , wherein the synapses or filters activated in the plurality of inference iterations cumulatively span only a subset of all the synapses or filters of the neural network such that a plurality of the remaining synapses or filters are not activated.
8. The method of claim 1 comprising determining the subset of the plurality of synapses or filters to activate based on an input into the neural network.
9. The method of claim 1 , wherein synapses or filters are activated in discrete groups selected from the groups consisting of: sequences of synapses or filters forming neural network pathways linking a continuous sequence of synapses or filters from an input layer to an output layer, groups of synapses each directly connected to a neuron, and groups of filters each directly connected to a group of neurons in a convolutional neural network.
10. The method of claim 1 comprising performing a hybrid combination of partially-activating and fully-activating the neural network for different iterations, layers, filters, channels, or modes of operation.
11. The method of claim 10 comprising alternating between partially-activating and fully-activating the neural network according to a predetermined schedule.
12. The method of claim 10 comprising switching between partially-activating and fully-activating the neural network when a measure of error or accuracy crosses a threshold.
13. The method of claim 1 comprising reactivating an individual or group of synapses or filters that are inactive for a greater than threshold time or number of iterations.
14. The method of claim 1 comprising initiating training or prediction by fully activating all synapses or filters of the neural network and incrementally deactivating individual or groups of synapses or filters with below threshold historical metrics of activity.
15. The method of claim 1 comprising initiating training or prediction by partially activating a random or semi-random subset of synapses or filters of the neural network and activating additional synapses or filters with above threshold historical metrics of activity and deactivating synapses or filters with below threshold historical metrics of activity.
16. The method of claim 1 comprising generating a sparse neural network by pruning or eliminating synapses or filters with below threshold historical metrics of activity or above threshold numbers of deactivated iterations.
17. The method of claim 1 comprising tuning the portion of total iterations in which the individual or group of synapse or filters are activated.
18. The method of claim 1 comprising:
partially activating the neural network only during training and fully-activating the neural network only during prediction,
partially activating the neural network only during prediction and fully-activating the neural network only during training mode, or
partially activating the neural network during both training and prediction.
19. A system for training or prediction of a neural network, the system comprising:
one or more memories to store a current value of each of a plurality of synapses or filters in the neural network;
one or more processors configured to:
generate a heat map comprising a historical metric of activity for each individual or group of one or more of the synapses or filters during one or more past iterations,
iteratively execute a plurality of partial activations of the neural network, wherein each partial-activation iteration activates a subset of the plurality of synapses or filters in the neural network based on the heat map of historical metrics of activity of the synapses or filters, wherein the synapses or filters in hot spots of the heat map that were activated with relatively high weights or frequencies during the one or more past iterations are activated more often in the partial-activation iterations than synapses or filters activated in cold spots of the heat map that were activated with relatively low weights or frequencies during the one or more past iterations, and
train or predict using the neural network based on the plurality of partial activations of the neural network.
20. The system of claim 19 , wherein the historical metric of activity is selected from the group consisting of: a frequency that each individual or group of synapses or filters was activated in one or more past iterations, a magnitude of the synapse or filter weights in one or more past iterations, a magnitude of a change in the synapse or filter weights in one or more past error correction iterations, an average median or standard deviation of any of those measures, and a value derived from any individual or combination thereof.
21. The system of claim 19 , wherein the one or more processors are configured to dynamically update the historical metrics of activity periodically or each iteration or time an individual or group of synapses or filters is activated.
22. The system of claim 19 , wherein the one or more processors are configured to visualize, on a display device, a moving image of the heat map comprising a sequence of frames showing the dynamically updated historical metrics of activity over a sequence of one or more respective iterations.
23. The system of claim 19 , wherein the one or more processors are configured to, during training, perform a plurality of backward pass error correction iterations to correct each individual or group of activated synapses or filters.
24. The system of claim 19 , wherein the one or more processors are configured to, during training or prediction:
perform a plurality of forward pass inference iterations to generate a plurality of partial outputs, respectively, based on computations of each individual or group of activated synapses or filters, and
aggregate the plurality of partial outputs to generate an aggregated output approximating an output generated by fully-activating the neural network by executing a single instance of all synapses or filters of the neural network.
25. The system of claim 19 , wherein the one or more processors are configured to determine the subset of the plurality of synapses or filters to activate based on an input into the neural network.
26. The system of claim 19 , wherein the one or more processors are configured to activate the synapses or filters in discrete groups selected from the groups consisting of: sequences of synapses or filters forming neural network pathways linking a continuous sequence of synapses or filters from an input layer to an output layer, groups of synapses each directly connected to a neuron, and groups of filters each directly connected to a group of neurons in a convolutional neural network.
27. The system of claim 19 , wherein the one or more processors are configured to perform a hybrid combination of partially-activating and fully-activating the neural network for different iterations, layers, filters, channels, or modes of operation.
28. The system of claim 19 , wherein the one or more processors are configured to generate a sparse neural network by pruning or eliminating synapses or filters with below threshold historical metrics of activity or above threshold numbers of deactivated iterations.
29. The system of claim 19 , wherein the one or more processors are configured to tune the portion of total iterations in which the individual or group of synapse or filters are activated.
30. The system of claim 19 , wherein the one or more processors are configured to:
partially-activate the neural network only during training and fully-activate the neural network only during prediction,
partially-activate the neural network only during prediction and fully-activate the neural network only during training mode, or
partially-activate the neural network during both training and prediction.