Granted Patent B2

US 12,260,337 · App. 18/312,011 · Granted Mar 25, 2025

Inventors: Subutai Ahmad (Palo Alto, CA); Luiz Scheinkman (Sunnyvale, CA)

Assignee: Numenta, Inc.

G06N3/084G06N3/04

Abstract

Claims (41)

1. A method, comprising:

receiving input data by a neural network including a plurality of layers of nodes including a current layer and a previous layer;

generating a layer output of the previous layer based at least on the input data;

increasing sparsity of a set of weights that represents connections between nodes of the current layer and nodes of the previous layer to generate a sparsified set of weights that indicates each of the nodes of the current layer as having a predetermined number of connections to the nodes of the previous layer;

applying, to the layer output of the previous layer, the sparsified set of weights;

generating intermediate outputs for the current layer by applying the set of weights for the current layer to a layer output of the previous layer; and

generating a layer output for nodes of the current layer by increasing sparsity of the intermediate outputs.

2. The method of claim 1 , wherein the generating of the layer output comprises:

selecting intermediate outputs having values above a value or within a percentage of highest values, and

setting values of the selected intermediate outputs as values of corresponding nodes of the current layer.

3. The method of claim 2 , wherein generating of the layer output further comprises:

setting, to zero, values of nodes of the current layer corresponding to non-selected intermediate outputs.

4. The method of claim 1 , further comprising backpropagating error terms to update the set of weights, wherein the error terms are derived from the layer output and training data corresponding to the input data.

5. The method of claim 4 , further comprising determining a boosting term for each node of the current layer, the boosting term indicating how frequently the node was selected during previous iterations.

6. The method of claim 5 , wherein the boosting term increases as a duty cycle of each node increases and decreases as the duty cycle of each node decreases.

7. The method of claim 1 , wherein the neural network is a convolution neural network.

8. The method of claim 1 , wherein the neural network is implemented on field-programmable gate arrays.

9. A non-transitory computer readable storage medium storing instructions thereon, the instructions when executed by a processor cause the processor to:

receive input data by a neural network including a plurality of layers of nodes including a current layer and a previous layer;

generate a layer output of the previous layer based at least on the input data;

increase sparsity of a set of weights that represents connections between nodes of the current layer and nodes of the previous layer to generate a sparsified set of weights that indicates each of the nodes of the current layer as having a predetermined number of connections to the nodes of the previous layer;

apply, to the layer output of the previous layer, the sparsified set of weights;

generate intermediate outputs for the current layer by applying the set of weights for the current layer to a layer output of the previous layer; and

generate a layer output for nodes of the current layer by increasing sparsity of the intermediate outputs.

10. The non-transitory computer readable storage medium of claim 9 , wherein the instructions to generate the layer output comprises instructions to:

select intermediate outputs having values above a value or within a percentage of highest values; and

set values of the selected intermediate outputs as values of corresponding nodes of the current layer.

11. The non-transitory computer readable storage medium of claim 10 , wherein the instructions to generate the layer output comprises instructions to:

set, to zero, values of nodes of the current layer corresponding to non-selected intermediate outputs.

12. The non-transitory computer readable storage medium of claim 9 , further comprising instructions to backpropagate error terms to update the set of weights, wherein the error terms are derived from the layer output and training data corresponding to the input data.

13. The non-transitory computer readable storage medium of claim 12 , further comprising determining a boosting term for each node of the current layer, the boosting term indicating how frequently the node was selected during previous iterations.

14. The non-transitory computer readable storage medium of claim 13 , wherein the boosting term increases as a duty cycle of each node increases and decreases as the duty cycle of each node decreases.

15. The non-transitory computer readable storage medium of claim 9 , wherein the neural network is a convolution neural network.

16. A non-transitory storage medium storing a digital representation of a neural network, the neural network generated by:

receiving input data by a neural network including a plurality of layers of nodes including a current layer and a previous layer;

generating a layer output of the previous layer based at least on the input data;

increase sparsity of a set of weights that represents connections between nodes of the current layer and nodes of the previous layer to generate a sparsified set of weights that indicates each of the nodes of the current layer as having a predetermined number of connections to the nodes of the previous layer;

apply, to the layer output of the previous layer, the sparsified set of weights;

generating intermediate outputs for the current layer by applying the set of weights for the current layer to a layer output of the previous layer;

generating a layer output for nodes of the current layer by increasing sparsity of the intermediate outputs to be equal to or above a second threshold; and

backpropagating error terms to update the set of weights, wherein the error terms are derived from the layer output and training data corresponding to the input data.

Continuity (2)

References Cited (395)