Cluster-connected neural network
A device, system, and method is provided for training or prediction using a cluster-connected neural network. The cluster-connected neural network may be divided into a plurality of clusters of artificial neurons connected by weights or convolutional channels connected by convolutional filters. Within each cluster is a locally dense sub-network of intra-cluster weights or filters with a majority of pairs of neurons or channels connected by intra-cluster weights or filters that are co-activated together as an activation block during training or prediction. Outside each cluster is a globally sparse network of inter-cluster weights or filters with a minority of pairs of neurons or channels separated by a cluster border across different clusters connected by inter-cluster weights or filters. Training or predicting is performed using the cluster-connected neural network.
1. A method for training or prediction using a cluster-connected neural network at a local endpoint device, the method comprising:
storing a cluster-connected neural network at the local endpoint device, the cluster-connected neural network having a neural network axis in an orientation extending from an input layer to an output layer and orthogonal to a plurality of intermediate layers, wherein the cluster-connected neural network is divided into a plurality of clusters, wherein each cluster comprises a different plurality of artificial neurons or convolutional channels, wherein the artificial neurons or convolutional channels of each cluster are in a region extending parallel to the direction of the neural network axis resulting in a predominant direction of neuron activation extending from the input layer toward the output layer, wherein each pair of neurons or channels are uniquely connected by a weight or convolutional filter;
within each cluster of the cluster-connected neural network, generating or maintaining a locally dense sub-network of intra-cluster weights or filters, in which a majority of pairs of neurons or channels within the same cluster are connected by intra-cluster weights or filters, such that, the connected majority of pairs of neurons or channels in each cluster are co-activated together as an activation block during training or prediction using the cluster-connected neural network;
outside each cluster of the cluster-connected neural network, generating or maintaining a globally sparse network of inter-cluster weights or filters, in which a minority of pairs of neurons or channels separated by a cluster border across different clusters are connected by inter-cluster weights or filters; and
performing prediction using the cluster-connected neural network at the local endpoint device.
2. The method of claim 1 comprising testing neuron or channel activation patterns in the cluster-connected neural network to determine an optimal cluster shape that most closely resembles activation patterns of highly linked neurons or channels resulting from the test.
3. The method of claim 2 comprising dynamically adjusting the optimal cluster shape as activation patterns change during training.
4. The method of claim 1 , wherein the cluster border of one or more of the plurality of clusters has a shape selected from the group consisting of: a column, row, circle, polygon, irregular shape, rectangular prism, cylinder, polyhedron, and another two-dimensional, three-dimensional, or N-dimensional shape.
5. The method of claim 1 comprising training the cluster-connected neural network by initializing a neural network with disconnected clusters and adding a minority of inter-cluster weights or filters.
6. The method of claim 1 comprising training the cluster-connected neural network by initializing a fully-connected neural network and pruning a majority of the inter-cluster weights or filters.
7. The method of claim 6 , wherein said pruning is performed during a training phase by biasing in favor of intra-cluster weights, and biasing against inter-cluster weights.
8. The method of claim 6 , wherein said pruning is performed using one or more techniques selected from the group consisting of: L 1 regularization, L p regularization, thresholding, random zero-ing, and bias based pruning.
9. The method of claim 1 , wherein the cluster-connected neural network is trained such that the strength of its weights or filters are biased inversely proportionally to the distance between the neurons or channels connected by the weights of filters.
10. The method of claim 1 comprising training the cluster-connected neural network using an evolutionary algorithm or reinforcement learning.
11. The method of claim 1 , wherein border neurons or channels in one cluster are connected by inter-cluster weights or filters to border neurons or channels in one or more different clusters, whereas interior neurons or channels spaced from the cluster border are only connected by intra-cluster weights or filters to other neurons or channels in the same cluster.
12. The method of claim 1 , wherein the neurons or channels in each cluster are fully-connected or partially connected.
13. The method of claim 1 , wherein the cluster-connected neural network is a hybrid of cluster-connected regions and standard non-cluster-connected regions.
14. The method of claim 1 comprising storing inter-cluster weights or filters in each channel of the cluster-connected neural network with an association to a unique cluster index, and using a cluster-specific matrix representing the intra-cluster weights in the cluster by their matrix positions.
15. The method of claim 1 comprising storing each of the plurality of inter-cluster weights or filters of the cluster-connected neural network with an association to a unique index, the unique index uniquely identifying a pair of artificial neurons or channels that have a connection represented by the inter-cluster weight or filter, wherein only non-zero inter-cluster weights or filters are stored that represent connections between pairs of neurons or channels in different clusters and zero inter-cluster weights or filters are not stored that represent no connections between pairs of neurons or channels.
16. The method of claim 15 comprising storing a triplet of values identifying each inter-cluster weight or filter comprising:
a first value of the unique index identifying a first neuron or channel of a pair of neurons or channels in a first cluster,
a second value of the unique index identifying a second neuron or channel of a pair of neurons or channels in a second different cluster, and
a value of the inter-cluster weight or filter.
17. The method of claim 15 comprising:
fetching inter-cluster weights or filters from a main memory that are stored in non-sequential locations in the main memory according to a non-sequential pattern of the indices associated with a sparse distribution of non-zero inter-cluster weights or filters in the cluster-connected neural network; and
storing the inter-cluster weights or filters fetched from non-sequential locations in the main memory to sequential locations in a cache memory.
18. The method of claim 15 comprising storing values of the inter-cluster weights or filters of the cluster-connected neural network using one or more data representations selected from the group consisting of: compressed sparse row (CSR) representation, compressed sparse column (CSC) representation, sparse tensor representation, map representation, list representation and sparse vector representation.
19. A system for training or prediction using a cluster-connected neural network at a local endpoint device, the system comprising:
one or more memories of the local endpoint device configured to store a neural network having a cluster-connected neural network axis in an orientation extending from an input layer to an output layer and orthogonal to a plurality of intermediate layers, wherein the cluster-connected neural network is divided into a plurality of clusters, wherein each cluster comprises a different plurality of artificial neurons or convolutional channels in a region extending parallel to the direction of the neural network axis resulting in a predominant direction of neuron activation extending from the input layer toward the output layer, wherein each pair of neurons or channels are uniquely connected by a weight or convolutional filter; and
one or more processors of the local endpoint device configured to:
within each cluster of the cluster-connected neural network, generate or maintain a locally dense sub-network of intra-cluster weights or filters, in which a majority of pairs of neurons or channels within the same cluster are connected by intra-cluster weights or filters, such that, the connected majority of pairs of neurons or channels in each cluster are co-activated together as an activation block during training or prediction using the cluster-connected neural network,
outside each cluster of the cluster-connected neural network, generate or maintain a globally sparse network of inter-cluster weights or filters, in which a minority of pairs of neurons or channels separated by a cluster border across different clusters are connected by inter-cluster weights or filters, and
perform prediction using the cluster-connected neural network at the local endpoint device.
20. The system of claim 19 , wherein the one or more processors are configured to test neuron or channel activation patterns in the cluster-connected neural network to determine an optimal cluster shape that most closely resembles activation patterns of highly linked neurons or channels resulting from the test.
21. The system of claim 20 , wherein the one or more processors are configured to dynamically adjust the optimal cluster shape as activation patterns change during training.
22. The system of claim 19 , wherein the cluster border of one or more of the plurality of clusters has a shape selected from the group consisting of: a column, row, circle, polygon, irregular shape, rectangular prism, cylinder, polyhedron, and another two-dimensional, three-dimensional, or N-dimensional shape.
23. The system of claim 19 , wherein the one or more processors are configured to train the cluster-connected neural network by initializing a neural network with disconnected clusters and adding a minority of inter-cluster weights or filters.
24. The system of claim 19 , wherein the one or more processors are configured to train the cluster-connected neural network by initializing a fully-connected neural network and pruning a majority of the inter-cluster weights or filters.
25. The system of claim 19 , wherein border neurons or channels in one cluster are connected by inter-cluster weights or filters to border neurons or channels in one or more different clusters, whereas interior neurons or channels spaced from the cluster border are only connected by intra-cluster weights or filters to other neurons or channels in the same cluster.
26. The system of claim 19 , wherein the one or more memories are configured to store inter-cluster weights or filters in each channel of the cluster-connected neural network with an association to a unique cluster index, and use a cluster-specific matrix representing the intra-cluster weights in the cluster by their matrix positions.
27. The system of claim 19 , wherein the one or more memories are configured to store each of the plurality of inter-cluster weights or filters of the cluster-connected neural network with an association to a unique index, the unique index uniquely identifying a pair of artificial neurons or channels that have a connection represented by the inter-cluster weight or filter, wherein only non-zero inter-cluster weights or filters are stored that represent connections between pairs of neurons or channels in different clusters and zero inter-cluster weights or filters are not stored that represent no connections between pairs of neurons or channels.
28. The system of claim 19 , wherein the one or more memories are configured to store a triplet of values identifying each inter-cluster weight or filter comprising:
a first value of the unique index identifying a first neuron or channel of a pair of neurons or channels in a first cluster,
a second value of the unique index identifying a second neuron or channel of a pair of neurons or channels in a second different cluster, and
a value of the inter-cluster weight or filter.
29. The system of claim 19 , wherein the one or more processors are configured to:
fetch inter-cluster weights or filters from a main memory that are stored in non-sequential locations in the main memory according to a non-sequential pattern of indices associated with a sparse distribution of non-zero inter-cluster weights or filters in the cluster-connected neural network, and
store the inter-cluster weights or filters fetched from non-sequential locations in the main memory to sequential locations in a cache memory.
30. The system of claim 19 , wherein the one or more memories are configured to store values of the inter-cluster weights or filters of the cluster-connected neural network using one or more data representations selected from the group consisting of: compressed sparse row (CSR) representation, compressed sparse column (CSC) representation, sparse tensor representation, map representation, list representation and sparse vector representation.