Executing sublayers of a fully-connected layer
Disclosed herein are techniques for implementing a large fully-connected layer in an artificial neural network. The large fully-connected layer is grouped into multiple fully-connected subnetworks. Each fully-connected subnetwork is configured to classify an object into an unknown class or a class in a subset of target classes. If the object is classified as the unknown class by a fully-connected subnetwork, a next fully-connected subnetwork may be used to further classify the object. In some embodiments, the fully-connected layer is grouped based on a ranking of target classes.
1. A method of training an artificial neural network to classify objects into target classes, the method comprising:
ranking the target classes of the artificial neural network based on probabilities that the objects to be classified belong to the target classes, each of the target classes corresponding to one of a set of nodes on an output layer in the artificial neural network;
grouping the target classes and the set of nodes based on the ranking to form a plurality of groups of nodes, each group of nodes of the plurality of groups of nodes comprising a respective node corresponding to a respective unknown class and forming a respective subnetwork of a plurality of subnetworks, the plurality of subnetworks including a first subnetwork and a second subnetwork, wherein the probabilities for the target classes of the first subnetwork are greater than the probabilities for the target classes of the second subnetwork; and
training the plurality of subnetworks by:
labeling training samples as corresponding to one of the target classes of the first subnetwork or to the respective unknown class of the first subnetwork by labeling a first portion of the training samples that do not correspond to any of the target classes of the first subnetwork as corresponding to the respective unknown class of the first subnetwork;
training the first subnetwork using the training samples as labeled to correspond to the target classes or the respective unknown class of the first subnetwork;
labeling the training samples as corresponding to one of the target classes of the second subnetwork or to the respective unknown class of the second subnetwork by labeling a second portion of the training samples that do not correspond to any of the target classes of the second subnetwork as corresponding to the respective unknown class of the second subnetwork; and
training the second subnetwork using the training samples as labeled to correspond to the target classes or the respective unknown class of the second subnetwork.
2. The method of claim 1 , wherein the output layer is part of a fully-connected layer, and wherein each of the plurality of subnetworks is a fully-connected subnetwork.
3. The method of claim 1 , wherein a number of the target classes of the first subnetwork is equal to N/K−1, wherein N is a number of the set of nodes on the output layer and K is a number of the plurality subnetworks.
4. The method of claim 3 , wherein a number of the target classes of the second subnetwork is equal to N/K−1.
5. The method of claim 1 , further comprising:
prior to training the second subnetwork, removing the first portion of the training samples.
6. The method of claim 1 , wherein the plurality of subnetworks are trained sequentially, such that the first subnetwork is trained prior to the second subnetwork.
7. The method of claim 1 , wherein the artificial neural network comprises:
a multi-layer perceptron;
a convolutional neural network; or
a recurrent neural network.
8. The method of claim 1 , wherein the target classes comprise 10,000 or more target classes.
9. A non-transitory computer-readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
ranking target classes of an artificial neural network, each of the target classes corresponding to one of a set of nodes on an output layer in the artificial neural network based on probabilities that objects to be classified belong to the target classes;
grouping the target classes and the set of nodes based on the ranking to form a plurality of groups of nodes, each group of nodes of the plurality of groups of nodes comprising a respective node corresponding to a respective unknown class and forming a respective subnetwork of a plurality of subnetworks, the plurality of subnetworks including a first subnetwork and a second subnetwork, wherein the probabilities for the target classes of the first subnetwork are greater than the probabilities for the target classes of the second subnetwork; and
training the plurality of subnetworks by:
labeling training samples as corresponding to one of the target classes of the first subnetwork or to the respective unknown class of the first subnetwork by labeling a first portion of the training samples that do not correspond to any of the target classes of the first subnetwork as corresponding to the respective unknown class of the first subnetwork;
training the first subnetwork using the training samples as labeled to correspond to the target classes or the respective unknown class of the first subnetwork;
labeling the training samples as corresponding to one of the target classes of the second subnetwork or to the respective unknown class of the second subnetwork by labeling a second portion of the training samples that do not correspond to any of the target classes of the second subnetwork as corresponding to the respective unknown class of the second subnetwork; and
training the second subnetwork using the training samples as labeled to correspond to the target classes or the respective unknown class of the second subnetwork.
10. The non-transitory computer-readable medium of claim 9 , wherein the output layer is part of a fully-connected layer, and wherein each of the plurality of subnetworks is a fully-connected subnetwork.
11. The non-transitory computer-readable medium of claim 9 , wherein a number of the target classes of the first subnetwork is equal to N/K−1, wherein N is a number of the set of nodes on the output layer and K is a number of the plurality subnetworks.
12. The non-transitory computer-readable medium of claim 11 , wherein a number of the target classes of the second subnetwork is equal to N/K−1.
13. The non-transitory computer-readable medium of claim 9 , wherein the operations further comprise:
prior to training the second subnetwork, removing the first portion of the training samples.
14. The non-transitory computer-readable medium of claim 9 , wherein the plurality of subnetworks are trained sequentially, such that the first subnetwork is trained prior to the second subnetwork.
15. A system comprising:
one or more processors; and
a non-transitory computer-readable medium comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
ranking target classes of an artificial neural network, each of the target classes corresponding to one of a set of nodes on an output layer in the artificial neural network based on probabilities that objects to be classified belong to the target classes;
grouping the target classes and the set of nodes based on the ranking to form a plurality of groups of nodes, each group of nodes of the plurality of groups of nodes comprising a respective node corresponding to a respective unknown class and forming a respective subnetwork of a plurality of subnetworks, the plurality of subnetworks including a first subnetwork and a second subnetwork, wherein the probabilities for the target classes of the first subnetwork are greater than the probabilities for the target classes of the second subnetwork; and
training the plurality of subnetworks by:
labeling training samples as corresponding to one of the target classes of the first subnetwork or to the respective unknown class of the first subnetwork by labeling a first portion of the training samples that do not correspond to any of the target classes of the first subnetwork as corresponding to the respective unknown class of the first subnetwork;
training the first subnetwork using the training samples as labeled to correspond to the target classes or the respective unknown class of the first subnetwork;
labeling the training samples as corresponding to one of the target classes of the second subnetwork or to the respective unknown class of the second subnetwork by labeling a second portion of the training samples that do not correspond to any of the target classes of the second subnetwork as corresponding to the respective unknown class of the second subnetwork; and
training the second subnetwork using the training samples as labeled to correspond to the target classes or the respective unknown class of the second subnetwork.
16. The system of claim 15 , wherein the output layer is part of a fully-connected layer, and wherein each of the plurality of subnetworks is a fully-connected subnetwork.
17. The system of claim 15 , wherein a number of the target classes of the first subnetwork is equal to N/K−1, wherein N is a number of the set of nodes on the output layer and K is a number of the plurality subnetworks.
18. The system of claim 17 , wherein a number of the target classes of the second subnetwork is equal to N/K−1.
19. The system of claim 15 , wherein the operations further comprise:
prior to training the second subnetwork, removing the first portion of the training samples.