Training an autoencoder with a classifier
Computer systems and methods train a deep neural network through machine learning. In response to detection of a training condition, computer system replaces a target node of the network with a split detector compound node, where, prior to replacement, the target node detected a pattern that activated the target node beyond a specified threshold. The split detector compound node comprises first and second nodes, such that: the first node is activated when significant evidence exists in favor of detection of the pattern in inputs to the first node; and the second node is activated when significant evidence exists against detection of the pattern in inputs to the second node, such that activations of the first and second nodes are computed independently. After replacing the target node with the split detector compound node, training of the network through machine learning is resumed.
1. A method of training an autoencoder via machine learning, wherein the autoencoder comprises an encoder and a decoder, wherein each of the encoder and the decoder comprise a deep neural network, the method comprising:
processing, with a programmed computer system that comprises one or more processing units, each of multiple labeled training data examples in a labeled training dataset, wherein each of the multiple labeled training data examples in the labeled training dataset has a corresponding label, and wherein the processing comprises:
in a forward propagation phase:
generating, by the encoder, an encoder output;
generating, by the decoder, an output from the encoder output for the labeled training data example, wherein the decoder is trained, for each labeled training data example, to generate the labeled training data example from the encoder output and wherein the decoder has a first objective function; and
generating, by a classifier, a classification output from the encoder output for the labeled training data example, wherein the classifier is trained, for each labeled training data example, to generate a corresponding label for the labeled training data example, wherein the classifier has a second objective function, and wherein the classifier comprises a deep neural network; and
after the forward propagation phase, in a back propagation phase:
generating, by the decoder, a first set of partial derivatives of a first cost function for the first objective through the decoder;
generating, by the classifier, a second set of partial derivatives of a second cost function for the second objective through the classifier; and
generating, by the encoder, a third set of partial derivatives of a third cost function with respect to both the first and second objectives through the encoder; and
updating, by the computer system, learned parameters for the encoder based on the third set of partial derivatives.
2. The method of claim 1 , wherein updating the learned parameters for the encoder comprises updating the learned parameters after processing each of the multiple labeled training data examples.
3. The method of claim 1 , further comprises further training, by the programmed computer system, the encoder with multiple unlabeled training data examples in an unlabeled training dataset, wherein the further training comprises:
processing each of multiple unlabeled training, wherein the processing comprises in a forward propagation phase:
generating, by the encoder, an encoder output; and
generating, by the decoder, an output from the encoder output for the labeled training data example, wherein the decoder is trained, for each labeled training data example, to generate the labeled training data example from the encoder output and wherein the decoder has a first objective function; and
after the forward propagation phase, in a back propagation phase:
generating, by the decoder, a fourth set of partial derivatives of the first cost function for the first objective through the decoder; and
generating, by the encoder, a fifth set of partial derivatives of the third cost function with respect to the first objective through the encoder, such that the first set of partial derivatives are generated without backpropagation from the classifier for the unlabeled training data examples; and
updating, by the programmed computer system, learned parameters for the encoder based on the fifth set of partial derivatives.
4. The method of claim 1 , further comprising:
storing, in a repository of the programmed computer system, the encoder after updating the learned parameters for the encoder;
adding, by the programmed computer system, the encoder to an existing neural network, wherein the existing neural network comprises an input layer, an output layer, and one or more hidden layers between the input and output layers, wherein the one or more hidden layers comprises a first hidden layer, and adding the encoder to the existing neural network comprises having outputs from the encoder being input to the first hidden layer of the existing neural network; and
training, by the programmed computer system, via machine learning, the existing neural network with the encoder added.
5. The method of claim 4 , wherein:
adding the encoder comprises:
connecting, with directed arcs, nodes in an output layer of the encoder to nodes of the first hidden layer, wherein the directed arcs; and
initializing the directed arcs to have weights of zero prior to training the existing neural network with the encoder added; and
training the existing neural network with the encoder added comprises learning, via machine learning, updated weights for the directed arcs.
6. The method of claim 4 , wherein:
storing the encoder further comprising storing the decoder in the repository;
adding the encoder to the existing neural network further comprises adding to the decoder to the existing neural network, such that the outputs of the encoder are input to the decoder; and
training the existing neural network with the encoder added comprises training the existing neural network with both the encoder and decoder added.
7. The method of claim 6 , wherein:
storing the encoder further comprising storing the classifier in the repository;
adding the encoder to the existing neural network further comprises adding to the classifier to the existing neural network, such that the outputs of the encoder are input to the classifier; and
training the existing neural network with the encoder added comprises training the existing neural network with the encoder, decoder and classifier added.
8. The method of claim 3 , further comprising:
storing, in a repository, the encoder after updating the learned parameters for the encoder;
adding, by the programmed computer system, the encoder to an existing neural network, wherein the existing neural network comprises an input layer, an output layer, and one or more hidden layers between the input and output layers, wherein the one or more hidden layers comprises a first hidden layer, and adding the encoder to the existing neural network comprises having outputs from the encoder being input to the first hidden layer of the existing neural network; and
training, by the programmed computer system, via machine learning, the existing neural network with the encoder added.
9. The method of claim 8 , wherein:
adding the encoder comprises:
connecting, with directed arcs, nodes in an output layer of the encoder to nodes of the first hidden layer, wherein the directed arcs; and
initializing the directed arcs to have weights of zero prior to training the existing neural network with the encoder added; and
training the existing neural network with the encoder added comprises learning, via machine learning, updated weights for the directed arcs.
10. The method of claim 8 , wherein:
storing the encoder further comprising storing the decoder in the repository;
adding the encoder to the existing neural network further comprises adding the decoder to the existing neural network, such that the outputs of the encoder are input to the decoder; and
training the existing neural network with the encoder added comprises training the existing neural network with both the encoder and decoder added.
11. The method of claim 10 , wherein:
storing the encoder further comprising storing the classifier in the repository;
adding the encoder to the existing neural network further comprises adding the classifier to the existing neural network, such that the outputs of the encoder are input to the classifier; and
training the existing neural network with the encoder added comprises training the existing neural network with the encoder, decoder and classifier added.
12. The method of claim 4 , wherein the repository comprises a nonvolatile computer memory.
13. The method of claim 4 , wherein training the existing neural network with the encoder added comprises, for a training data item in the training of the existing neural network with the encoder added that was used in training the autoencoder, regularizing the outputs of the encoder in existing neural network to agree with outputs of the encoder in the training of the autoencoder.
14. The method of claim 4 , wherein training the existing neural network with the encoder added comprises reducing a learning rate for the outputs of the encoder.
15. The method of claim 1 , wherein:
the one or more processing units comprises two or more graphical processing units (GPUs); and
processing each of the multiple labeled training data examples in the training dataset comprises processing in parallel each of the multiple labeled training data examples in the training dataset with the two or more GPUs.
16. A computer system comprising:
one or more processing units; and
computer memory in communication with the one or more processing units, wherein the computer memory stores computer instructions that when executed by the one or more processing units cause the one or more processing units to train an autoencoder via machine learning, wherein the autoencoder comprises an encoder and a decoder, wherein each of the encoder and the decoder comprise a deep neural network, wherein the computer instructions cause the one or more processing units to train the autoencoder via steps that comprise:
processing each of multiple labeled training data examples in a labeled training dataset, wherein each of the multiple labeled training data examples in the labeled training dataset has a corresponding label, and wherein the processing comprises:
in a forward propagation phase:
generating, by the encoder, an encoder output;
generating, by the decoder, an output from the encoder output for the labeled training data example, wherein the decoder is trained, for each labeled training data example, to generate the labeled training data example from the encoder output and wherein the decoder has a first objective function; and
generating, by a classifier, a classification output from the encoder output for the labeled training data example, wherein the classifier is trained, for each labeled training data example, to generate a corresponding label for the labeled training data example, wherein the classifier has a second objective function, and wherein the classifier comprises a deep neural network; and
after the forward propagation phase, in a back propagation phase:
generating, by the decoder, a first set of partial derivatives of a first cost function for the first objective through the decoder;
generating, by the classifier, a second set of partial derivatives of a second cost function for the second objective through the classifier; and
generating, by the encoder, a third set of partial derivatives of a third cost function with respect to both the first and second objectives through the encoder; and
updating learned parameters for the encoder based on the third set of partial derivatives.
17. The computer system of claim 16 , wherein the computer memory stores computer instructions that when executed by the one or more processing units cause the one or more processing units to update the learned parameters for the encoder by updating the learned parameters after processing each of the multiple labeled training data examples.
18. The computer system of claim 16 , wherein the computer memory stores computer instructions that when executed by the one or more processing units cause the one or more processing units further train the encoder with multiple unlabeled training data examples in an unlabeled training dataset, wherein the further training comprises:
processing each of multiple unlabeled training, wherein the processing comprises in a forward propagation phase:
generating, by the encoder, an encoder output; and
generating, by the decoder, an output from the encoder output for the labeled training data example, wherein the decoder is trained, for each labeled training data example, to generate the labeled training data example from the encoder output and wherein the decoder has a first objective function; and
after the forward propagation phase, in a back propagation phase:
generating, by the decoder, a fourth set of partial derivatives of the first cost function for the first objective through the decoder; and
generating, by the encoder, a fifth set of partial derivatives of the third cost function with respect to the first objective through the encoder, such that the first set of partial derivatives are generated without backpropagation from the classifier for the unlabeled training data examples; and
updating learned parameters for the encoder based on the fifth set of partial derivatives.
19. The computer system of claim 16 , the computer memory stores computer instructions that when executed by the one or more processing units cause the one or more processing units to:
store, in a repository of the computer system, the encoder after updating the learned parameters for the encoder;
add the encoder to an existing neural network, wherein the existing neural network comprises an input layer, an output layer, and one or more hidden layers between the input and output layers, wherein the one or more hidden layers comprises a first hidden layer, and adding the encoder to the existing neural network comprises having outputs from the encoder being input to the first hidden layer of the existing neural network; and
train, via machine learning, the existing neural network with the encoder added.
20. The computer system of claim 19 , wherein the computer memory stores computer instructions that when executed by the one or more processing units cause the one or more processing units to:
add the encoder by:
connecting, with directed arcs, nodes in an output layer of the encoder to nodes of the first hidden layer, wherein the directed arcs; and
initializing the directed arcs to have weights of zero prior to training the existing neural network with the encoder added; and
train the existing neural network with the encoder added by learning, via machine learning, updated weights for the directed arcs.
21. The computer system of claim 19 , wherein the computer memory stores computer instructions that when executed by the one or more processing units cause the one or more processing units to:
store the decoder in the repository;
add the decoder to the existing neural network, such that the outputs of the encoder are input to the decoder; and
train the existing neural network with the both the encoder and decoder added.
22. The computer system of claim 21 , wherein the computer memory stores computer instructions that when executed by the one or more processing units cause the one or more processing units to:
store the classifier in the repository;
add the classifier to the existing neural network, such that the outputs of the encoder are input to the classifier; and
train the existing neural network with encoder, decoder and classifier added.
23. The computer system of claim 18 , wherein the computer memory stores computer instructions that when executed by the one or more processing units cause the one or more processing units to:
store, in a repository, the encoder after updating the learned parameters for the encoder;
add the encoder to an existing neural network, wherein the existing neural network comprises an input layer, an output layer, and one or more hidden layers between the input and output layers, wherein the one or more hidden layers comprises a first hidden layer, and adding the encoder to the existing neural network comprises having outputs from the encoder being input to the first hidden layer of the existing neural network; and
train, via machine learning, the existing neural network with the encoder added.
24. The computer system of claim 23 , wherein the computer memory stores computer instructions that when executed by the one or more processing units cause the one or more processing units to:
add the encoder by:
connecting, with directed arcs, nodes in an output layer of the encoder to nodes of the first hidden layer, wherein the directed arcs; and
initializing the directed arcs to have weights of zero prior to training the existing neural network with the encoder added; and
train the existing neural network with the encoder added by learning, via machine learning, updated weights for the directed arcs.
25. The computer system of claim 23 , wherein the computer memory stores computer instructions that when executed by the one or more processing units cause the one or more processing units to:
store the decoder in the repository;
add the decoder to the existing neural network, such that the outputs of the encoder are input to the decoder; and
training the existing neural network with both the encoder and decoder added.
26. The computer system of claim 25 , wherein the computer memory stores computer instructions that when executed by the one or more processing units cause the one or more processing units to:
store the classifier in the repository;
add the classifier to the existing neural network, such that the outputs of the encoder are input to the classifier; and
train the existing neural network with the encoder, decoder and classifier added.
27. The computer system of claim 19 , wherein the computer memory stores computer instructions that when executed by the one or more processing units cause the one or more processing units to train the existing neural network with the encoder added by, for a training data item in the training of the existing neural network with the encoder added that was used in training the autoencoder, regularizing the outputs of the encoder in existing neural network to agree with outputs of the encoder in the training of the autoencoder.
28. The computer system of claim 16 , wherein:
the one or more processing units comprises two or more graphical processing units (GPUs); and
each of the two or more GPUs are for processing in parallel each of the multiple labeled training data examples in the training dataset.