Mixture of generators model
Computer systems and computer-implemented methods train and/or operate, once trained, a machine-learning system that comprises a plurality of generator-detector pairs. The machine-learning computer system comprises a set of processor cores and computer memory that stores software. When executed by the set of processor cores, the software causes the set of processor cores to implement a plurality of generator-detector pairs, in which: (i) each generator-detector pair comprises a machine-learning data generator and a machine-learning data detector; and (ii) each generator-detector pair is for a corresponding cluster of data examples respectively, such that, for each generator-detector pair, the generator is for generating data examples in the corresponding cluster and the detector is for detecting whether data examples are within the corresponding cluster.
1. A machine-learning computer system for training a category classifier, wherein the category is modeled as having a plurality of clusters, including at least a first cluster and a second cluster, the machine-learning computer system comprising:
a set of processor cores: and
computer memory that stores software that, when executed by the set of processor cores, causes the set of processor cores to train, through machine learning, a plurality of generator-detector pairs, wherein:
the plurality of generator-detector pairs comprise at least a first generator-detector pair and a second generator-detector pair;
each of the plurality of generator-detector pairs comprises a generator and a detector, such that:
the first generator-detector pair comprises a first generator and a first detector; and
the second generator-detector pair comprises a second generator and a second detector;
each generator of each generator-detector pair comprises a decoder, wherein the decoder comprises a neural network, and such that:
the first generator comprises a first decoder; and
the second generator comprises a second decoder;
the detector of each generator-detector pair comprises a neural network that is different from the neural network of the decoder of the generator of the generator-detector pair;
each generator of the plurality of generator-detector pairs is trained, through machine learning, to generate generated data examples, respectively, in the plurality of clusters of the category, such that:
the first generator is trained, through machine learning, to generate generated data examples in the first cluster; and
the second generator is trained, through machine learning, to generate generated data examples in the second cluster;
each detector of the plurality of generator-detector pairs is trained, through machine learning, to determine, respectively, whether input data items to the detector are in one of the plurality of clusters of the category, such that:
the first detector is trained, through machine learning, to determine whether input data items to the first detector are in the first cluster of the category;
the second detector is trained, through machine learning, to determine whether input data items to the second detector are in the second cluster of the category; and
each detector is trained with, at least, the generated data examples generated by the generator of the generator-detector pair, such that:
the first detector is trained, at least partially, with the generated data examples from the first generator; and
the second detector is trained, at least partially, with the generated data examples from the second generator;
outputs of the generators of the generator-detector pairs are connected to inputs of the corresponding detector of the generator detector pairs, such that;
an output of the first decoder of the first generator of the first generator-detector pair is connected to an input of the first detector of the first generator-detector pair; and
an output of the second decoder of the second generator of the second generator-detector pair is connected to an input of the second detector of the second generator-detector pair; and
training the plurality of generator-detector pairs comprises, for each of the generator-detector pairs, back-propagating partial derivatives from the detector of the generator-detector pair to the generator of the generator-detector pair, such that:
partial derivatives of an error cost function for the first detector are back-propagated from the first detector to the first generator; and
partial derivatives of an error cost function for the second detector are back-propagated from the second detector to the second generator.
2. The machine-learning computer system of claim 1 , wherein, collectively, outputs of the plurality of generator-detector pairs, after training, are operable the category classifier.
3. The machine-learning computer system of claim 2 , wherein:
an output of the first generator, after training, is operable as a generator for the first cluster; and
an output of the second generator, after training, is operable as a generator for the second cluster.
4. The machine-learning computer system of claim 2 , wherein:
an output of the first detector, after training, is operable as a cluster classifier for the first cluster; and
an output of the second detector, after training, is operable as a cluster classifier for the second classifier.
5. The machine-learning computer system of claim 2 , wherein, when operated as the category classifier, an output of the machine-learning system corresponds to a maximum activation level for a detection among the detectors of the plurality of generator-detector pairs.
6. The machine-learning computer system of claim 1 , wherein each generator of the plurality of generator-detector pairs comprises a generator type selected from the group consisting of an autoencoder, a variational autoencoder, a stochastic categorical autoencoder network, and a generative adversarial network.
7. The machine-learning computer system of claim 1 , wherein each of the generators and each of the detectors are trained by stochastic gradient descent.
8. The machine-learning computer system of claim 1 , wherein each generator comprises an encoder.
9. A computer-implemented method for training a category classifier, wherein the category is modeled as having a plurality of clusters, including at least a first cluster and a second cluster, the method comprising:
training, by a computer system that comprises a set of processor cores, through machine learning, a plurality of generator-detector pairs of a machine-learning system such that:
the plurality of generator-detector pairs comprises at least a first generator-detector pair and a second generator-detector pair;
each generator-detector pair comprises a generator and a detector, such that;
the first generator-detector pair comprises a first generator and a first detector; and
the second generator-detector pair comprises a second generator and a second detector;
the generator of each generator-detector pair comprises a decoder, wherein each decoder comprises a neural network, and such that:
the first generator comprises a first decoder; and
the second generator comprises a second decoder;
the detector of each generator-detector pair comprises a neural network that is different from the neural network of the decoder of the generator of the generator-detector pair;
each generator of the plurality of generator-detector pairs is trained, through machine learning, to generate generated data examples, respectively, in the plurality of clusters of the category, such that:
the first generator is trained, through machine learning, to generate generated data examples in the first cluster; and
the second generator is trained, through machine learning, to generate generated data examples in the second cluster;
each detector of the plurality of generator-detector pairs is trained, through machine learning, to determine, respectively, whether input data items to the detector are in one of the plurality of clusters of the category, such that:
the first detector is trained, through machine learning, to determine whether input data items to the first detector are in the first cluster of the category;
the second detector is trained, through machine learning, to determine whether input data items to the second detector are in the second cluster of the category; and
each detector is trained with, at least, the generated data examples generated by the generator of the generator-detector pair, such that:
the first detector is trained, at least partially, with the generated data examples from the first generator; and
the second detector is trained, at least partially, with the generated data examples from the second generator;
an output of the decoder of each generator-detector pair is connected to an input of the detector of the generator-detector pair, such that:
an output of the first decoder of the first generator of the first generator-detector pair is connected to an input of the first detector of the first generator-detector pair; and
an output of the second decoder of the second generator of the second generator-detector pair is connected to an input of the second detector of the second generator-detector pair; and
training the plurality of generator-detector pairs comprises, for each of the generator-detector pairs, back-propagating partial derivatives from the detector of the generator-detector pair to the generator of the generator-detector pair, such that:
partial derivatives of an error cost function for the first detector are back-propagated from the first detector to the first generator; and
partial derivatives of an error cost function for the second detector are back-propagated from the second detector to the second generator.
10. The method of claim 9 , wherein training the plurality of generator-detector pairs comprises, for each of the plurality of generator-detector pairs:
with a plurality of data examples in the cluster corresponding to the generator-detector pair, initially training the generator without back-propagation from the corresponding detector of the generator-detector pair;
after initially training the generator:
generating, by the generator, within-cluster generated data examples that are within the cluster corresponding to the generator-detector pair;
classifying, by the detector, the within-cluster generated data examples generated by the generator; and
secondarily training the generator with back-propagation from the detector;
after training the generator with back-propagation from the detector, training the detector with within-cluster data examples and out-of-cluster data examples.
11. The method of claim 10 , wherein training the plurality of generator-detector pairs comprises training the plurality of generator-detector pairs such that the machine learning system is operable, after training, as a generator, as a cluster classifier, and as the category classifier, wherein:
when operated as a generator, a generator of the plurality of generator-detector pairs outputs generated data examples corresponding to the cluster for the generator;
when operated as a cluster classifier, the machine-learning system determines that a proper cluster for an input data example is the cluster corresponding to the detector of the plurality of generator-detector pairs with the with the greatest activation level for a detection; and
when operated as the category classifier, an output of the machine-learning system corresponds to a maximum activation level for a detection among the detectors of the plurality of generator-detector pairs.
12. The method of claim 10 , wherein each generator of the plurality of generator-detector pairs comprises a generator type selected from the group consisting of an autoencoder, a variational autoencoder, a stochastic categorical autoencoder network, and a generative adversarial network.
13. The method of claim 9 , wherein secondarily training the generator comprises secondarily training the generator by stochastic gradient descent.
14. The method of claim 11 , wherein each generator of the plurality of generator-detector pairs comprises a generator type selected from the group consisting of an autoencoder, a variational autoencoder, a stochastic categorical autoencoder network, and a generative adversarial network.
15. The method of claim 9 , wherein training the plurality of generator-detector pairs comprises training the first detector of the first generator-detector pair with negative examples of the first cluster.
16. The method of claim 15 , wherein the negative examples of the first cluster comprise generated data examples from the second generator.