Granted Patent B2

US 12,450,891 · App. 17/345,702 · Granted Oct 21, 2025

Inventors: Didrik Nielsen (København K, DK); Emiel Hoogeboom (Amsterdam, NL); Kaspar Sakmann (Stuttgart, DE); Max Welling (Amsterdam, NL); Priyank Jaini (Amsterdam, NL)

Assignee: ROBERT BOSCH GMBH

G06V10/82G06F18/211G06F18/2155G06F18/2415G06F18/2431G06F18/25G06N3/048G06N3/08G06V10/764G06V10/7715G06V10/776

Abstract

Claims (81)

1. A computer-implemented method of training an image classifier, the image classifier being configured to classify an input image into a class from a set of classes, the method comprising the following steps:

accessing a training dataset, the training dataset including at least one labelled training image labelled with a training class from the set of classes and at least one unlabelled training image;

defining an inverse model for the image classifier, the inverse model configured to map output classes of the image classifier to input images, wherein the image classifier includes a set of transformations, the set of transformations including at least one deterministic and non-injective transformation, an inverse of the deterministic and non-injective transformation being approximated in the inverse model by a stochastic inverse transformation, the inverse model including trainable parameters, and sharing a plurality of the trainable parameters with the image classifier;

training the image classifier using a log-likelihood optimization, the training including:

selecting a training image from the training dataset,

applying the image classifier to the training image, including applying the deterministic and non-injective transformation to transformation inputs of the deterministic and non-injective transformation to obtain transformation outputs of the deterministic and non-injective transformation,

determining a likelihood contribution for the deterministic and non-injective transformation of the image classifier based on a probability that the stochastic inverse transformation of the inverse model generates the transformation inputs given the transformation outputs,

when the training image is the labelled training image, using the likelihood contribution to determine a log-likelihood for the labelled training image and its label according to a joint probability distribution of input images and classes determined by the image classifier,

when the training image is the unlabelled training image, using the determined likelihood contribution to determine a log-likelihood for the unlabelled training image according to a probability distribution of input images being generated by the inverse model; and

optimizing parameters of the image classifier to maximize the log-likelihood for the labelled training image occurring according to the joint probability distribution and optimizing parameters of the inverse model to maximize the log-likelihood for the unlabelled training image for the unlabelled training image occurring according to the probability distribution of input images being generated by the inverse model.

2. The method of claim 1 , wherein the determining of the log-likelihood for the training image includes determining a sum of likelihood contributions for respective transformations of the set of transformations.

3. The method of claim 2 , wherein the image classifier includes a densely connected component given by a linear bijective transformation and a slicing transformation, wherein the slicing transformation is configured to select a subset of outputs of the linear bijective transformation, wherein an inverse of the slicing transformation is approximated in the inverse model by a stochastic inverse transformation configured to sample non-selected outputs for the linear bijective transformation based on the selected outputs of the linear bijective transformation.

4. The method of claim 2 , wherein the image classifier includes a coupling transformation configured to determine first and second transformation outputs given first and second transformation inputs by combining the first transformation input with a first function of the second transformation input to obtain the first transformation output, and combining the second transformation input with a second function of the first transformation output, wherein the first and second functions are convolutions.

5. The method of claim 1 , wherein the image classifier includes a max pooling transformation computing a transformation output as a maximum of multiple transformation inputs, wherein an inverse of the max pooling transformation is approximated in the inverse model by an inverse transformation configured to sample an index of a maximal transformation input and values of non-maximal transformation inputs.

6. The method of claim 1 , wherein the image classifier includes a ReLU transformation configured to compute a transformation output by mapping a transformation input from an interval to a constant, wherein an inverse of the ReLU transformation is approximated in the inverse model by an inverse transformation configured to, given a transformation output equal to the given constant, sample a transformation input from the given interval.

7. The method of claim 1 , wherein the image classifier is configured to classify the input image into the class by determining a vector of class probabilities for respective classes, and, in an output layer, determining the class from the vector, wherein an inverse of the output layer is approximated in the inverse model based on a conditional probability distribution for the vector of class probabilities given the determined class.

8. The method of claim 1 , wherein the image classifier further includes a stochastic transformation with a deterministic inverse transformation, and the method further comprises computing a likelihood contribution of the stochastic transformation based on a probability that the deterministic and non-injective transformation generates the transformation outputs of the stochastic transformation given the transformation inputs of the stochastic transformation.

9. The method of claim 1 , further comprising:

obtaining an input image data using a sensor of an at least semi-autonomous vehicle;

classifying the input image data using the trained image classifier, including applying the trained image classifier to classify the input image into a class from the set of classes; and

controlling steering and/or braking of the vehicle based on the classification of the input image data.

10. The method of claim 1 , further comprising:

obtaining a target class;

applying the trained inverse model to the target class to generate, using the trained inverse model, multiple synthetic images representative of the target class; and

further training the trained image classifier using the generated synthetic images.

11. The method of claim 1 , further comprising:

obtaining a target class;

applying the trained inverse model to the target class to generate, using the trained inverse model, multiple synthetic images representative of the target class; and

training a machine learning model using the generated synthetic images.

12. A computer-implemented method of using a trained image classifier, the trained image classifier being configured to classify an input image into a class from a set of classes, the method comprising the following steps:

accessing model data representing the trained image classifier, wherein the image classifier is trained by:

accessing a training dataset, the training dataset including at least one labelled training image labelled with a training class from the set of classes and at least one unlabelled training image,

defining an inverse model for the image classifier, the inverse model configured to map output classes of the image classifier to input images, wherein the image classifier includes a set of transformations, the set of transformations including at least one deterministic and non-injective transformation, an inverse of the deterministic and non-injective transformation being approximated in the inverse model by a stochastic inverse transformation, the inverse model including trainable parameters, and sharing a plurality of the trainable parameters with the image classifier,

training the image classifier using a log-likelihood optimization, the training including:

selecting a training image from the training dataset,

applying the image classifier to the training image, including applying the deterministic and non-injective transformation to transformation inputs of the deterministic and non-injective transformation to obtain transformation outputs of the deterministic and non-injective transformation,

determining a likelihood contribution for the deterministic and non-injective transformation of the image classifier based on a probability that the stochastic inverse transformation of the inverse model generates the transformation inputs given the transformation outputs,

when the training image is the labelled training image, using the likelihood contribution to determine a log-likelihood for the labelled training image and its label according to a joint probability distribution of input images and classes determined by the image classifier,

when the training image is the unlabelled training image, using the determined likelihood contribution to determine a log-likelihood for the unlabelled training image according to a probability distribution of input images being generated by the inverse model, and

optimizing parameters of the image classifier to maximize the log-likelihood for the labelled training image occurring according to the joint probability distribution and optimizing parameters of the inverse model to maximize the log-likelihood for the unlabelled training image for the unlabelled training image occurring according to the probability distribution of input images being generated by the inverse model;

using the trained image classifier by:

obtaining an input image and applying the trained image classifier to classify the input image into a class from the set of classes, and/or

applying the inverse model for the trained image classifier to generate a synthetic image, including sampling transformation inputs of the deterministic and non-injective transformation of the trained model based on transformation outputs of the deterministic and non-injective transformation according to the stochastic inverse transformation.

13. The method of claim 12 , further comprising obtaining a target class, and applying the inverse model based on the target class to generate a synthetic image representative of the target class.

14. The method of claim 12 , further comprising generating multiple images by applying the inverse model, and using the generated images as training and/or test data to train a machine learning model.

15. The method of claim 12 , wherein the image classifier is further configured to determine a confidence score of the classification into the class.

16. A system for training an image classifier, the image classifier being configured to classify an input image into a class from a set of classes, the system comprising:

a data interface configured to access a training dataset, the training dataset including at least one labelled training image labelled with a training class from the set of classes and at least one unlabelled training image;

a processor subsystem configured to:

define an inverse model for the image classifier, the inverse model configured to map output classes of the image classifier to input images, wherein the image classifier includes a set of transformations, the set of transformations comprising at least one deterministic and non-injective transformation, an inverse of the deterministic and non-injective transformation being approximated in the inverse model by a stochastic inverse transformation, the inverse model including trainable parameters parameters, and sharing a plurality of the trainable parambers with the image classifier;

train the image classifier using a log-likelihood optimization, the training including:

selecting a training image from the training dataset;

applying the image classifier to the training image, including applying the deterministic and non-injective transformation to transformation inputs of the deterministic and non-injective transformation to obtain transformation outputs of the deterministic and non-injective transformation;

determining a likelihood contribution for the deterministic and non-injective transformation of the image classifier based on a probability that the stochastic inverse transformation of the inverse model generates the transformation inputs given the transformation outputs;

when the training image is a labelled training image, using the determined likelihood contribution to determine a log-likelihood for the labelled training image and its label according to a joint probability distribution of input images and classes determined by the image classifier;

when the training image is an unlabelled training image, using the determined likelihood contribution to determine a log-likelihood for the unlabelled training image according to a probability distribution of input images being generated by the inverse model; and

optimizing parameters of the image classifier to maximize the log-likelihood for the labelled training image occurring according to the joint probability distribution and optimizing parameters of the inverse model to maximize the log-likelihood for the unlabelled training image for the unlabelled training image occurring according to the probability distribution of input images being generated by the inverse model.

17. A system for using a trained image classifier, the trained image classifier being configured to classify an input image into a class from a set of classes, the system comprising:

a data interface configured to access model data representing the trained image classifier, wherein the image classifier is trained by:

accessing a training dataset, the training dataset including at least one labelled training image labelled with a training class from the set of classes and at least one unlabelled training image;

defining an inverse model for the image classifier, the inverse model configured to map output classes of the image classifier to input images, wherein the image classifier includes a set of transformations, the set of transformations including at least one deterministic and non-injective transformation, an inverse of the deterministic and non-injective transformation being approximated in the inverse model by a stochastic inverse transformation, the inverse model including trainable parameters, and sharing a plurality of the trainable parameter with the image classifier;

training the image classifier using a log-likelihood optimization, the training including:

selecting a training image from the training dataset,

applying the image classifier to the training image, including applying the deterministic and non-injective transformation to transformation inputs of the deterministic and non-injective transformation to obtain transformation outputs of the deterministic and non-injective transformation,

determining a likelihood contribution for the deterministic and non-injective transformation of the image classifier based on a probability that the stochastic inverse transformation of the inverse model generates the transformation inputs given the transformation outputs,

when the training image is the labelled training image, using the likelihood contribution to determine a log-likelihood for the labelled training image and its label according to a joint probability distribution of input images and classes determined by the image classifier,

when the training image is the unlabelled training image, using the determined likelihood contribution to determine a log-likelihood for the unlabelled training image according to a probability distribution of input images being generated by the inverse model, and

optimizing parameters of the image classifier to maximize the log-likelihood for the labelled training image occurring according to the joint probability distribution and optimizing parameters of the inverse model to maximize the log-likelihood for the unlabelled training image for the unlabelled training image occurring according to the probability distribution of input images being generated by the inverse model; and

a processor subsystem configured to use the trained image classifier by:

obtaining an input image and applying the trained image classifier to classify the input image into a class from the set of classes, and/or

applying the inverse model for the trained image classifier to generate a synthetic image, including sampling transformation inputs of the deterministic and non-injective transformation of the trained model based on transformation outputs of the deterministic and non-injective transformation according to a stochastic inverse transformation.

18. A non-transitory computer-readable medium on which is stored instructions training an image classifier, the image classifier being configured to classify an input image into a class from a set of classes, the instructions, when executed by a processor system, causing the processor system to perform the following steps:

accessing a training dataset, the training dataset including at least one labelled training image labelled with a training class from the set of classes and at least one unlabelled training image;

defining an inverse model for the image classifier, the inverse model configured to map output classes of the image classifier to input images, wherein the image classifier includes a set of transformations, the set of transformations including at least one deterministic and non-injective transformation, an inverse of the deterministic and non-injective transformation being approximated in the inverse model by a stochastic inverse transformation, the inverse model including trainable parameters, and sharing a plurality of trainable parameters with the image classifier;

training the image classifier using a log-likelihood optimization, the training including:

selecting a training image from the training dataset,

applying the image classifier to the training image, including applying the deterministic and non-injective transformation to transformation inputs of the deterministic and non-injective transformation to obtain transformation outputs of the deterministic and non-injective transformation,

determining a likelihood contribution for the deterministic and non-injective transformation of the image classifier based on a probability that the stochastic inverse transformation of the inverse model generates the transformation inputs given the transformation outputs,

when the training image is the labelled training image, using the likelihood contribution to determine a log-likelihood for the labelled training image and its label according to a joint probability distribution of input images and classes determined by the image classifier,

when the training image is the unlabelled training image, using the determined likelihood contribution to determine a log-likelihood for the unlabelled training image according to a probability distribution of input images being generated by the inverse model, and

optimizing parameters of the image classifier to maximize the log-likelihood for the labelled training image occurring according to the joint probability distribution and optimizing parameters of the inverse model to maximize the log-likelihood for the unlabelled training image for the unlabelled training image occurring according to the probability distribution of input images being generated by the inverse model.

Assignments (1)

Priority Claims (1)

Continuity (1)

References Cited (34)