System for detection and management of uncertainty in perception systems
A perception system comprises a segmenting neural network and an uncertainty detector. The segmenting neural network is configured and trained for segmentation of an input image pixel matrix to generate a segment composed of elements that correspond the pixels of the input image pixel matrix. Each element of the segment map is assigned to one of a plurality of object classes the segmenting neural network is trained for by way of class prediction. Elements being assigned to the same object class form a segment of the segment map. The uncertainty detector is configured to generate an uncertainty score map that is composed of elements that correspond the pixels of the input image pixel matrix. Each element of the uncertainty map has an uncertainty score that is determined by the uncertainty detector and reflects an amount of uncertainty involved in a class prediction for a corresponding element in the segment map.
1 . A perception system, comprising:
a segmenting neural network that is configured and trained for segmentation of an input image pixel matrix to thus generate a segment map composed of elements that correspond to the pixels of the input image pixel matrix, each element of the segment map being assigned to one of a plurality of object classes the segmenting neural network is trained for by way of class prediction, elements being assigned to the same object class forming a segment of the segment map,
wherein the perception system further comprises an uncertainty detector that is configured to generate an uncertainty score map composed of elements that correspond to the pixels of the input image pixel matrix, each element of the uncertainty score map having an uncertainty score that is determined by the uncertainty detector and that reflects an amount of uncertainty involved in a class prediction for a corresponding element in the segment map,
wherein the uncertainty detector is configured to access feature score maps generated by the segmenting neural network prior to generating the segment map from said feature score maps and to determine the amount of uncertainty and thus the uncertainty score for each element of the uncertainty score map by determining a variance of the activation levels of elements of the feature score maps, and
wherein the uncertainty detector is configured to determine the amount of uncertainty and thus the uncertainty score for each element of the uncertainty score map by determining an inter-sample variance of the activation levels of an element of a feature score map in different samples of the feature score map.
2 . The perception system according to claim 1 , wherein the uncertainty detector is configured to generate inter-sample variances of the activation levels of an element of a feature score map in different samples of the feature score map by processing an input image pixel matrix with the segmenting neural network in multiple passes while for each pass the segmenting neural network is randomly modified.
3 . The perception system according to claim 2 , wherein the random modification of the segmenting neural network includes at least one of dropping nodes from a convolutional layer of the segmenting neural network, altering activation functions in at least some nodes of at least some layers of the segmenting neural network and/or introducing noise in at least some nodes of at least some layers of the segmenting neural network.
4 . The perception system according to claim 1 , wherein the uncertainty detector is configured to determine an inter-sample variance of the activation levels of corresponding elements of feature score maps that are generated from consecutive image pixel matrices corresponding to frames of a video sequence.
5 . The perception system according to claim 1 , wherein the uncertainty detector comprises a generative neural network, in particular a variational autoencoder that is trained for the plurality of object classes, the segmenting neural network is trained for, wherein a reconstruction loss between the input image pixel matrix and a predicted image pixel matrix reconstructed by the generative neural network corresponds to the amount of uncertainty.
6 . The perception system according to claim 1 , wherein the uncertainty detector is configured for:
detecting regions in the uncertainty score map that are composed of elements having a high uncertainty score; and
labeling those regions that are composed of elements having a high uncertainty score as candidates for representing an object of a yet unknown object class.
7 . A method for segmentation of input image pixel matrices by way of class prediction performed by a segmenting neural network and for determining an amount of uncertainty involved in the class predictions for each pixel of an input image pixel matrix, said method comprising:
segmenting an input image pixel matrix by means of a segmenting neural network that is trained for a plurality of object classes and that generates for each object class a feature score map and therefrom a segment map for the input image pixel matrix by assigning elements of the segment map to one of a plurality of object classes the segmenting neural network is trained for by way of class prediction, elements being assigned to the same object class forming a segment of the segment map; and
generating an uncertainty score map composed of elements that correspond the pixels of the input image pixel matrix, each element of the uncertainty score map having an uncertainty score that is determined by the uncertainty detector and that reflects an amount of uncertainty involved in a class prediction for a corresponding element in the segment map,
wherein the uncertainty score is determined by determining an inter-class variance of the activation levels of elements of the feature score maps for different object classes.
8 . The method according to claim 7 , further comprising:
detecting a region in the uncertainty score map that are composed of elements having a high uncertainty score; and
labeling the region that is composed of elements having a high uncertainty score as candidates for representing an object of a yet unknown object class, a high uncertainty score being an uncertainty score that is higher than an average uncertainty score of all elements of the uncertainty score map.
9 . The method according to claim 8 , further comprising:
creating a new object class if a region that is composed of elements having a high uncertainty score is detected, said new object class representing objects as shown in a region of the input image pixel matrix corresponding the region in the uncertainty score map that are composed of elements having a high uncertainty score.
10 . The method according to claim 7 , wherein the method is used in an autonomous vehicle.
11 . A perception system, comprising:
a segmenting neural network that is configured and trained for segmentation of an input image pixel matrix to thus generate a segment map composed of elements that correspond to the pixels of the input image pixel matrix, each element of the segment map being assigned to one of a plurality of object classes the segmenting neural network is trained for by way of class prediction, elements being assigned to the same object class forming a segment of the segment map,
wherein the perception system further comprises an uncertainty detector that is configured to generate an uncertainty score map composed of elements that correspond to the pixels of the input image pixel matrix, each element of the uncertainty score map having an uncertainty score that is determined by the uncertainty detector and that reflects an amount of uncertainty involved in a class prediction for a corresponding element in the segment map, and
wherein the uncertainty detector is configured to determine the amount of uncertainty and thus the uncertainty score for each element of the uncertainty score map by determining inter-class variances between activation levels of corresponding elements of feature score maps for different object classes as provided by the segmenting neural network.
12 . A method for segmentation of input image pixel matrices by way of class prediction performed by a segmenting neural network and for determining an amount of uncertainty involved in the class predictions for each pixel of an input image pixel matrix, said method comprising:
segmenting an input image pixel matrix by means of a segmenting neural network that is trained for a plurality of object classes and that generates for each object class a feature score map and therefrom a segment map for the input image pixel matrix by assigning elements of the segment map to one of a plurality of object classes the segmenting neural network is trained for by way of class prediction, elements being assigned to the same object class forming a segment of the segment map; and
generating an uncertainty score map composed of elements that correspond to the pixels of the input image pixel matrix, each element of the uncertainty score map having an uncertainty score that is determined by the uncertainty detector and that reflects an amount of uncertainty involved in a class prediction for a corresponding element in the segment map,
wherein the uncertainty score is determined by determining an inter-sample variance of the activation levels of elements of different samples of a feature score map for one object class.