Instance segmentation by instance label factorization
A computer system trains a neural network on an instance segmentation task by casting the problem as one of mapping each pixel to a probability distribution over arbitrary instance labels. This simplifies both the training and inference problems, because the formulation is end-to-end trainable and requires no post-processing to extract maximum a posteriori estimates of the instance labels.
1. A method for training an artificial neural network, performed by at least one computer processor executing computer program instructions stored on at lea one non-transitory computer-readable medium, the method comprising:
(A) receiving a plurality of images and a plurality of sets of corresponding ground truth instance labels, wherein each of the ground truth instance labels associates a region of an image with a specific instance of an object;
(B) receiving a training objective, wherein the training objective specifies that a divergence of a probability that a first sample and a second sample correspond to the same instance label as each other in the plurality of sets of corresponding ground truth instance labels is to be minimized, wherein:
the first sample corresponds to a first pixel in an image;
a first pixel label distribution comprises the first sample;
the second sample corresponds to a second pixel in the image;
a second pixel label distribution comprises the second sample; and
(C) training the artificial neural network to minimize a loss function based on the plurality of images, the plurality of sets of corresponding ground truth instance labels, and the training objective, comprising:
determining an injective mapping g from ground truth instance labels T onto labels L predicted by the artificial neural network to minimize loss of the artificial neural network;
applying the injective mapping g during a forward pass of the artificial neural network, wherein the loss function after applying the injective mapping is permutation-invariant with respect to the labels L predicted by the artificial neural network; and
backpropagating a loss gradient through the injective mapping g to update the artificial neural network.
2. The method of claim 1 , further comprising:
(D) applying the artificial neural network, after the training, to a particular image comprising a plurality of pixels, to produce a categorical probability distribution of the plurality of pixels over a plurality of instance labels.
3. The method of claim 2 , wherein the categorical probability distribution maps a first one of the plurality of pixels to a first instance of a first class and maps a second one of the plurality of pixels to a second instance of the first class.
4. The method of claim 2 , wherein the categorical probability distribution maps a first one of the plurality of pixels to a first instance of a first class and maps a second one of the plurality of pixels to a first instance of a second class.
5. The method of claim 1 , wherein (C) comprises training the artificial neural network by minimizing a divergence of permutation-invariant auxiliary distributions derived from the ground truth instance labels and network distributions.
6. The method of claim 1 , wherein the artificial neural network comprises a Fully Convolutional Neural Network.
7. The method of claim 6 , wherein the Fully Convolutional Neural Network is configured to directly output a probability distribution by using a Softmax activation.
8. The method of claim 1 , wherein the training in (C) enforces that L, the probability distribution over N instance labels, factorizes over H×W independent categorical distributions.
9. A system for training an artificial neural network, the system comprising at least one non-transitory computer-readable medium comprising computer program instructions executable by at least one computer processor to perform a method, the method comprising:
(A) receiving a plurality of images and a plurality of sets of corresponding ground truth instance labels, wherein each of the ground truth instance labels associates a region of an image with a specific instance of an object;
(B) receiving a training objective, wherein the training objective specifies that a divergence of a probability that a first sample and a second sample correspond to the same instance label as each other in the plurality of sets of corresponding ground truth instance labels is to be minimized, wherein:
the first sample corresponds to a first pixel in an image;
a first pixel label distribution comprises the first sample;
the second sample corresponds to a second pixel in the image;
a second pixel label distribution comprises the second sample; and
(C) training the artificial neural network to minimize a loss function based on the plurality of images, the plurality of sets of corresponding ground truth instance labels, and the training objective, comprising:
determining an injective mapping g from ground truth instance labels T onto labels L predicted by the artificial neural network to minimize loss of the artificial neural network;
applying the injective mapping g during a forward pass of the artificial neural network, wherein the loss function after applying the injective mapping is permutation-invariant with respect to the labels L predicted by the artificial neural network; and
backpropagating a loss gradient through the injective mapping g to update the artificial neural network.
10. The system of claim 9 , wherein the method further comprises:
(D) applying the artificial neural network, after the training, to a particular image comprising a plurality of pixels, to produce a categorical probability distribution of the plurality of pixels over a plurality of instance labels.
11. The system of claim 10 , wherein the categorical probability distribution maps a first one of the plurality of pixels to a first instance of a first class and maps a second one of the plurality of pixels to a second instance of the first class.
12. The system of claim 10 , wherein the categorical probability distribution maps a first one of the plurality of pixels to a first instance of a first class and maps a second one of the plurality of pixels to a first instance of a second class.
13. The system of claim 9 , wherein (C) comprises training the artificial neural network by minimizing a divergence of permutation-invariant auxiliary distributions derived from the ground truth instance labels and network distributions.
14. The system of claim 9 , wherein the artificial neural network comprises a Fully Convolutional Neural Network.
15. The system of claim 14 , wherein the Fully Convolutional Neural Network is configured to directly output a probability distribution by using a Softmax activation.
16. The system of claim 9 , wherein the training in (C) enforces that L, the probability distribution over N instance labels, factorizes as the product of H×W independent categorical distributions.
17. A method for training an artificial neural network, performed by at least one computer processor executing computer program instructions stored on at least one non-transitory computer-readable medium, the method comprising:
(A) receiving a plurality of images and a plurality of sets of corresponding ground truth instance labels, wherein each of the ground truth instance labels associates a region of an image with a specific instance of an object;
(B) receiving a training objective, wherein the training objective specifies to minimize, for each pixel in an image, a divergence between (1) a distribution over instance labels produced by the artificial neural network and (2) a distribution resulting from applying an injective mapping of ground truth instance labels to neural network output labels to the distribution over instance labels determined from the corresponding ground truth instance labels, wherein the injective mapping is chosen for each image at every update step to minimize the divergence between (1) and (2); and
(C) training the artificial neural network to minimize a loss function based on the plurality of images, the plurality of sets of corresponding ground truth instance labels, and the training objective, comprising:
determining an injective mapping g from ground truth instance labels T onto labels L predicted by the artificial neural network to minimize loss of the artificial neural network;
applying the injective mapping g during a forward pass of the artificial neural network, wherein the loss function after applying the injective mapping is permutation-invariant with respect to the labels L predicted by the artificial neural network; and
backpropagating a loss gradient through the injective mapping g to update the artificial neural network.
18. A system for training an artificial neural network, the system comprising at least one non-transitory computer-readable medium comprising computer program instructions executable by at least one computer processor to perform a method, the method comprising:
(A) receiving a plurality of images and a plurality of sets of corresponding instance labels, wherein each of the ground truth instance labels associates a region of an image with a specific instance of an object;
(B) receiving a training objective, wherein the training objective specifies to minimize, for each pixel in an image, a divergence between (1) a distribution over instance labels produced by the artificial neural network and (2) a distribution resulting from applying an injective mapping of ground truth instance labels to neural network output labels to the distribution over instance labels determined from the corresponding ground truth instance labels, wherein the injective mapping is chosen for each image at every update step to minimize the divergence between (1) and (2); and
(C) training the artificial neural network to minimize a loss function based on the plurality of images, the plurality of sets of corresponding ground truth instance labels, and the training objective, comprising:
determining an injective mapping g from ground truth instance labels T onto labels L predicted by the artificial neural network to minimize loss of the artificial neural network;
applying the injective mapping g during a forward pass of the artificial neural network, wherein the loss function after applying the injective mapping is permutation-invariant with respect to the labels L predicted by the artificial neural network; and
backpropagating a loss gradient through the injective mapping g to update the artificial neural network.