Apparatus and method for image segmentation using a deep convolutional neural network with a nested U-structure
A device receives an image. The image is applied to a sequence of encoders that includes a first encoder and one or more subsequent encoders, each including a U network encoder which performs convolution neural network processing. The one or more subsequent encoders receive a down-sampled version of the image from a prior encoder. The sequence of encoders form a first dimension orthogonal to the orthogonal dimension. A result of the sequence of encoders is applied to a sequence of decoders that includes a first decoder and one or more subsequent decoders. A respective decoder comprises a U network decoder that performs convolution neural network processing of an up-sampled version of the image from the first decoder. Probability outputs are produced from paired encoders and decoders in the sequence of encoders and the sequence of decoders. The probability outputs are combined to form a final output.
1 . A non-transitory computer readable storage medium with instructions which, when executed by a processor, cause the processor to:
receive an image;
apply the image to a sequence of encoders that includes a first encoder and one or more subsequent encoders, wherein each encoder in the sequence of encoders comprises a U network encoder which performs convolution neural network processing in an orthogonal dimension, and the one or more subsequent encoders receive a down-sampled version of the image from a prior encoder, the sequence of encoders forming a first dimension orthogonal to the orthogonal dimension;
apply a result of the sequence of encoders to a sequence of decoders that includes a first decoder and one or more subsequent decoders, wherein a respective decoder in the one or more subsequent decoders comprises a U network decoder that performs convolution neural network processing of an up-sampled version of the image from the first decoder, and the sequence of decoders forms a second dimension parallel to the first dimension;
produce probability outputs from paired encoders and decoders in the sequence of encoders and the sequence of decoders; and
combine the probability outputs to form a final output.
2 . The non-transitory computer readable storage medium of claim 1 , wherein the instructions further cause a respective encoder in the sequence of encoders and a respective decoder in the sequence of decoders to receive input feature maps and generate multi-scale features and local features.
3 . The non-transitory computer readable storage medium of claim 1 , wherein the image has a shape with an associated height, width, and input channel number.
4 . The non-transitory computer readable storage medium of claim 3 , wherein the final output is characterized as Y output =K(W,X input ), wherein X input is the image, Y output is a probability map with pixel values ranging from zero to one with the shape of the associated height and width, and W denotes weights of kernel filters.
5 . The non-transitory computer readable storage medium of claim 1 , wherein the final output is a multi-channel feature map fed to an activation function to produce multi-class segmentation results.
6 . The non-transitory computer readable storage medium of claim 1 , wherein the final output is characterized as a feature map with regression results without using activation functions.
7 . The non-transitory computer readable storage medium of claim 3 , wherein the input channel number is two or more.
8 . The non-transitory computer readable storage medium of claim 1 , wherein a respective encoder in the sequence of encoders and a respective decoder in the sequence of decoders have configurable heights.
9 . The non-transitory computer readable storage medium of claim 1 , wherein a respective encoder in the sequence of encoders and a respective decoder in the sequence of decoders have configurable filter numbers.
10 . The non-transitory computer readable storage medium of claim 1 , wherein each decoder in the one or more subsequent decoders concatenates an up-sampled feature map from a previous decoder with a down-sampled feature from a paired encoder.
11 . The non-transitory computer readable storage medium of claim 1 , wherein the instructions to combine the probability outputs include instructions which, when executed by the processor, cause the processor to:
concatenate the probability outputs to form concatenated probability outputs.
12 . The non-transitory computer readable storage medium of claim 11 , wherein the instructions further cause the processor to:
apply a convolution layer to the concatenated probability outputs to form convoluted concatenated probability outputs.
13 . The non-transitory computer readable storage medium of claim 12 , wherein the instructions further cause the processor to apply a sigmoid function to the convoluted concatenated probability outputs to form the final output.
14 . A method, comprising:
at a computer system that includes a processor and memory:
receiving an image;
applying the image to a sequence of encoders that includes a first encoder and one or more subsequent encoders, wherein each encoder in the sequence of encoders comprises a U network encoder which performs convolution neural network processing in an orthogonal dimension, and the one or more subsequent encoders receive a down-sampled version of the image from a prior encoder, the sequence of encoders forming a first dimension orthogonal to the orthogonal dimension;
applying a result of the sequence of encoders to a sequence of decoders that includes a first decoder and one or more subsequent decoders, wherein a respective decoder in the one or more subsequent decoders comprises a U network decoder that performs convolution neural network processing of an up-sampled version of the image from the first decoder, and the sequence of decoders forms a second dimension parallel to the first dimension;
producing probability outputs from paired encoders and decoders in the sequence of encoders and the sequence of decoders; and
combining the probability outputs to form a final output.
15 . The method of claim 14 , further comprising:
causing a respective encoder in the sequence of encoders and a respective decoder in the sequence of decoders to receive input feature maps and generate multi-scale features and local features.
16 . The method of claim 15 , wherein the image has a shape with an associated height, width, and input channel number.
17 . The method of claim 16 , wherein the final output is characterized as Y output =K(W,X input ), wherein X input is the image, Y output is a probability map with pixel values ranging from zero to one with the shape of the associated height and width, and W denotes weights of kernel filters.
18 . The method of claim 14 , wherein the final output is a multi-channel feature map fed to an activation function to produce multi-class segmentation results.
19 . A computer system, comprising:
a processor; and
memory storing one or more programs, the one or more programs including instructions for:
receiving an image;
applying the image to a sequence of encoders that includes a first encoder and one or more subsequent encoders, wherein each encoder in the sequence of encoders comprises a U network encoder which performs convolution neural network processing in an orthogonal dimension, and the one or more subsequent encoders receive a down-sampled version of the image from a prior encoder, the sequence of encoders forming a first dimension orthogonal to the orthogonal dimension;
applying a result of the sequence of encoders to a sequence of decoders that includes a first decoder and one or more subsequent decoders, wherein a respective decoder in the one or more subsequent decoders comprises a U network decoder that performs convolution neural network processing of an up-sampled version of the image from the first decoder, and the sequence of decoders forms a second dimension parallel to the first dimension;
producing probability outputs from paired encoders and decoders in the sequence of encoders and the sequence of decoders; and
combining the probability outputs to form a final output.
20 . The computer system of claim 19 , wherein the memory further includes for causing a respective encoder in the sequence of encoders and a respective decoder in the sequence of decoders to receive input feature maps and generate multi-scale features and local features.