High resolution human imaging using neural network
High resolution human imaging using neural network is described. In one example, a described method comprises: assembling a k1-dimensional (k1-D) imaging matrix by a processor based on arranging and concatenating a plurality of k2-dimensional (k2-D) input imaging matrices; encoding the k1-D imaging matrix by an encoder, which is a k1-D encoding neural network, to generate a (k1+k3)-D first intermediate matrix; generating a (k2+k3)-D second intermediate matrix based on the (k1+k3)-D first intermediate matrix; and decoding the (k2+k3)-D second intermediate matrix by a decoder, which is a k2-D decoding neural network, to generate a k2-D output imaging matrix. There is at least one skip connection between the encoder and the decoder. An imaging resolution of the k2-D output imaging matrix is greater than the imaging resolution of any one of the plurality of k2-D input imaging matrices.
1 . A method for encoder-decoder-based high resolution imaging, comprising:
assembling a k1-dimensional (k1-D) imaging matrix by a processor based on arranging and concatenating a plurality of k2-dimensional (k2-D) input imaging matrices;
encoding the k1-D imaging matrix by an encoder, which is a k1-D encoding neural network, to generate a (k1+k3)-D first intermediate matrix that includes k1 original dimensions of the k1-D imaging matrix and k3 expanded dimensions;
generating a (k2+k3)-D second intermediate matrix by removing (k1−k2) of the k1 original dimensions in the (k1+k3)-D first intermediate matrix, wherein each of k1, k2, k3 is a positive integer; and
decoding the (k2+k3)-D second intermediate matrix by a decoder, which is a k2-D decoding neural network, to generate a k2-D output imaging matrix, wherein:
there is at least one skip connection between the encoder and the decoder, and
an imaging resolution of the k2-D output imaging matrix is greater than the imaging resolution of any one of the plurality of k2-D input imaging matrices.
2 . The method of claim 1 , further comprising:
determining a (k1+k3)-D expanded matrix by adding k3 expanded dimensions to the k1 original dimensions of the k1-D imaging matrix, wherein:
a size of the (k1+k3)-D expanded matrix is 1 in each of the k3 expanded dimensions,
the encoder comprises L1 layers, each associated with multiple convolution filters,
the (k1+k3)-D expanded matrix is a (k1+k3)-D layer-1 input matrix to a layer 1 of the encoder, the first layer of the L1 layers of the encoder.
3 . The method of claim 2 , further comprising:
for each n between 1 and L1:
for each respective convolution filter of the multiple convolution filters associated with layer n of the encoder:
filtering a (k1+k3)-D layer-n input matrix with the respective convolution filter to generate a respective (k1+k3)-D layer-n per-filter output matrix, and
down-sampling the respective (k1+k3)-D layer-n per-filter output matrix in at least one of the k1 original dimensions with max-pooling; and
concatenating all of the (k1+k3)-D layer-n per-filter output matrices to form a (k1+k3)-D layer-n output matrix which is also a layer-(n+1) input matrix,
wherein:
a size of the layer-n output matrix in each of the k1 original dimensions is smaller than or equal to a size of a corresponding original dimension of the layer-n input matrix,
a size of the layer-n output matrix in each of the k3 expanded dimensions is greater than or equal to a size of a corresponding expanded dimension of the layer-n input matrix,
the (k1+k3)-D first intermediate matrix is the layer-(L1) output matrix, and
a size of the (k1+k3)-D first intermediate matrix is 1 in each of the k1 original dimensions.
4 . The method of claim 3 ,
wherein:
the decoder comprises L2 layers, each associated with multiple de-convolution filters,
the (k2+k3)-D second intermediate matrix is a (k2+k3)-D layer-(L1+1) input matrix to a layer 1 of the decoder, the first layer of the L2 layers of the decoder.
5 . The method of claim 4 , further comprising:
for each m between 1 and L2:
up-sampling a (k2+k3)-D layer-(L1+m) input matrix in at least one of the remaining k2 original dimensions to generate a (k2+k3)-D layer-(L1+m) up-sampled input matrix,
for each respective de-convolution filter of the multiple de-convolution filters associated with layer m of the decoder, filtering the (k2+k3)-D layer-(L1+m) up-sampled input matrix with the respective de-convolution filter to generate a respective (k2+k3)-D layer-(L1+m) per-filter output matrix, and
combining and concatenating all of the (k2+k3)-D layer-(L1+m) per-filter output matrices to form a (k2+k3)-D layer-(L1+m) output matrix which is also a layer-(L1+m+1) input matrix, wherein:
a size of the layer-(L1+m) output matrix in each of the remaining k2 original dimensions is greater than or equal to a size of a corresponding original dimension of the layer-(L1+m) input matrix,
a size of the layer-(L1+m) output matrix in each of the k3 expanded dimensions is less than or equal to a size of a corresponding expanded dimension of the layer-(L1+m) input matrix; and
removing the k3 expanded dimensions of the layer-(L1+L2) output matrix to generate the k2-D output imaging matrix.
6 . The method of claim 5 , further comprising:
processing a m1−n1 skip connection, which is a skip connection between layer m1 of the encoder and layer n1 of the decoder, with 1<=m1<=L1, 1<=n1<=L2, wherein the m1−n1 skip connection is processed based on:
adjusting the (k1+k3)-D layer-(m1) output matrix of the encoder to generate a (k2+k3)-D adjusted layer-(m1) output matrix of the encoder, wherein the (k2+k3)-D adjusted layer-(m1) output matrix has a same dimension and a same size as the (k2+k3)-D layer-(L1+n1) output matrix of the decoder;
computing a temporary output matrix as a weighted average of the (k2+k3)-D adjusted layer-(m1) output matrix of the encoder and the (k2+k3)-D layer-(L1+n1) output matrix of the decoder; and
replacing the (k2+k3)-D layer-(L1+n1+1) input matrix of the decoder by the temporary output matrix.
7 . The method of claim 6 , wherein:
there is a m2-n2 skip connection, which is a skip connection between layer m2 of the encoder and layer n2 of the decoder, with 1<=m1<m2<=L1, 1<=n2<n1<=L2; and
the method further comprises processing all skip connections in an ascending order of layer index of the decoder such that the m2-n2 skip connection is processed before the m1−n1 skip connection.
8 . The method of claim 6 , further comprising:
for any n1 and the (k2+k3)-D layer-(L1+n1) output matrix of the decoder: performing a search among all L1 layers of the encoder to identify the m1 layer of the encoder which has an adjusted output matrix with a size matching the size of the layer-(L1+n1) output matrix of the decoder, in order to form the m1−n1 skip connection, wherein:
when no matching layer of the encoder is found, a best match is used for the m1-n1 skip connection,
the best match m1 is a layer of the encoder with a minimum absolute difference between the size of its adjusted output matrix and the size of the layer-(L1+n1) output matrix of the decoder.
9 . The method of claim 6 , further comprising:
computing each element of the (k2+k3)-D adjusted layer-(m1) output matrix of the encoder as a characteristic value of a respective (k1−k2)-D sub-matrix of the (k1+k3)-D layer-(m1) output matrix of the encoder, wherein:
the characteristic value is an aggregate of the elements in the respective (k1−k2)-D submatrix,
the aggregate comprises at least one of: mean, arithmetic mean, geometric mean, harmonic mean, weighted average, percentile, minimum (0-percentile), median (50-percentile), maximum (100-percentile), trimmed mean (mean of percentile from A1-percentile to A2-percentile), mode, sum, weighted sum, product or weighted product.
10 . The method of claim 1 , further comprising:
training the encoder and the decoder as part of a conditional generative adversarial network (cGAN) comprising a generator and a discriminator, wherein:
the generator comprises the encoder and the decoder with the at least one skip connection;
the k1-D imaging matrix is an input of the generator;
the k2-D output imaging matrix is an output of the generator; and
the discriminator comprises:
the encoder with the k1-D imaging matrix as input and the (k2+k3)-D second intermediate matrix as output,
an additional encoder with a k2-D supplementary input imaging matrix as input, and
a full-connected network configured to compute a real-fake score.
11 . The method of claim 10 , further comprising:
training the cGAN with a cost function which is a weighted sum of a standard generative adversarial network (GAN) loss function and an additional cost function, wherein the additional cost function is at least one of:
a L-k loss function or a weighted L-k loss function between the k2-D output imaging matrix and the k2-D supplement input imaging matrix, with k being an integer,
a visual loss function,
a pre-trained perceptual loss function,
a SSIM loss function,
an image structural loss function, or
an image-based loss function.
12 . A system for encoder-decoder-based high resolution imaging, comprising:
a processor configured to assemble a k1-dimensional (k1-D) imaging matrix by arranging and concatenating a plurality of k2-dimensional (k2-D) input imaging matrices;
an encoder, which is a k1-D encoding neural network, configured to encode the k1-D imaging matrix to generate a (k1+k3)-D first intermediate matrix that includes k1 original dimensions of the k1-D imaging matrix and k3 expanded dimensions, wherein the processor is further configured to generate a (k2+k3)-D second intermediate matrix by removing (k1−k2) of the k1 original dimensions in the (k1+k3)-D first intermediate matrix, wherein each of k1, k2, k3 is a positive integer; and
a decoder, which is a k2-D decoding neural network, configured to decode the (k2+k3)-D second intermediate matrix to generate a k2-D output imaging matrix, wherein:
there is at least one skip connection between the encoder and the decoder, and
an imaging resolution of the k2-D output imaging matrix is greater than the imaging resolution of any one of the plurality of k2-D input imaging matrices.
13 . The system of claim 12 , wherein the processor is further configured to:
determine a (k1+k3)-D expanded matrix by adding k3 expanded dimensions to the k1 original dimensions of the k1-D imaging matrix, wherein:
a size of the (k1+k3)-D expanded matrix is 1 in each of the k3 expanded dimensions,
the encoder comprises L1 layers, each associated with multiple convolution filters,
the (k1+k3)-D expanded matrix is a (k1+k3)-D layer-1 input matrix to a layer 1 of the encoder, the first layer of the L1 layers of the encoder.
14 . The system of claim 13 , wherein the processor is further configured to:
for each n between 1 and L1:
for each respective convolution filter of the multiple convolution filters associated with layer n of the encoder:
filter a (k1+k3)-D layer-n input matrix with the respective convolution filter to generate a respective (k1+k3)-D layer-n per-filter output matrix, and
down-sample the respective (k1+k3)-D layer-n per-filter output matrix in at least one of the k1 original dimensions with max-pooling; and
concatenate all of the (k1+k3)-D layer-n per-filter output matrices to form a (k1+k3)-D layer-n output matrix which is also a layer-(n+1) input matrix,
wherein:
a size of the layer-n output matrix in each of the k1 original dimensions is smaller than or equal to a size of a corresponding original dimension of the layer-n input matrix,
a size of the layer-n output matrix in each of the k3 expanded dimensions is greater than or equal to a size of a corresponding expanded dimension of the layer-n input matrix,
the (k1+k3)-D first intermediate matrix is the layer-(L1) output matrix, and
a size of the (k1+k3)-D first intermediate matrix is 1 in each of the k1 original dimensions.
15 . The system of claim 14 ,
wherein:
the decoder comprises L2 layers, each associated with multiple de-convolution filters,
the (k2+k3)-D second intermediate matrix is a (k2+k3)-D layer-(L1+1) input matrix to a layer 1 of the decoder, the first layer of the L2 layers of the decoder.
16 . The system of claim 15 , wherein the processor is further configured to:
for each m between 1 and L2:
up-sample a (k2+k3)-D layer-(L1+m) input matrix in at least one of the remaining k2 original dimensions to generate a (k2+k3)-D layer-(L1+m) up-sampled input matrix,
for each respective de-convolution filter of the multiple de-convolution filters associated with layer m of the decoder, filter the (k2+k3)-D layer-(L1+m) up-sampled input matrix with the respective de-convolution filter to generate a respective (k2+k3)-D layer-(L1+m) per-filter output matrix, and
combine and concatenate all of the (k2+k3)-D layer-(L1+m) per-filter output matrices to form a (k2+k3)-D layer-(L1+m) output matrix which is also a layer-(L1+m+1) input matrix, wherein:
a size of the layer-(L1+m) output matrix in each of the remaining k2 original dimensions is greater than or equal to a size of a corresponding original dimension of the layer-(L1+m) input matrix,
a size of the layer-(L1+m) output matrix in each of the k3 expanded dimensions is less than or equal to a size of a corresponding expanded dimension of the layer-(L1+m) input matrix; and
remove the k3 expanded dimensions of the layer-(L1+L2) output matrix to generate the k2-D output imaging matrix.
17 . The system of claim 16 , wherein the processor is further configured to:
process a m1−n1 skip connection, which is a skip connection between layer m1 of the encoder and layer n1 of the decoder, with 1<=m1<=L1, 1<=n1<=L2, wherein the m1−n1 skip connection is processed based on:
adjusting the (k1+k3)-D layer-(m1) output matrix of the encoder to generate a (k2+k3)-D adjusted layer-(m1) output matrix of the encoder, wherein the (k2+k3)-D adjusted layer-(m1) output matrix has a same dimension and a same size as the (k2+k3)-D layer-(L1+n1) output matrix of the decoder;
computing a temporary output matrix as a weighted average of the (k2+k3)-D adjusted layer-(m1) output matrix of the encoder and the (k2+k3)-D layer-(L1+n1) output matrix of the decoder; and
replacing the (k2+k3)-D layer-(L1+n1+1) input matrix of the decoder by the temporary output matrix.
18 . The system of claim 17 , wherein:
there is a m2-n2 skip connection, which is a skip connection between layer m2 of the encoder and layer n2 of the decoder, with 1<=m1<m2<=L1, 1<=n2<n1<=L2; and
the processor is further configured to process all skip connections in an ascending order of layer index of the decoder such that the m2-n2 skip connection is processed before the m1−n1 skip connection.
19 . The system of claim 17 , wherein the processor is further configured to:
for any n1 and the (k2+k3)-D layer-(L1+n1) output matrix of the decoder: perform a search among all L1 layers of the encoder to identify the m1 layer of the encoder which has an adjusted output matrix with a size matching the size of the layer-(L1+n1) output matrix of the decoder, in order to form the m1−n1 skip connection, wherein:
when no matching layer of the encoder is found, a best match is used for the m1-n1 skip connection,
the best match m1 is a layer of the encoder with a minimum absolute difference between the size of its adjusted output matrix and the size of the layer-(L1+n1) output matrix of the decoder.
20 . The system of claim 17 , wherein the processor is further configured to:
compute each element of the (k2+k3)-D adjusted layer-(m1) output matrix of the encoder as a characteristic value of a respective (k1−k2)-D sub-matrix of the (k1+k3)-D layer-(m1) output matrix of the encoder, wherein:
the characteristic value is an aggregate of the elements in the respective (k1−k2)-D submatrix,
the aggregate comprises at least one of: mean, arithmetic mean, geometric mean, harmonic mean, weighted average, percentile, minimum (0-percentile), median (50-percentile), maximum (100-percentile), trimmed mean (mean of percentile from A1-percentile to A2-percentile), mode, sum, weighted sum, product or weighted product.
21 . The system of claim 12 , wherein the processor is further configured to:
train the encoder and the decoder as part of a conditional generative adversarial network (cGAN) comprising a generator and a discriminator, wherein:
the generator comprises the encoder and the decoder with the at least one skip connection;
the k1-D imaging matrix is an input of the generator;
the k2-D output imaging matrix is an output of the generator; and
the discriminator comprises:
the encoder with the k1-D imaging matrix as input and the (k2+k3)-D second intermediate matrix as output,
an additional encoder with a k2-D supplementary input imaging matrix as input, and
a full-connected network configured to compute a real-fake score.
22 . The system of claim 21 , wherein the processor is further configured to:
train the cGAN with a cost function which is a weighted sum of a standard generative adversarial network (GAN) loss function and an additional cost function, wherein the additional cost function is at least one of:
a L-k loss function or a weighted L-k loss function between the k2-D output imaging matrix and the k2-D supplement input imaging matrix, with k being an integer,
a visual loss function,
a pre-trained perceptual loss function,
a SSIM loss function,
an image structural loss function, or
an image-based loss function.
23 . A device for encoder-decoder-based high resolution imaging, comprising:
a processor; and
a memory storing instructions, which when executed, cause the processor to perform operations comprising:
assembling a k1-dimensional (k1-D) imaging matrix by arranging and concatenating a plurality of k2-dimensional (k2-D) input imaging matrices,
encoding the k1-D imaging matrix by an encoder, which is a k1-D encoding neural network, to generate a (k1+k3)-D first intermediate matrix that includes k1 original dimensions of the k1-D imaging matrix and k3 expanded dimensions,
generating a (k2+k3)-D second intermediate matrix by removing (k1−k2) of the k1 original dimensions in the (k1+k3)-D first intermediate matrix, wherein each of k1, k2, k3 is a positive integer, and
decoding the (k2+k3)-D second intermediate matrix using a decoder, which is a k2-D decoding neural network, to generate a k2-D output imaging matrix, wherein:
there is at least one skip connection between the encoder and the decoder, and
an imaging resolution of the k2-D output imaging matrix is greater than the imaging resolution of any one of the plurality of k2-D input imaging matrices.
24 . The device of claim 23 , wherein the operations further comprise:
determining a (k1+k3)-D expanded matrix by adding k3 expanded dimensions to the k1 original dimensions of the k1-D imaging matrix, wherein:
a size of the (k1+k3)-D expanded matrix is 1 in each of the k3 expanded dimensions,
the encoder comprises L1 layers, each associated with multiple convolution filters,
the (k1+k3)-D expanded matrix is a (k1+k3)-D layer-1 input matrix to a layer 1 of the encoder, the first layer of the L1 layers of the encoder.
25 . The device of claim 24 , wherein the operations further comprise:
for each n between 1 and L1:
for each respective convolution filter of the multiple convolution filters associated with layer n of the encoder:
filtering a (k1+k3)-D layer-n input matrix with the respective convolution filter to generate a respective (k1+k3)-D layer-n per-filter output matrix, and
down-sampling the respective (k1+k3)-D layer-n per-filter output matrix in at least one of the k1 original dimensions with max-pooling; and
concatenating all of the (k1+k3)-D layer-n per-filter output matrices to form a (k1+k3)-D layer-n output matrix which is also a layer-(n+1) input matrix,
wherein:
a size of the layer-n output matrix in each of the k1 original dimensions is smaller than or equal to a size of a corresponding original dimension of the layer-n input matrix,
a size of the layer-n output matrix in each of the k3 expanded dimensions is greater than or equal to a size of a corresponding expanded dimension of the layer-n input matrix,
the (k1+k3)-D first intermediate matrix is the layer-(L1) output matrix, and
a size of the (k1+k3)-D first intermediate matrix is 1 in each of the k1 original dimensions.
26 . The device of claim 25 ,
wherein:
the decoder comprises L2 layers, each associated with multiple de-convolution filters,
the (k2+k3)-D second intermediate matrix is a (k2+k3)-D layer-(L1+1) input matrix to a layer 1 of the decoder, the first layer of the L2 layers of the decoder.
27 . The device of claim 26 , wherein the operations further comprise:
for each m between 1 and L2:
up-sampling a (k2+k3)-D layer-(L1+m) input matrix in at least one of the remaining k2 original dimensions to generate a (k2+k3)-D layer-(L1+m) up-sampled input matrix,
for each respective de-convolution filter of the multiple de-convolution filters associated with layer m of the decoder, filtering the (k2+k3)-D layer-(L1+m) up-sampled input matrix with the respective de-convolution filter to generate a respective (k2+k3)-D layer-(L1+m) per-filter output matrix, and
combining and concatenating all of the (k2+k3)-D layer-(L1+m) per-filter output matrices to form a (k2+k3)-D layer-(L1+m) output matrix which is also a layer-(L1+m+1) input matrix, wherein:
a size of the layer-(L1+m) output matrix in each of the remaining k2 original dimensions is greater than or equal to a size of a corresponding original dimension of the layer-(L1+m) input matrix,
a size of the layer-(L1+m) output matrix in each of the k3 expanded dimensions is less than or equal to a size of a corresponding expanded dimension of the layer-(L1+m) input matrix; and
removing the k3 expanded dimensions of the layer-(L1+L2) output matrix to generate the k2-D output imaging matrix.
28 . The device of claim 27 , wherein the operations further comprise:
processing a m1−n1 skip connection, which is a skip connection between layer m1 of the encoder and layer n1 of the decoder, with 1<=m1<=L1, 1<=n1<=L2, wherein the m1−n1 skip connection is processed based on:
adjusting the (k1+k3)-D layer-(m1) output matrix of the encoder to generate a (k2+k3)-D adjusted layer-(m1) output matrix of the encoder, wherein the (k2+k3)-D adjusted layer-(m1) output matrix has a same dimension and a same size as the (k2+k3)-D layer-(L1+n1) output matrix of the decoder;
computing a temporary output matrix as a weighted average of the (k2+k3)-D adjusted layer-(m1) output matrix of the encoder and the (k2+k3)-D layer-(L1+n1) output matrix of the decoder; and
replacing the (k2+k3)-D layer-(L1+n1+1) input matrix of the decoder by the temporary output matrix.
29 . The device of claim 28 , wherein:
there is a m2-n2 skip connection, which is a skip connection between layer m2 of the encoder and layer n2 of the decoder, with 1<=m1<m2<=L1, 1<=n2<n1<=L2; and
the operations further comprise processing all skip connections in an ascending order of layer index of the decoder such that the m2-n2 skip connection is processed before the m1−n1 skip connection.
30 . The device of claim 29 , wherein the operations further comprise:
for any n1 and the (k2+k3)-D layer-(L1+n1) output matrix of the decoder: performing a search among all L1 layers of the encoder to identify the m1 layer of the encoder which has an adjusted output matrix with a size matching the size of the layer-(L1+n1) output matrix of the decoder, in order to form the m1−n1 skip connection, wherein:
when no matching layer of the encoder is found, a best match is used for the m1-n1 skip connection,
the best match m1 is a layer of the encoder with a minimum absolute difference between the size of its adjusted output matrix and the size of the layer-(L1+n1) output matrix of the decoder.