High resolution human imaging using neural network

1 . A method for encoder-decoder-based high resolution imaging, comprising:

assembling a k1-dimensional (k1-D) imaging matrix by a processor based on arranging and concatenating a plurality of k2-dimensional (k2-D) input imaging matrices;

encoding the k1-D imaging matrix by an encoder, which is a k1-D encoding neural network, to generate a (k1+k3)-D first intermediate matrix that includes k1 original dimensions of the k1-D imaging matrix and k3 expanded dimensions;

generating a (k2+k3)-D second intermediate matrix by removing (k1−k2) of the k1 original dimensions in the (k1+k3)-D first intermediate matrix, wherein each of k1, k2, k3 is a positive integer; and

decoding the (k2+k3)-D second intermediate matrix by a decoder, which is a k2-D decoding neural network, to generate a k2-D output imaging matrix, wherein:

there is at least one skip connection between the encoder and the decoder, and

an imaging resolution of the k2-D output imaging matrix is greater than the imaging resolution of any one of the plurality of k2-D input imaging matrices.

2 . The method of claim 1 , further comprising:

determining a (k1+k3)-D expanded matrix by adding k3 expanded dimensions to the k1 original dimensions of the k1-D imaging matrix, wherein:

a size of the (k1+k3)-D expanded matrix is 1 in each of the k3 expanded dimensions,

the encoder comprises L1 layers, each associated with multiple convolution filters,

the (k1+k3)-D expanded matrix is a (k1+k3)-D layer-1 input matrix to a layer 1 of the encoder, the first layer of the L1 layers of the encoder.

3 . The method of claim 2 , further comprising:

for each n between 1 and L1:

for each respective convolution filter of the multiple convolution filters associated with layer n of the encoder:

filtering a (k1+k3)-D layer-n input matrix with the respective convolution filter to generate a respective (k1+k3)-D layer-n per-filter output matrix, and

down-sampling the respective (k1+k3)-D layer-n per-filter output matrix in at least one of the k1 original dimensions with max-pooling; and

concatenating all of the (k1+k3)-D layer-n per-filter output matrices to form a (k1+k3)-D layer-n output matrix which is also a layer-(n+1) input matrix,

wherein:

a size of the layer-n output matrix in each of the k1 original dimensions is smaller than or equal to a size of a corresponding original dimension of the layer-n input matrix,

a size of the layer-n output matrix in each of the k3 expanded dimensions is greater than or equal to a size of a corresponding expanded dimension of the layer-n input matrix,

the (k1+k3)-D first intermediate matrix is the layer-(L1) output matrix, and

a size of the (k1+k3)-D first intermediate matrix is 1 in each of the k1 original dimensions.

4 . The method of claim 3 ,

wherein:

the decoder comprises L2 layers, each associated with multiple de-convolution filters,

the (k2+k3)-D second intermediate matrix is a (k2+k3)-D layer-(L1+1) input matrix to a layer 1 of the decoder, the first layer of the L2 layers of the decoder.

5 . The method of claim 4 , further comprising:

for each m between 1 and L2:

up-sampling a (k2+k3)-D layer-(L1+m) input matrix in at least one of the remaining k2 original dimensions to generate a (k2+k3)-D layer-(L1+m) up-sampled input matrix,

for each respective de-convolution filter of the multiple de-convolution filters associated with layer m of the decoder, filtering the (k2+k3)-D layer-(L1+m) up-sampled input matrix with the respective de-convolution filter to generate a respective (k2+k3)-D layer-(L1+m) per-filter output matrix, and

combining and concatenating all of the (k2+k3)-D layer-(L1+m) per-filter output matrices to form a (k2+k3)-D layer-(L1+m) output matrix which is also a layer-(L1+m+1) input matrix, wherein:

a size of the layer-(L1+m) output matrix in each of the remaining k2 original dimensions is greater than or equal to a size of a corresponding original dimension of the layer-(L1+m) input matrix,

a size of the layer-(L1+m) output matrix in each of the k3 expanded dimensions is less than or equal to a size of a corresponding expanded dimension of the layer-(L1+m) input matrix; and

removing the k3 expanded dimensions of the layer-(L1+L2) output matrix to generate the k2-D output imaging matrix.

6 . The method of claim 5 , further comprising:

processing a m1−n1 skip connection, which is a skip connection between layer m1 of the encoder and layer n1 of the decoder, with 1<=m1<=L1, 1<=n1<=L2, wherein the m1−n1 skip connection is processed based on:

adjusting the (k1+k3)-D layer-(m1) output matrix of the encoder to generate a (k2+k3)-D adjusted layer-(m1) output matrix of the encoder, wherein the (k2+k3)-D adjusted layer-(m1) output matrix has a same dimension and a same size as the (k2+k3)-D layer-(L1+n1) output matrix of the decoder;

computing a temporary output matrix as a weighted average of the (k2+k3)-D adjusted layer-(m1) output matrix of the encoder and the (k2+k3)-D layer-(L1+n1) output matrix of the decoder; and

replacing the (k2+k3)-D layer-(L1+n1+1) input matrix of the decoder by the temporary output matrix.

7 . The method of claim 6 , wherein:

there is a m2-n2 skip connection, which is a skip connection between layer m2 of the encoder and layer n2 of the decoder, with 1<=m1<m2<=L1, 1<=n2<n1<=L2; and

the method further comprises processing all skip connections in an ascending order of layer index of the decoder such that the m2-n2 skip connection is processed before the m1−n1 skip connection.

8 . The method of claim 6 , further comprising:

for any n1 and the (k2+k3)-D layer-(L1+n1) output matrix of the decoder: performing a search among all L1 layers of the encoder to identify the m1 layer of the encoder which has an adjusted output matrix with a size matching the size of the layer-(L1+n1) output matrix of the decoder, in order to form the m1−n1 skip connection, wherein:

when no matching layer of the encoder is found, a best match is used for the m1-n1 skip connection,

the best match m1 is a layer of the encoder with a minimum absolute difference between the size of its adjusted output matrix and the size of the layer-(L1+n1) output matrix of the decoder.

9 . The method of claim 6 , further comprising:

computing each element of the (k2+k3)-D adjusted layer-(m1) output matrix of the encoder as a characteristic value of a respective (k1−k2)-D sub-matrix of the (k1+k3)-D layer-(m1) output matrix of the encoder, wherein:

the characteristic value is an aggregate of the elements in the respective (k1−k2)-D submatrix,

the aggregate comprises at least one of: mean, arithmetic mean, geometric mean, harmonic mean, weighted average, percentile, minimum (0-percentile), median (50-percentile), maximum (100-percentile), trimmed mean (mean of percentile from A1-percentile to A2-percentile), mode, sum, weighted sum, product or weighted product.

10 . The method of claim 1 , further comprising:

training the encoder and the decoder as part of a conditional generative adversarial network (cGAN) comprising a generator and a discriminator, wherein:

the generator comprises the encoder and the decoder with the at least one skip connection;

the k1-D imaging matrix is an input of the generator;

the k2-D output imaging matrix is an output of the generator; and

the discriminator comprises:

the encoder with the k1-D imaging matrix as input and the (k2+k3)-D second intermediate matrix as output,

an additional encoder with a k2-D supplementary input imaging matrix as input, and

a full-connected network configured to compute a real-fake score.

11 . The method of claim 10 , further comprising:

training the cGAN with a cost function which is a weighted sum of a standard generative adversarial network (GAN) loss function and an additional cost function, wherein the additional cost function is at least one of:

a L-k loss function or a weighted L-k loss function between the k2-D output imaging matrix and the k2-D supplement input imaging matrix, with k being an integer,

a visual loss function,

a pre-trained perceptual loss function,

a SSIM loss function,

an image structural loss function, or

an image-based loss function.

12 . A system for encoder-decoder-based high resolution imaging, comprising:

a processor configured to assemble a k1-dimensional (k1-D) imaging matrix by arranging and concatenating a plurality of k2-dimensional (k2-D) input imaging matrices;

an encoder, which is a k1-D encoding neural network, configured to encode the k1-D imaging matrix to generate a (k1+k3)-D first intermediate matrix that includes k1 original dimensions of the k1-D imaging matrix and k3 expanded dimensions, wherein the processor is further configured to generate a (k2+k3)-D second intermediate matrix by removing (k1−k2) of the k1 original dimensions in the (k1+k3)-D first intermediate matrix, wherein each of k1, k2, k3 is a positive integer; and

a decoder, which is a k2-D decoding neural network, configured to decode the (k2+k3)-D second intermediate matrix to generate a k2-D output imaging matrix, wherein:

there is at least one skip connection between the encoder and the decoder, and

an imaging resolution of the k2-D output imaging matrix is greater than the imaging resolution of any one of the plurality of k2-D input imaging matrices.

13 . The system of claim 12 , wherein the processor is further configured to:

determine a (k1+k3)-D expanded matrix by adding k3 expanded dimensions to the k1 original dimensions of the k1-D imaging matrix, wherein:

a size of the (k1+k3)-D expanded matrix is 1 in each of the k3 expanded dimensions,

the encoder comprises L1 layers, each associated with multiple convolution filters,

the (k1+k3)-D expanded matrix is a (k1+k3)-D layer-1 input matrix to a layer 1 of the encoder, the first layer of the L1 layers of the encoder.

14 . The system of claim 13 , wherein the processor is further configured to:

for each n between 1 and L1:

for each respective convolution filter of the multiple convolution filters associated with layer n of the encoder:

filter a (k1+k3)-D layer-n input matrix with the respective convolution filter to generate a respective (k1+k3)-D layer-n per-filter output matrix, and

down-sample the respective (k1+k3)-D layer-n per-filter output matrix in at least one of the k1 original dimensions with max-pooling; and

concatenate all of the (k1+k3)-D layer-n per-filter output matrices to form a (k1+k3)-D layer-n output matrix which is also a layer-(n+1) input matrix,

wherein:

a size of the layer-n output matrix in each of the k1 original dimensions is smaller than or equal to a size of a corresponding original dimension of the layer-n input matrix,

a size of the layer-n output matrix in each of the k3 expanded dimensions is greater than or equal to a size of a corresponding expanded dimension of the layer-n input matrix,

the (k1+k3)-D first intermediate matrix is the layer-(L1) output matrix, and

a size of the (k1+k3)-D first intermediate matrix is 1 in each of the k1 original dimensions.

15 . The system of claim 14 ,

wherein:

the decoder comprises L2 layers, each associated with multiple de-convolution filters,

the (k2+k3)-D second intermediate matrix is a (k2+k3)-D layer-(L1+1) input matrix to a layer 1 of the decoder, the first layer of the L2 layers of the decoder.

16 . The system of claim 15 , wherein the processor is further configured to:

for each m between 1 and L2:

up-sample a (k2+k3)-D layer-(L1+m) input matrix in at least one of the remaining k2 original dimensions to generate a (k2+k3)-D layer-(L1+m) up-sampled input matrix,

for each respective de-convolution filter of the multiple de-convolution filters associated with layer m of the decoder, filter the (k2+k3)-D layer-(L1+m) up-sampled input matrix with the respective de-convolution filter to generate a respective (k2+k3)-D layer-(L1+m) per-filter output matrix, and

combine and concatenate all of the (k2+k3)-D layer-(L1+m) per-filter output matrices to form a (k2+k3)-D layer-(L1+m) output matrix which is also a layer-(L1+m+1) input matrix, wherein:

a size of the layer-(L1+m) output matrix in each of the remaining k2 original dimensions is greater than or equal to a size of a corresponding original dimension of the layer-(L1+m) input matrix,

a size of the layer-(L1+m) output matrix in each of the k3 expanded dimensions is less than or equal to a size of a corresponding expanded dimension of the layer-(L1+m) input matrix; and

remove the k3 expanded dimensions of the layer-(L1+L2) output matrix to generate the k2-D output imaging matrix.

17 . The system of claim 16 , wherein the processor is further configured to:

process a m1−n1 skip connection, which is a skip connection between layer m1 of the encoder and layer n1 of the decoder, with 1<=m1<=L1, 1<=n1<=L2, wherein the m1−n1 skip connection is processed based on:

adjusting the (k1+k3)-D layer-(m1) output matrix of the encoder to generate a (k2+k3)-D adjusted layer-(m1) output matrix of the encoder, wherein the (k2+k3)-D adjusted layer-(m1) output matrix has a same dimension and a same size as the (k2+k3)-D layer-(L1+n1) output matrix of the decoder;

computing a temporary output matrix as a weighted average of the (k2+k3)-D adjusted layer-(m1) output matrix of the encoder and the (k2+k3)-D layer-(L1+n1) output matrix of the decoder; and

replacing the (k2+k3)-D layer-(L1+n1+1) input matrix of the decoder by the temporary output matrix.

18 . The system of claim 17 , wherein:

there is a m2-n2 skip connection, which is a skip connection between layer m2 of the encoder and layer n2 of the decoder, with 1<=m1<m2<=L1, 1<=n2<n1<=L2; and

the processor is further configured to process all skip connections in an ascending order of layer index of the decoder such that the m2-n2 skip connection is processed before the m1−n1 skip connection.

19 . The system of claim 17 , wherein the processor is further configured to:

for any n1 and the (k2+k3)-D layer-(L1+n1) output matrix of the decoder: perform a search among all L1 layers of the encoder to identify the m1 layer of the encoder which has an adjusted output matrix with a size matching the size of the layer-(L1+n1) output matrix of the decoder, in order to form the m1−n1 skip connection, wherein:

when no matching layer of the encoder is found, a best match is used for the m1-n1 skip connection,

the best match m1 is a layer of the encoder with a minimum absolute difference between the size of its adjusted output matrix and the size of the layer-(L1+n1) output matrix of the decoder.

20 . The system of claim 17 , wherein the processor is further configured to:

compute each element of the (k2+k3)-D adjusted layer-(m1) output matrix of the encoder as a characteristic value of a respective (k1−k2)-D sub-matrix of the (k1+k3)-D layer-(m1) output matrix of the encoder, wherein:

the characteristic value is an aggregate of the elements in the respective (k1−k2)-D submatrix,

the aggregate comprises at least one of: mean, arithmetic mean, geometric mean, harmonic mean, weighted average, percentile, minimum (0-percentile), median (50-percentile), maximum (100-percentile), trimmed mean (mean of percentile from A1-percentile to A2-percentile), mode, sum, weighted sum, product or weighted product.

21 . The system of claim 12 , wherein the processor is further configured to:

train the encoder and the decoder as part of a conditional generative adversarial network (cGAN) comprising a generator and a discriminator, wherein:

the generator comprises the encoder and the decoder with the at least one skip connection;

the k1-D imaging matrix is an input of the generator;

the k2-D output imaging matrix is an output of the generator; and

the discriminator comprises:

the encoder with the k1-D imaging matrix as input and the (k2+k3)-D second intermediate matrix as output,

an additional encoder with a k2-D supplementary input imaging matrix as input, and

a full-connected network configured to compute a real-fake score.

22 . The system of claim 21 , wherein the processor is further configured to:

train the cGAN with a cost function which is a weighted sum of a standard generative adversarial network (GAN) loss function and an additional cost function, wherein the additional cost function is at least one of:

a L-k loss function or a weighted L-k loss function between the k2-D output imaging matrix and the k2-D supplement input imaging matrix, with k being an integer,

a visual loss function,

a pre-trained perceptual loss function,

a SSIM loss function,

an image structural loss function, or

an image-based loss function.

23 . A device for encoder-decoder-based high resolution imaging, comprising:

a processor; and

a memory storing instructions, which when executed, cause the processor to perform operations comprising:

assembling a k1-dimensional (k1-D) imaging matrix by arranging and concatenating a plurality of k2-dimensional (k2-D) input imaging matrices,

encoding the k1-D imaging matrix by an encoder, which is a k1-D encoding neural network, to generate a (k1+k3)-D first intermediate matrix that includes k1 original dimensions of the k1-D imaging matrix and k3 expanded dimensions,

generating a (k2+k3)-D second intermediate matrix by removing (k1−k2) of the k1 original dimensions in the (k1+k3)-D first intermediate matrix, wherein each of k1, k2, k3 is a positive integer, and

decoding the (k2+k3)-D second intermediate matrix using a decoder, which is a k2-D decoding neural network, to generate a k2-D output imaging matrix, wherein:

there is at least one skip connection between the encoder and the decoder, and

an imaging resolution of the k2-D output imaging matrix is greater than the imaging resolution of any one of the plurality of k2-D input imaging matrices.

24 . The device of claim 23 , wherein the operations further comprise:

determining a (k1+k3)-D expanded matrix by adding k3 expanded dimensions to the k1 original dimensions of the k1-D imaging matrix, wherein:

a size of the (k1+k3)-D expanded matrix is 1 in each of the k3 expanded dimensions,

the encoder comprises L1 layers, each associated with multiple convolution filters,

the (k1+k3)-D expanded matrix is a (k1+k3)-D layer-1 input matrix to a layer 1 of the encoder, the first layer of the L1 layers of the encoder.

25 . The device of claim 24 , wherein the operations further comprise:

for each n between 1 and L1:

for each respective convolution filter of the multiple convolution filters associated with layer n of the encoder:

filtering a (k1+k3)-D layer-n input matrix with the respective convolution filter to generate a respective (k1+k3)-D layer-n per-filter output matrix, and

down-sampling the respective (k1+k3)-D layer-n per-filter output matrix in at least one of the k1 original dimensions with max-pooling; and

concatenating all of the (k1+k3)-D layer-n per-filter output matrices to form a (k1+k3)-D layer-n output matrix which is also a layer-(n+1) input matrix,

wherein:

a size of the layer-n output matrix in each of the k1 original dimensions is smaller than or equal to a size of a corresponding original dimension of the layer-n input matrix,

a size of the layer-n output matrix in each of the k3 expanded dimensions is greater than or equal to a size of a corresponding expanded dimension of the layer-n input matrix,

the (k1+k3)-D first intermediate matrix is the layer-(L1) output matrix, and

a size of the (k1+k3)-D first intermediate matrix is 1 in each of the k1 original dimensions.

26 . The device of claim 25 ,

wherein:

the decoder comprises L2 layers, each associated with multiple de-convolution filters,

the (k2+k3)-D second intermediate matrix is a (k2+k3)-D layer-(L1+1) input matrix to a layer 1 of the decoder, the first layer of the L2 layers of the decoder.

27 . The device of claim 26 , wherein the operations further comprise:

for each m between 1 and L2:

up-sampling a (k2+k3)-D layer-(L1+m) input matrix in at least one of the remaining k2 original dimensions to generate a (k2+k3)-D layer-(L1+m) up-sampled input matrix,

for each respective de-convolution filter of the multiple de-convolution filters associated with layer m of the decoder, filtering the (k2+k3)-D layer-(L1+m) up-sampled input matrix with the respective de-convolution filter to generate a respective (k2+k3)-D layer-(L1+m) per-filter output matrix, and

combining and concatenating all of the (k2+k3)-D layer-(L1+m) per-filter output matrices to form a (k2+k3)-D layer-(L1+m) output matrix which is also a layer-(L1+m+1) input matrix, wherein:

a size of the layer-(L1+m) output matrix in each of the remaining k2 original dimensions is greater than or equal to a size of a corresponding original dimension of the layer-(L1+m) input matrix,

a size of the layer-(L1+m) output matrix in each of the k3 expanded dimensions is less than or equal to a size of a corresponding expanded dimension of the layer-(L1+m) input matrix; and

removing the k3 expanded dimensions of the layer-(L1+L2) output matrix to generate the k2-D output imaging matrix.

28 . The device of claim 27 , wherein the operations further comprise:

processing a m1−n1 skip connection, which is a skip connection between layer m1 of the encoder and layer n1 of the decoder, with 1<=m1<=L1, 1<=n1<=L2, wherein the m1−n1 skip connection is processed based on:

adjusting the (k1+k3)-D layer-(m1) output matrix of the encoder to generate a (k2+k3)-D adjusted layer-(m1) output matrix of the encoder, wherein the (k2+k3)-D adjusted layer-(m1) output matrix has a same dimension and a same size as the (k2+k3)-D layer-(L1+n1) output matrix of the decoder;

computing a temporary output matrix as a weighted average of the (k2+k3)-D adjusted layer-(m1) output matrix of the encoder and the (k2+k3)-D layer-(L1+n1) output matrix of the decoder; and

replacing the (k2+k3)-D layer-(L1+n1+1) input matrix of the decoder by the temporary output matrix.

29 . The device of claim 28 , wherein:

there is a m2-n2 skip connection, which is a skip connection between layer m2 of the encoder and layer n2 of the decoder, with 1<=m1<m2<=L1, 1<=n2<n1<=L2; and

the operations further comprise processing all skip connections in an ascending order of layer index of the decoder such that the m2-n2 skip connection is processed before the m1−n1 skip connection.

30 . The device of claim 29 , wherein the operations further comprise:

for any n1 and the (k2+k3)-D layer-(L1+n1) output matrix of the decoder: performing a search among all L1 layers of the encoder to identify the m1 layer of the encoder which has an adjusted output matrix with a size matching the size of the layer-(L1+n1) output matrix of the decoder, in order to form the m1−n1 skip connection, wherein:

when no matching layer of the encoder is found, a best match is used for the m1-n1 skip connection,

the best match m1 is a layer of the encoder with a minimum absolute difference between the size of its adjusted output matrix and the size of the layer-(L1+n1) output matrix of the decoder.