Method, system, and computer-readable medium for improving color quality of images

1. A computer-implemented method, comprising:

receiving and processing a first color image by an encoder,

wherein

the first color image comprises a first portion of the first color image and a second portion of the first color image located at different locations of the first color image; and

the encoder is configured to output at least one first feature map comprising fused global information and local information such that whether a color consistency relationship between the first portion of the first color image and the second portion of the first color image exists is encoded into the fused global information and local information, wherein the encoder comprises:

a first block;

a second block; and

a first skip connection,

wherein

the first block comprises:

a convolutional block configured to output at least one second feature map comprising local information and has a first receptive field; and

the second block comprises:

a global pooling layer configured to perform global pooling on the at least one second feature map, and output at least one third feature map comprising global information, and has a second receptive field wider than the first receptive field; and

an upscaling layer configured to upscale the at least one third feature map and output at least one fourth feature map having a same scale as the at least one second feature map and comprising the global information; and

the first skip connection is configured to fuse the at least one second feature map and the at least one fourth feature map, to generate the at least one first feature map, such that the at least one first feature map has a same number of channels as a number of channels of the at least one second feature map, wherein the fused global information and local information is obtained from the global information and the local information.

2. The computer-implemented method of claim 1 , wherein the first block causes a scale of the at least one first feature map to be smaller than a scale of the first color image.

3. The computer-implemented method of claim 1 , wherein

when the convolutional block considers each of the first portion of the first color image and the second portion of the first color image in a corresponding first view with a size of the first receptive field, no semantics attributed to any of the first portion of the first color image and the second portion of the first color image is encoded into the local information; and

when the global pooling layer considers both of the first portion of the first color image and the second portion of the first color image in a second view with a size of the second receptive field, a first semantics attributed to the first portion of the first color image and the second portion of the first color image is encoded into the global information, whereby the color consistency relationship between the first portion of the first color image and the second portion of the first color image exists.

4. The computer-implemented method of claim 1 , wherein

when the convolutional block considers one of the first portion of the first color image and the second portion of the first color image in a first view with a size of the first receptive field, a first semantics attributed to the one of the first portion of the first color image and the second portion of the first color image is encoded into the local information; and

when the global pooling layer considers both of the first portion of the first color image and the second portion of the first color image in a second view with a size of the second receptive field, a second semantics attributed to the first portion of the first color image and the second portion of the first color image is encoded into the global information, whereby the color consistency relationship between the first portion of the first color image and the second portion of the first color image does not exist.

5. The computer-implemented method of claim 1 , wherein the upscaling layer is an upsampling layer free of learnable parameters.

6. The computer-implemented method of claim 5 , wherein the upsampling layer is configured to upscale the at least one third feature map by performing a bilinear upsampling operation on the at least one third feature map.

7. The computer-implemented method of claim 1 , wherein the first skip connection is configured to fuse the at least one second feature map and the at least one fourth feature map by performing an element-wise summation operation on the at least one second feature map and the at least one fourth feature map.

8. The computer-implemented method of claim 1 , further comprising:

receiving and processing the at least one first feature map by a decoder, wherein the decoder is configured to output a second color image generated from the at least one first feature map, wherein a first portion of the second color image corresponding to the first portion of the first color image and a second portion of the second color image corresponding to the second portion of the first color image are restored considering whether the color consistency relationship between the first portion of the first color image and the second portion of the first color image exists, wherein the decoder comprises a plurality of convolutional blocks.

9. The computer-implemented method of claim 8 , wherein the decoder causes a scale of the second color image to be larger than a scale of the at least one first feature map.

10. A system, comprising:

at least one memory configured to store program instructions;

at least one processor configured to execute the program instructions, which cause the at least one processor to perform steps comprising:

receiving and processing a first color image by an encoder,

wherein

the first color image comprises a first portion of the first color image and a second portion of the first color image located at different locations of the first color image; and

the encoder is configured to output at least one first feature map comprising fused global information and local information such that whether a color consistency relationship between the first portion of the first color image and the second portion of the first color image exists is encoded into the fused global information and local information, wherein the encoder comprises:

a first block;

a second block; and

a first skip connection,

wherein

 the first block comprises:

 a convolutional block configured to output at least one second feature map comprising local information and has a first receptive field;

 the second block comprises:

 a global pooling layer configured to perform global pooling on the at least one second feature map, and output at least one third feature map comprising global information, and has a second receptive field wider than the first receptive field; and

 an upscaling layer configured to upscale the at least one third feature map and output at least one fourth feature map having a same scale as the at least one second feature map and comprising the global information; and

 the first skip connection is configured to fuse the at least one second feature map and the at least one fourth feature map, to generate the at least one first feature map, such that the at least one first feature map has a same number of channels as a number of channels of the at least one second feature map, wherein the fused global information and local information is obtained from the global information and the local information.

11. The system of claim 10 , wherein the first block causes a scale of the at least one first feature map to be smaller than a scale of the first color image.

12. The system of claim 10 , wherein

when the convolutional block considers each of the first portion of the first color image and the second portion of the first color image in a corresponding first view with a size of the first receptive field, no semantics attributed to any of the first portion of the first color image and the second portion of the first color image is encoded into the local information; and

when the global pooling layer considers both of the first portion of the first color image and the second portion of the first color image in a second view with a size of the second receptive field, a first semantics attributed to the first portion of the first color image and the second portion of the first color image is encoded into the global information, whereby the color consistency relationship between the first portion of the first color image and the second portion of the first color image exists.

13. The system of claim 10 , wherein

when the convolutional block considers one of the first portion of the first color image and the second portion of the first color image in a first view with a size of the first receptive field, a first semantics attributed to the one of the first portion of the first color image and the second portion of the first color image is encoded into the local information; and

when the global pooling layer considers both of the first portion of the first color image and the second portion of the first color image in a second view with a size of the second receptive field, a second semantics attributed to the first portion of the first color image and the second portion of the first color image is encoded into the global information, whereby the color consistency relationship between the first portion of the first color image and the second portion of the first color image does not exist.

14. The system of claim 10 , wherein the upscaling layer is an upsampling layer free of learnable parameters.

15. The system of claim 14 , wherein the upsampling layer is configured to upscale the at least one third feature map by performing a bilinear upsampling operation on the at least one third feature map.

16. The system of claim 10 , wherein the first skip connection is configured to fuse the at least one second feature map and the at least one fourth feature map by performing an element-wise summation operation on the at least one second feature map and the at least one fourth feature map.

17. The system of claim 10 , wherein the steps further comprises:

receiving and processing the at least one first feature map by a decoder, wherein the decoder is configured to output a second color image generated from the at least one first feature map, wherein a first portion of the second color image corresponding to the first portion of the first color image and a second portion of the second color image corresponding to the second portion of the first color image are restored considering whether the color consistency relationship between the first portion of the first color image and the second portion of the first color image exists.

18. The system of claim 17 , wherein the decoder causes a scale of the second color image to be larger than a scale of the at least one first feature map.

19. A non-transitory computer-readable medium with program instructions stored thereon, that when executed by at least one processor, cause the at least one processor to perform steps comprising:

receiving and processing a first color image by an encoder,

wherein

the first color image comprises a first portion of the first color image and a second portion of the first color image located at different locations of the first color image; and

the encoder is configured to output at least one first feature map comprising fused global information and local information such that whether a color consistency relationship between the first portion of the first color image and the second portion of the first color image exists is encoded into the fused global information and local information, wherein the encoder comprises:

a first block;

a second block; and

a first skip connection,

wherein

the first block comprises:

a convolutional block configured to output at least one second feature map comprising local information and has a first receptive field;

the second block comprises:

a global pooling layer configured to perform global pooling on the at least one second feature map, and output at least one third feature map comprising global information, and has a second receptive field wider than the first receptive field; and

an upscaling layer configured to upscale the at least one third feature map and output at least one fourth feature map having a same scale as the at least one second feature map and comprising the global information; and

the first skip connection is configured to fuse the at least one second feature map and the at least one fourth feature map, to generate the at least one first feature map, such that the at least one first feature map has a same number of channels as a number of channels of the at least one second feature map, wherein the fused global information and local information is obtained from the global information and the local information.