Granted Patent B2

US 12,327,329 · App. 17/891,687 · Granted Jun 10, 2025

Inventor: Yukihiro Sasagawa (Yokohama, JP)

Assignee: SOCIONEXT INC.

G06T5/00G06T2207/10024G06T2207/10048G06T2207/20081G06T2207/20084

Abstract

Claims (33)

1. A color image inpainting method comprising:

obtaining a color image of an object to be recognized, the color image including a missing portion where at least part of image information is missing;

obtaining an infrared image of the object;

identifying the missing portion in the color image; and

after the missing portion in the color image is identified, inpainting the missing portion in the color image identified in the identifying to obtain an inpainted color image of the object,

wherein the inpainting includes inpainting the missing portion by using information which is obtained only from a part of the infrared image, the part corresponding to the missing portion in the color image.

2. The color image inpainting method according to claim 1 ,

wherein an image of a high-frequency component extracted from the infrared image is used for the inpainting.

3. The color image inpainting method according to claim 1 ,

wherein the identifying includes identifying the missing portion based on (i) at least one of a chroma or a luminance of the color image and (ii) at least one of a contrast or a sharpness in the infrared image.

4. The color image inpainting method according to claim 2 ,

wherein the identifying includes identifying the missing portion based on (i) at least one of a chroma or a luminance of the color image and (ii) at least one of a contrast or a sharpness in the infrared image.

5. The color image inpainting method according to claim 1 ,

wherein the inpainting includes inpainting the missing portion by an inference engine trained to infer a color image with no missing portion from a color image with a missing portion and an infrared image.

6. The color image inpainting method according to claim 3 ,

wherein the inpainting includes inpainting the missing portion by an inference engine trained to infer a color image with no missing portion from a color image with a missing portion and an infrared image.

7. The color image inpainting method according to claim 4 ,

wherein the inpainting includes inpainting the missing portion by an inference engine trained to infer a color image with no missing portion from a color image with a missing portion and an infrared image.

8. The color image inpainting method according to claim 2 ,

wherein the inpainting includes:

preparing a first image, a first infrared image and missing portion information; and

inpainting the missing portion by an inference engine trained to infer a first color image from a second color image with a missing portion and a second infrared image, and

wherein the second infrared image is an image of a high-frequency component extracted from the first infrared image, and the second color image with the missing portion is generated from the first color image and the missing portion information.

9. A training method of training a neural network which infers an inpainted color image of an object to be recognized from a color image of the object including a missing portion where at least part of image information is missing, the inpainted color image being the color image in which the missing portion is inpainted, the training method comprising:

preparing a color image of the object;

preparing an infrared image of the object;

preparing missing portion information indicating the missing portion and including (i) at least one of a chroma or a luminance of the color image and (ii) at least one of a contrast or a sharpness in the infrared image;

preparing a masked color image by masking the color image with the missing portion information, the masked color image being a color image including the missing portion; and

inputting the masked color image, the infrared image, and the missing portion information to the neural network, and training the neural network with the color image as training data.

10. The training method according to claim 9 ,

wherein the preparing of the infrared image includes preparing the infrared image by capturing the object with an infrared camera.

11. The training method according to claim 9 ,

wherein the preparing of the infrared image includes preparing the infrared image by generating a grayscale image from the color image and further extracting a high-frequency component from the grayscale image.

Assignments (1)

Continuity (3)

References Cited (19)