Image processing method, image processing apparatus, image processing system, and learnt model manufacturing method

1. An image processing method comprising steps of:

obtaining an input image that is at least part of a captured image obtained by an image sensor;

obtaining a first map representing a region outside a dynamic range of the input image based on a signal value in the input image and a threshold of the signal value; and

inputting input data including the input image and the first map to a neural network and executing a recognition task or a regression task,

wherein the threshold of the signal value is set based on at least one of a luminance saturation value and a black level in the input image that is the at least part of the captured image obtained by the image sensor, and

wherein the input data is obtained by concatenating the input image and the first map in a channel direction.

2. The image processing method according to claim 1 , wherein the first map is a map representing at least one of a luminance saturated area and a blocked-up shadow area in the input image.

3. The image processing method according to claim 1 , wherein the input data includes the input image and the first map as a channel component.

4. The image processing method according to claim 3 , wherein the inputting step inputs only one of the input image and the first map to a first layer of the neural network, concatenates, in a channel direction, a feature map that is an output from at least the first layer, with the other of the input image and the first map that has not been input to the first layer, and inputs concatenated data to a subsequent layer of the neural network.

5. The image processing method according to claim 3 , wherein the inputting step branches an input part of the neural network, converts the input image and the first map into feature maps in different layers, concatenates the feature maps, and inputs that to a subsequent layer.

6. The image processing method according to claim 1 , wherein pixel numbers per one channel are equal to each other between the input image and the first map.

7. The image processing method according to claim 1 , wherein the task is to deblur the input image.

8. The image processing method according to claim 1 , further comprising steps of:

obtaining a weight map based on the signal value in the input image and the threshold of the signal value; and

generating a weighted average image based on the output from the neural network, the input image, and the weight map.

9. The image processing method according to claim 8 , wherein the input image includes a plurality of color components, and

wherein in the input image, when luminance saturation or a blocked-up shadow occurs in all of a target pixel and pixels having a color component different from that of the target pixel in a predetermined area, the weight map is generated so that a weight at a position of the target pixel in the input image is larger than the output from the neural network.

10. The image processing method according to claim 8 , wherein the input image has a plurality of color components, and

wherein in the input image, when neither luminance saturation nor a blocked-up shadow occurs in any of the target pixel and a pixel having a color component different from that of the target pixel in the predetermined area, the weight map is generated so that a weight at the position of the target pixel in the input image is smaller than the output from the neural network.

11. An image processing apparatus comprising:

an obtaining unit configured to obtain (i) an input image that is at least part of a captured image obtained by an image sensor and (ii) a first map representing a region outside a dynamic range of the input image based on a signal value in the input image and a threshold of the signal value; and

a processing unit configured to input data including the input image and the first map to a neural network, and to execute a recognition task or a regression task,

wherein the threshold of the signal value is set based on at least one of a luminance saturation value and a black level in the input image that is the at least part of the captured image obtained by the image sensor,

wherein the input data is obtained by concatenating the input image and the first map in a channel direction, and

wherein at least one processor or circuit is configured to perform a function of at least one of the units.

12. The image processing apparatus according to claim 11 , further comprising a memory configured to store weight information used in the neural network.

13. A non-transitory computer-readable storage medium storing a computer program that causes a computer to execute the image processing method according to claim 1 .

14. An image processing system comprising:

a first apparatus; and

a second apparatus communicable with the first apparatus,

wherein the first apparatus includes a transmitter configured to transmit a request to make the second apparatus execute processing on a captured image obtained by an image sensor,

wherein the second apparatus includes:

a receiver configured to receive the request transmitted by the transmitter;

an obtainer configured to obtain (i) at least part of the captured image obtained by the image sensor and (ii) a first map representing a region outside a dynamic range of the captured image based on a signal value in the captured image and a threshold of the signal value;

a processor configured to input data including the captured image and the first map to a neural network and to execute a recognition task or a regression task; and

a transmitter configured to transmit a result of the executed task,

wherein the threshold of the signal value is set based on at least one of a luminance saturation value and a black level in the at least part of the captured image obtained by the image sensor, and

wherein the input data is obtained by concatenating the captured image and the first map in a channel direction.

15. An image processing method comprising steps of:

obtaining a training image that is at least part of a captured image obtained by an image sensor, a first map representing a region outside a dynamic range of the training image based on a signal value in the training image and a threshold of the signal value, and ground truth data; and

making a neural network learn for executing a recognition task or a regression task, using input data including the training image, the first map, and the ground truth data,

wherein the threshold of the signal value is set based on at least one of a luminance saturation value and a black level in the training image that is the at least part of the captured image obtained by the image sensor, and

wherein the input data is obtained by concatenating the training image and the first map in a channel direction.

16. A non-transitory computer-readable storage medium storing a computer program that causes a computer to execute the image processing method according to claim 15 .

17. A learnt model manufacturing method comprising steps of:

obtaining a training image that is at least part of a captured image obtained by an image sensor, a first map representing a region outside a dynamic range of the training image based on a signal value in the training image and a threshold of the signal value, and ground truth data; and

making a neural network learn for executing a recognition task or a regression task using input data including the training image, the first map, and the ground truth data,

wherein the threshold of the signal value is set based on at least one of a luminance saturation value and a black level in the training image that is the at least part of the captured image obtained by the image sensor, and

wherein the input data is obtained by concatenating the training image and the first map in a channel direction.

18. An image processing apparatus comprising:

an obtaining unit configured to obtain a training image that is at least part of a captured image obtained by an image sensor, a first map representing a region outside a dynamic range of the training image based on a signal value in the training image and a threshold of the signal value, and ground truth data; and

a learning unit configured to make a neural network learn for executing a recognition task or a regression task using input data including the training image, the first map, and the ground truth data,

wherein the threshold of the signal value is set based on at least one of a luminance saturation value and a black level in the training image that is the at least part of the captured image obtained by the image sensor,

wherein the input data is obtained by concatenating the training image and the first map in a channel direction, and

wherein at least one processor or circuit is configured to perform a function of at least one of the units.