Image processing apparatus, image processing method, image processing system, and storage medium

1 . An image processing apparatus, comprising:

an image sensor configured to acquire a captured image;

at least one memory storing a program; and

at least one processor that executes the stored program, which causes the at least one processor to:

receive an instruction with respect to the captured image;

input the captured image to a trained neural network having a plurality of layers and obtain, as outputs of the trained neural network:

(i) a likelihood map comprising a one-channel tensor having a plurality of rows and a plurality of columns, each element of the likelihood map indicating likelihood of presence of a center of an object in a corresponding predetermined region of regions into which the captured image is divided; and

(ii) an object region candidate tensor comprising four channels having the plurality of rows and the plurality of columns, the four channels respectively indicating an x-direction offset, a y-direction offset, a width, and a height for an object region candidate;

correct the likelihood map based on a distance from the position indicated by the instruction to each element of the likelihood map to generate a corrected likelihood map;

determine an object region corresponding to the instruction using the corrected likelihood map and one or more object region candidates selected based on the position indicated by the instruction; and

generate a focus adjustment signal for the image capturing apparatus based on the determined object region.

2 . The image processing apparatus according to claim 1 , wherein the at least one processor further sequentially selects a predetermined number of candidates in order from an object region at a nearest neighbor position with respect to the position indicated by the instruction.

3 . The image processing apparatus according to claim 1 , wherein the at least one processor further determines the object region corresponding to the instruction without using an object region that is estimated to have likelihood that is less than or equal to a threshold and low probability of presence of the object.

4 . The image processing apparatus according to claim 1 , wherein the at least one processor further integrates the object region candidates using weighted averaging of values of the likelihood map.

5 . The image processing apparatus according to claim 1 , wherein the at least one processor further integrates the object region candidates based on a distance between the position indicated by the instruction and each of the object region candidates.

6 . The image processing apparatus according to claim 1 , wherein the at least one processor further makes a correction to shift a center position of the object region to a position at which likelihood increases, in a case where likelihood at the position indicated by the instruction is higher than likelihood in the object region on the likelihood map.

7 . The image processing apparatus according to claim 1 , wherein the at least one processor further selects an object region candidate corresponding to one or more coordinates in a concentric circle about the position indicated by the instruction.

8 . The image processing apparatus according to claim 1 , wherein the at least one processor further acquires the likelihood map using a neural network having a plurality of layers.

9 . The image processing apparatus according to claim 1 , wherein the at least one processor further estimates the region using a neural network having a plurality of layers.

10 . An image processing system, comprising:

an image sensor configured to acquire a captured image;

at least one memory storing a program; and

at least one processor that executes the stored program, which causes the at least one processor to:

receive an instruction with respect to the captured image;

input the captured image to a trained neural network having a plurality of layers and obtain, as outputs of the trained neural network:

(i) a likelihood map comprising a one-channel tensor having a plurality of rows and a plurality of columns, each element of the likelihood map indicating likelihood of presence of a center of an object in a corresponding predetermined region of regions into which the captured image is divided; and

(ii) an object region candidate tensor comprising four channels having the plurality of rows and the plurality of columns, the four channels respectively indicating an x-direction offset, a y-direction offset, a width, and a height for an object region candidate;

correct the likelihood map based on a distance from the position indicated by the instruction to each element of the likelihood map to generate a corrected likelihood map;

determine an object region corresponding to the instruction using the corrected likelihood map and one or more object region candidates selected based on the position indicated by the instruction received; and

generate a focus adjustment signal for the image capturing apparatus based on the determined object region.

11 . An image processing method, comprising:

acquiring a captured image;

receiving an instruction with respect to the captured image;

inputting the captured image to a trained neural network having a plurality of layers and obtain, as outputs of the trained neural network:

(i) a likelihood map comprising a one-channel tensor having a plurality of rows and a plurality of columns, each element of the likelihood map indicating likelihood of presence of a center of an object in a corresponding predetermined region of regions into which the captured image is divided; and

(ii) an object region candidate tensor comprising four channels having the plurality of rows and the plurality of columns, the four channels respectively indicating an x-direction offset, a y-direction offset, a width, and a height for an object region candidate;

correcting the likelihood map based on a distance from the position indicated by the instruction to each element of the likelihood map to generate a corrected likelihood map;

determining an object region corresponding to the instruction using the corrected likelihood map and one or more object region candidates selected based on the position indicated by the received instruction; and

generating a focus adjustment signal for the image capturing apparatus based on the determined object region.

12 . A non-transitory computer-readable storage medium storing a computer-executable program for causing a computer to perform the method according to claim 11 .