Image processing apparatus and image processing method
An image processing apparatus that estimates a vector representing a relative positional relationship between parts of a subject using machine learning is disclosed. The image processing apparatus, using a convolutional neural network (CNN), detects, from an image, a first part and a second part of a specific subject, and a relative position vector from the first part to the second part. The CNN is configured to generate a feature map indicating an approximate direction of the relative position vector. The image processing apparatus trains the CNN using a loss function pertaining to the relative position vector and a loss function pertaining to the feature map.
1 . An image processing apparatus comprising:
one or more processors that execute a program stored in a memory and cause the image processing apparatus to:
detect, using a convolutional neural network (CNN), from an image, a first part and a second part of a specific subject, and a relative position vector from the first part to the second part; and
train the CNN,
wherein:
the CNN is configured to generate a feature map indicating an approximate direction of the relative position vector,
the CNN is trained using a loss function pertaining to the relative position vector and a loss function pertaining to the feature map,
the feature map is indicated by an angular range having a predetermined magnitude as the approximate direction of the relative position vector, and
the feature map has a channel for each of the angular ranges, and each of the channels indicates a likelihood that the second part corresponding to the first part is present.
2 . The image processing apparatus according to claim 1 , wherein the program further includes causing the image processing apparatus to:
train the CNN further using a loss function pertaining to a position of the first part and a loss function pertaining to a position of the second part.
3 . The image processing apparatus according to claim 1 , wherein the program further includes causing the image processing apparatus to:
train the CNN by prioritizing the loss function pertaining to the feature map over any other loss function(s) when a number of instances of training is lower than a predetermined value.
4 . The image processing apparatus according to claim 1 , wherein the program further includes causing the image processing apparatus to:
set an area for which calculation of a loss function is not performed in the feature map.
5 . The image processing apparatus according to claim 4 , wherein the program further includes causing the image processing apparatus to:
set an area for which the calculation of the loss function is not performed in the feature map of a channel corresponding to an angular range that does not coincide with the direction of the relative position vector.
6 . The image processing apparatus according to claim 5 , wherein the program further includes causing the image processing apparatus to:
set, as the area for which the calculation of a loss function is not performed, a predetermined range centered on a first part different from the first part corresponding to the relative position vector, of the feature map of the channel corresponding to the angular range that does not coincide with the direction of the relative position vector.
7 . The image processing apparatus according to claim 1 , wherein the program further includes causing the image processing apparatus to:
associates part of a same subject based on the relative position vector detected, for the first part and the second part detected.
8 . The image processing apparatus according to claim 7 , wherein the program further includes causing the image processing apparatus to:
determine the reliability of the relative position vector based on a consistency with the approximate direction indicated by the feature map, and associates parts of a same subject based on the reliability of the relative position vector.
9 . An image processing method executed by an image processing apparatus, the image processing apparatus including a convolutional neural network (CNN), that detects, from an image, a first part and a second part of a specific subject, and a relative position vector from the first part to the second part, the image processing method comprising:
generating, by the CNN, a feature map indicating an approximate direction of the relative position vector; and
training the CNN using a loss function pertaining to the relative position vector and a loss function pertaining to the feature map,
wherein the feature map is indicated by an angular range having a predetermined magnitude as the approximate direction of the relative position vector, and
wherein the feature map has a channel for each of the angular ranges, and each of the channels indicates a likelihood that the second part corresponding to the first part is present.
10 . A non-transitory computer-readable medium that stores a program, which when executed by a computer, causes the computer to function as an image processing apparatus and to execute an image processing method comprising:
using a convolutional neural network (CNN), detects, from an image, a first part and a second part of a specific subject, and a relative position vector from the first part to the second part; and
training the CNN,
wherein;
the CNN is configured to generate a feature map indicating an approximate direction of the relative position vector,
the CNN is trained using a loss function pertaining to the relative position vector and a loss function pertaining to the feature map,
the feature map is indicated by an angular range having a predetermined magnitude as the approximate direction of the relative position vector, and
the feature map has a channel for each of the angular ranges, and each of the channels indicates a likelihood that the second part corresponding to the first part is present.