Learning apparatus, learning method, imaging apparatus, signal processing apparatus, and signal processing method
A learning apparatus according to the present technology includes a learning unit that trains a CNN by using, as training data, a ground truth label prepared for each training image and ground-truth position information indicating a position of a target object in the training image.
1 . A learning apparatus, comprising:
circuitry configured to:
train a convolutional neural network (CNN) by using, as training data, a ground truth label prepared for each training image and ground-truth position information indicating a position of a target object in the training image, wherein
the ground-truth position information is determined based on dividing each training image into a grid and determining a percentage overlap of the target object for each section of the grid, and
the training data includes a first set of training images for which the ground-truth position information is prepared and a second set of training images for which the ground-truth position information is not prepared;
calculate, for a training image included in the first set of training images, a position error between the ground-truth position information and a position of the target object in the training image indicated by a feature value map from an intermediate layer of the CNN;
update parameters of the CNN based on a combined error obtained by combining the position error and an inferred value error, the inferred value error being an error between an inferred value of the CNN with respect to the training image and the ground truth label; and
detect a position of the target object in an image based on the feature value map by determining a core grid having a largest feature index value and determining a boundary of an area based on ratios of the feature index value of the core grid to feature index values of grids adjacent to the core grid.
2 . The learning apparatus according to claim 1 , wherein the circuitry performs the parameter update by using, as the combined error, a value obtained by weighting the inferred value error and the position error.
3 . The learning apparatus according to claim 1 , wherein the circuitry trains the CNN using, as the training data, the ground truth label and the ground-truth position information for each of target objects of different types.
4 . A learning method, comprising:
training, by circuitry of an information processing apparatus, a convolutional neural network (CNN) by using, as training data, a ground truth label prepared for each training image and ground-truth position information indicating a position of a target object in the training image, wherein
the ground-truth position information is determined based on dividing each training image into a grid and determining a percentage overlap of the target object for each section of the grid, and
the training data includes a first set of training images for which the ground-truth position information is prepared and a second set of training images for which the ground-truth position information is not prepared;
calculating, for a training image included in the first set of training images, a position error between the ground-truth position information and a position of the target object in the training image indicated by a feature value map from an intermediate layer of the CNN;
updating parameters of the CNN based on a combined error obtained by combining the position error and an inferred value error, the inferred value error being an error between an inferred value of the CNN with respect to the training image and the ground truth label; and
detecting a position of the target object in an image based on the feature value map by determining a core grid having a largest feature index value and determining a boundary of an area based on ratios of the feature index value of the core grid to feature index values of grids adjacent to the core grid.
5 . An imaging apparatus, comprising:
a pixel array in which a plurality of pixels including a photoelectric conversion element is arranged; and
an image sensor including circuitry configured to:
train a convolutional neural network (CNN) trained by using, as training data, a ground truth label prepared for each training image and ground-truth position information indicating a position of a target object in the training image, and including a signal processing unit that performs processing for image recognition on a captured image obtained by photoelectric conversion in the pixel array, wherein
the ground-truth position information is determined based on dividing each training image into a grid and determining a percentage overlap of the target object for each section of the grid, and
the training data includes a first set of training images for which the ground-truth position information is prepared and a second set of training images for which the ground-truth position information is not prepared;
calculate, for a training image included in the first set of training images, a position error between the ground-truth position information and a position of the target object in the training image indicated by a feature value map from an intermediate layer of the CNN;
update parameters of the CNN based on a combined error obtained by combining the position error and an inferred value error, the inferred value error being an error between an inferred value of the CNN with respect to the training image and the ground truth label; and
detect a position of the target object in an image based on the feature value map by determining a core grid having a largest feature index value and determining a boundary of an area based on ratios of the feature index value of the core grid to feature index values of grids adjacent to the core grid.
6 . The imaging apparatus according to claim 5 , wherein the circuitry is further configured to perform activation control of in an own-apparatus on condition that a target object is recognized in the captured image by processing of image recognition using the CNN.
7 . A signal processing apparatus, comprising:
circuitry configured to detect a position of a target object in an image based on a feature value map obtained in an intermediate layer of a convolutional neural network (CNN), the CNN being trained using, as training data, a ground truth label prepared for each training image and ground-truth position information indicating a position of a target object in the training image, wherein
the ground-truth position information is determined based on dividing each training image into a grid and determining a percentage overlap of the target object for each section of the grid,
the training data includes a first set of training images for which the ground-truth position information is prepared and a second set of training images for which the ground-truth position information is not prepared,
for a training image included in the first set of training images, parameters of the CNN are updated based on a combined error obtained by combining a position error and an inferred value error, the position error is between the ground-truth position information and a position of the target object in the training image indicated by the feature value map, and the inferred value error is an error between an inferred value of the CNN with respect to the training image and the ground truth label, and
the circuitry detects the position of the target object in the image based on the feature value map by determining a core grid having a largest feature index value and determining a boundary of an area based on ratios of the feature index value of the core grid to feature index values of grids adjacent to the core grid.
8 . The signal processing apparatus according to claim 7 , wherein the circuitry detects a position of the target object on a basis of a magnitude of a feature index value for each region in the feature value map.
9 . The signal processing apparatus according to claim 8 , wherein the circuitry detects, as the position of the target object, a position in which the feature index value is equal to or more than a threshold value.
10 . The signal processing apparatus according to claim 7 , wherein the circuitry detects, on a basis of the feature value map generated by the CNN for each of target objects of different types, a position of each the target objects in an image.
11 . The signal processing apparatus according to claim 7 , wherein the circuitry is further configured to perform scene estimation on a basis of a change aspect of a position of the target object detected.
12 . The signal processing apparatus according to claim 11 , wherein
the circuitry detects, on a basis of the feature value map generated by the CNN for each of target objects of different types, a position of each of the target objects in an image, and
the circuitry performs the scene estimation on a basis of a change aspect of a detected position for each the target object.
13 . The signal processing apparatus according to claim 7 , wherein the circuitry further performs, on condition that the target object is recognized at a specific position in an image on a basis of a result of position detection, activation control of a predetermined unit.
14 . A signal processing method, comprising:
detecting, with a signal processing apparatus, a position of a target object in an image based on a feature value map obtained in an intermediate layer of a convolutional neural network (CNN), the CNN being trained using, as training data, a ground truth label prepared for each training image and ground-truth position information indicating a position of a target object in the training image, wherein
the ground-truth position information is determined based on dividing each training image into a grid and determining a percentage overlap of the target object for each section of the grid,
the training data includes a first set of training images for which the ground-truth position information is prepared and a second set of training images for which the ground-truth position information is not prepared,
for a training image included in the first set of training images, parameters of the CNN are updated based on a combined error obtained by combining a position error and an inferred value error, the position error is between the ground-truth position information and a position of the target object in the training image indicated by the feature value map, and the inferred value error is an error between an inferred value of the CNN with respect to the training image and the ground truth label, and
the detecting the position of the target object in the image based on the feature value map includes determining a core grid having a largest feature index value and determining a boundary of an area based on ratios of the feature index value of the core grid to feature index values of grids adjacent to the core grid.
15 . The learning apparatus according to claim 1 , wherein the target object is a human.
16 . The learning apparatus according to claim 1 , wherein
the circuitry is further configured to acquire the feature value map by performing a convolution operation on the training image using a kernel with a predetermined size, and
the feature value map indicates a feature index value for each region of the training image.
17 . The learning apparatus according to claim 1 , wherein the percentage overlap is indicated as a number between 0 and 1, wherein 0 indicates no overlap and 1 indicates maximum overlap.