Object detection based on joint feature extraction
In implementations of the subject matter described herein, a solution for object detection is proposed. First, a feature(s) is extracted from an image and used to identify a candidate object region in the image. Then another feature(s) is extracted from the identified candidate object region. Based on the features extracted in these two stages, a target object region in the image and a confidence for the target object region are determined. In this way, the features that characterize the image from the whole scale and a local scale are both taken into consideration in object recognition, thereby improving accuracy of the object detection.
1. A computer-implemented method comprising:
characterizing, by at least one processor, at least one first feature from an image, including by extracting via a first feature extraction stage the at least one first feature and characterizing the image in a whole scale providing context around object regions;
identifying a first candidate object region defined by a first bounding box in the image based on the at least one first feature, wherein the first candidate object region includes a candidate face region;
identifying a second candidate object region defined by a second bounding box in the image based on the at least one first feature;
determining whether the first candidate object region defined by the first bounding box is overlapped by the second candidate object region defined by the second bounding box;
responsive to a determination that the first candidate object region defined by the first bounding box is overlapped by the second candidate object region defined by the second bounding box:
retaining the first candidate object region and discarding the second candidate region, and
characterizing at least one second feature from the first candidate object region based on the at least one first feature, including by extracting via a second feature extraction stage the at least one second feature and characterizing the image in a local scale, which is at the same scale as the whole scale providing information from within the first candidate object region, the first feature extraction stage and the second feature extraction stage trained based on a joint optimization objective to improve accuracy of object detection;
responsive to a determination that the first candidate object region defined by the first bounding box is not overlapped by the second candidate object region defined by the second bounding box:
retaining both the first and second candidate object regions, and
extracting at least one second feature at the same scale of the whole scale from the first and second candidate object regions; and
determining, based on the at least one first feature and the at least one second feature, a target object region in the image and a confidence for the determined target object region.
2. The method of claim 1 , wherein the identifying the first candidate object region comprises:
detecting positions of facial landmarks based on the at least on first feature; and
identifying the candidate face region in the image based on the positions of facial landmarks.
3. The method of claim 2 , wherein identifying the candidate face region in the image based on the positions of facial landmarks comprises:
mapping the positions of facial landmarks to canonical positions based on a predefined mapping; and
identifying the candidate face region in the image based on the canonical positions.
4. The method of claim 1 , wherein extracting the at least one second feature comprises:
determining a confidence for the first candidate object region based on the at least one first feature;
determining whether to process the first candidate object region or not based on the determined confidence; and
in response to determining to process the first candidate object region, extracting the at least one second feature from the first candidate object region.
5. The method of claim 1 , wherein extracting the at least one first feature comprises:
identifying patches with a predefined size in the image;
constructing a first mask by binarizing values of pixels in the image based on one of the patches that is identified as a candidate object;
masking the image with the first mask; and
extracting the at least one first feature from the masked image.
6. The method of claim 5 , wherein constructing the first mask comprises:
generating an enlarged patch by increasing the predefined size of the patch; and
constructing the first mask by binarizing values of pixels in the image based on the enlarged patch.
7. The method of claim 1 , wherein extracting the at least one first feature comprises:
downsampling the image with a predefined sampling rate;
identifying patches with a predefined size in the downsampled image;
constructing a second mask by binarizing values of pixels in the downsampled image based on one of the patches that is identified as a candidate object;
masking the downsampled image with the second mask; and
extracting the at least one first feature from the masked downsampled image.
8. The method of claim 1 , wherein the at least one first feature is extracted by a first process and the at least one second feature is extracted by a second process, and the first and second processes are trained based on a joint optimization objective.
9. The method of claim 1 , further comprising:
assigning a first confidence score to the first candidate object region,
assigning a second confidence score to the second candidate object region;
determine whether to discard one of the first and second candidate object regions based on the assigned first and second confidence scores, respectively.
10. A device comprising:
a processing unit;
a memory coupled to the processing unit and storing instructions thereon, the instructions, when executed by the processing unit, causing the device to:
characterize at least one first feature from an image, including by extracting via a first feature extraction stage the at least one first feature and characterizing the image in a whole scale providing context around object regions;
identify a first candidate object region defined by a first bounding box in the image based on the at least one first feature, wherein the first candidate object region includes a candidate face region;
identify a second candidate object region defined by a second bounding box in the image based on the at least one first feature;
determine whether the first candidate object region defined by the first bounding box is overlapped by the second candidate object region defined by the second bounding box;
responsive to a determination that the first candidate object region defined by the first bounding box is overlapped by the second candidate object region defined by the second bounding box:
retain the first candidate object region and discard the second candidate region, and
characterize at least one second feature from the first candidate object region based on the at least one first feature, including by extracting via a second feature extraction stage the at least one second feature and characterizing the image in a local scale, which is at the same scale as the whole scale, providing information from within the first candidate object region, the first feature extraction stage and the second feature extraction stage trained based on a joint optimization objective to improve accuracy of object detection;
responsive to a determination that the first candidate object region defined by the first bounding box is not overlapped by the second candidate object region defined by the second bounding box:
retain both the first and second candidate object regions, and
extract at least one second feature at the same scale of the whole scale from the first and second candidate object regions; and
determine, based on the at least one first feature and the at least one second feature, a target object region in the image and a confidence for the determined target object region.
11. The device of claim 10 , wherein when executed by the processing unit, further cause the device to identify the first candidate object region by:
detecting positions of facial landmarks based on the at least one first feature; and
identifying the candidate face region in the image based on the positions of facial landmarks.
12. The device of claim 11 , wherein the instructions, when executed by the processing unit, further cause the device to identify the candidate face region in the image based on the positions of facial landmarks by:
mapping the positions of facial landmarks to canonical positions based on a predefined mapping; and
identifying the candidate face region in the image based on the canonical positions.
13. The device of claim 10 , wherein the instructions, when executed by the processing unit, further cause the device to extract the at least one second feature by:
determining a confidence for the candidate object region based on the at least one first feature;
determining whether to process the candidate object region or not based on the determined confidence; and
in response to determining to process the candidate object region, extracting the at least one second feature from the candidate object region.
14. The device of claim 10 , wherein the instructions, when executed by the processing unit, further cause the device to extract the at least one first feature by:
identifying patches with a predefined size in the image;
constructing a first mask by binarizing values of pixels in the image based on one of the patches that is identified as a candidate object;
masking the image with the first mask; and
extracting the at least one first feature from the masked image.
15. The device of claim 14 , wherein the instructions, when executed by the processing unit, further cause the device to construct the first mask by:
generating an enlarged patch by increasing the predefined size of the patch; and
constructing the first mask by binarizing values of pixels in the image based on the enlarged patch.
16. The device of claim 10 , wherein the instructions, when executed by the processing unit, further cause the device to extract the at least one first feature by:
downsampling the image with a predefined sampling rate;
identifying patches with a predefined size in the downsampled image;
constructing a second mask by binarizing values of pixels in the downsampled image based on one of the patches that is identified as a candidate object;
masking the downsampled image with the second mask; and
extracting the at least one first feature from the masked downsampled image.
17. The device of claim 10 , wherein the at least one first feature is extracted by a first process and the at least one second feature is extracted by a second process, and the first and second processes are trained based on a joint optimization objective.
18. The device of claim 10 , wherein at least one of the at least one first feature and at least one second feature is extracted based on a convolutional neural network (CNN).
19. A storage device comprising machine-executable instructions, the instructions, when executed on at least one processor, causing the processor to:
characterize at least one first feature from an image, including by extracting via a first feature extraction stage the at least one first feature and characterizing the image in a whole scale providing context around object regions;
identify a first candidate object region defined by a first bounding box in the image based on the at least one first feature, wherein the first candidate object region includes a candidate face region;
identify a second candidate object region defined by a second bounding box in the image based on the at least one first feature;
determine whether the first candidate object region defined by the first bounding box is overlapped by the second candidate object region defined by the second bounding box;
responsive to a determination that the first candidate object region defined by the first bounding box is overlapped by the second candidate object region defined by the second bounding box:
retain the first candidate object region and discard the second candidate region, and
characterize at least one second feature from the first candidate object region based on the at least one first feature, including by extracting via a second feature extraction stage the at least one second feature to characterize the image in a local scale, which is at the same scale as the whole scale providing information from within the first candidate object region, the first feature extraction stage and the second feature extraction stage trained based on a joint optimization objective to improve accuracy of object detection;
responsive to a determination that the first candidate object region defined by the first bounding box is not overlapped by the second candidate object region defined by the second bounding box:
retain both the first and second candidate object regions, and
extract at least one second feature at the same scale of the whole scale from the first and second candidate object regions; and
determine, based on the at least one first feature and the at least one second feature, a target object region in the image and a confidence for the determined target object region.
20. The storage device of claim 19 , wherein the instructions, when executed on the processor, further cause the processor to:
detect positions of facial landmarks based on the at least one first feature; and
identify the candidate face region in the image based on the positions of facial landmarks.