Mouth removal method for red-eye detection and correction
View Patent ↗An input image (e.g. a digital RGB color image) is subjected to an eye classifier that is targeted at discriminating a complete eye pattern from any non-eye patterns. The red-eye candidate list with associated bounding boxes that are generated by the red-eye classifier are received. The bounding rectangles are subjected to object segmentation. A connected component labeling procedure is then applied to obtain one or more red regions. The largest red region is then chosen for feature extraction. A number of features are then extracted from this region. Then these features are used to determine if the particular candidate red-eye object is a mouth.
1. A method for determining if a candidate for red-eye removal processing is a mouth, comprising:
receiving from an eye classifier a plurality of candidate red-eye objects, each of the candidate red-eye objects contained within a bounding rectangle;
for each particular candidate red-eye object:
segmenting the candidate red-eye object to obtain a segmented red region within the candidate red-eye object's bounding rectangle;
extracting a feature k w , defined as the ratio between a width of the segmented red region and a width of the bounding rectangle;
extracting a feature k h , defined as the ratio between a height of the segmented red region and a height of the bounding rectangle;
extracting a feature s, defined as the ratio between the width and height of the segmented red region;
extracting a feature v, defined as a decision score from the eye classifier;
extracting a feature m, defined as a geometrical relationship between the particular candidate red-eye object and others of the plurality of candidate red-eye objects; and
using the features k w , k h , s, v, and m to determine if the particular candidate red-eye object is a mouth.
2. The method as recited in claim 1 , wherein extracting the feature m comprises determining the geometrical relationship between the particular candidate red-eye object and others of the plurality of candidate red-eye objects and flagging the feature m as Mouth_Eye_Pair if one or two of the other of the plurality of candidate red-eye objects are found in the top side region within a predetermined distance of the particular candidate red-eye object; flagging the feature m as Eye_Eye_Pair if one of the other of the plurality of candidate red-eye objects is found in a left or right side region within a predetermined distance of the particular candidate red-eye object; and flagging the feature m as No_Pair in all other cases.
3. The method as recited in claim 2 , wherein using the features k w , k h , s, v, and m to determine if the particular candidate red-eye object is a mouth includes:
comparing k w to k h , and comparing k w to a threshold t kw and if k w >k h and k w >t kw , then a horizontal mouth condition is decided for the particular candidate red-eye object and feature m is extracted based on a horizontal mouth condition.
4. The method as recited in claim 3 where m is extracted based on a horizontal mouth condition and wherein,
the particular candidate red-eye object is determined to be a mouth if m is not flagged as Eye_Eye_Pair and feature s is larger than a threshold t s , where t kw =0.7 and t s =0.79.
5. The method as recited in claim 2 , wherein using the features k w , k h , s, v, and m to determine if the particular candidate red-eye object is a mouth includes:
comparing k h to k w , and comparing k h to a threshold t kh and if k h >k w and k h >t kh , then a vertical condition is decided for the particular candidate red-eye object and feature m is extracted based on a vertical mouth condition.
6. The method as recited in claim 5 where m is extracted based on a vertical mouth condition and wherein,
the particular candidate red-eye object is determined to be a mouth if v is larger than a threshold t v and Max(W LAB newFeaObj ,H LAB newFeaObj ) is larger than a threshold t hw , where t kh =0.6, t v =6, t hw =6, and Max(W LAB newFeaObj ,H LAB newFeaObj ) represents the larger value of W LAB newFeaObj and H LAB newFeaObj .
7. The method of claim 1 , wherein segmenting the candidate red-eye comprises:
for pixels in the bounding rectangle received from the eye classifier, transforming the pixels from RGB color space to LAB color space;
calculating thresholds for A and (A−B), denoted as T A and T A — B respectively, using an automatic threshold selection algorithm;
applying a binarization procedure by determining if each pixel inside the bounding rectangle received from the eye classifier is a red pixel or non-red pixel according to the following equation:
b
(
x
,
y
)
=
{
1
if
A
(
x
,
y
)
>
T
A
and
(
A
(
x
,
y
)
-
B
(
x
,
y
)
)
>
T
A
_
B
0
otherwise
where b(x,y)=1 denotes a red pixel at coordinate (x, y) and b(x,y)=0 denotes a non-red pixel at coordinate (x, y), the binarization procedure resulting in a binary image.
8. The method of claim 7 , further comprising applying a connected component labeling technique to the binary image to obtain one or more connected red regions, and labeling a red region with the largest size newFeaObj.
9. The method of claim 8 , further comprising calculating the width W newFeaObj and height H newFeaObj of the red region newFeaObj.
10. The method of claim 7 , wherein transforming the pixels from RGB color space to LAB color space comprises:
first converting the pixels from RGB color space to LAB color space; then normalizing the pixels in LAB space to [0, 1], and then mapping the normalized pixels into [0, 255].
11. One or more non-transitory computer-readable media having computer-readable instructions thereon, which, when executed by a processor, implement a method for determining if a candidate for red-eye removal processing is a mouth, comprising:
receiving from an eye classifier a plurality of candidate red-eye objects, each of the candidate red-eye objects contained within a bounding rectangle;
for each particular candidate red-eye object:
segmenting the candidate red-eye object to obtain a segmented red region within the candidate red-eye object's bounding rectangle;
extracting a feature k w , defined as the ratio between a width of the segmented red region and a width of the bounding rectangle;
extracting a feature k h , defined as the ratio between a height of the segmented red region and a height of the bounding rectangle;
extracting a feature s, defined as the ratio between the width and height of the segmented red region;
extracting a feature v, defined as a decision score from the eye classifier;
extracting a feature m, defined as a geometrical relationship between the particular candidate red-eye object and others of the plurality of candidate red-eye objects; and
using the features k w , k h , s, v, and m to determine if the particular candidate red-eye object is a mouth.
12. The one or more non-transitory computer-readable media as recited in claim 11 , wherein extracting the feature m comprises determining the geometrical relationship between the particular candidate red-eye object and others of the plurality of candidate red-eye objects and flagging the feature m as Mouth_Eye_Pair if one or two of the other of the plurality of candidate red-eye objects are found in the top side region within a predetermined distance of the particular candidate red-eye object; flagging the feature m as Eye_Eye_Pair if one of the other of the plurality of candidate red-eye objects is found in a left or right side region within a predetermined distance of the particular candidate red-eye object; and flagging the feature m as No_Pair in all other cases.
13. The one or more non-transitory computer-readable media as recited in claim 12 , wherein using the features k w , k h , s, v, and m to determine if the particular candidate red-eye object is a mouth includes:
comparing k w to k h , and comparing k w to a threshold t kw and if k w >k h and k w >t kw , then a horizontal mouth condition is decided for the particular candidate red-eye object and feature m is extracted based on a horizontal mouth condition.
14. The one or more non-transitory computer-readable media as recited in claim 13 where m is extracted based on a horizontal mouth condition and wherein,
the particular candidate red-eye object is determined to be a mouth if m is not flagged as Eye_Eye_Pair and feature s is larger than a threshold t s , where t kw =0.7 and t s =0.79.
15. The one or more non-transitory computer-readable media as recited in claim 12 , wherein using the features k w , k h , s, v, and m to determine if the particular candidate red-eye object is a mouth includes:
comparing k h to k w , and comparing k h to a threshold t kh and if k h >k w and k h >t kh , then a vertical condition is decided for the particular candidate red-eye object and feature m is extracted based on a vertical mouth condition.
16. An image capture device for determining if a candidate for red-eye removal processing is a mouth, comprising:
a processor that:
receives from an eye classifier a plurality of candidate red-eye objects, each of the candidate red-eye objects contained within a bounding rectangle;
for each particular candidate red-eye object:
segments the candidate red-eye object to obtain a segmented red region within the candidate red-eye object's bounding rectangle;
extracts a feature k w , defined as the ratio between a width of the segmented red region and a width of the bounding rectangle;
extracts a feature k h , defined as the ratio between a height of the segmented red region and a height of the bounding rectangle;
extracts a feature s, defined as the ratio between the width and height of the segmented red region;
extracts a feature v, defined as a decision score from the eye classifier;
extracts a feature m, defined as a geometrical relationship between the particular candidate red-eye object and others of the plurality of candidate red-eye objects; and
uses the features k w , k h , s, v, and m to determine if the particular candidate red-eye object is a mouth.
17. The image capture device as recited in claim 16 , wherein extracting the feature m comprises determining the geometrical relationship between the particular candidate red-eye object and others of the plurality of candidate red-eye objects and flagging the feature m as Mouth_Eye_Pair if one or two of the other of the plurality of candidate red-eye objects are found in the top side region within a predetermined distance of the particular candidate red-eye object; flagging the feature m as Eye_Eye_Pair if one of the other of the plurality of candidate red-eye objects is found in a left or right side region within a predetermined distance of the particular candidate red-eye object; and flagging the feature m as No_Pair in all other cases.
18. The image capture device as recited in claim 17 , wherein using the features k w , k h , s, v, and m to determine if the particular candidate red-eye object is a mouth includes:
comparing k w to k h , and comparing k w to a threshold t kw and if k w >k h and k w >t kw , then a horizontal mouth condition is decided for the particular candidate red-eye object and feature m is extracted based on a horizontal mouth condition.
19. The image capture device as recited in claim 18 where m is extracted based on a horizontal mouth condition and wherein,
the particular candidate red-eye object is determined to be a mouth if m is not flagged as Eye_Eye_Pair and feature s is larger than a threshold t s , where t kw =0.7 and t s =0.79.
20. The image capture device as recited in claim 17 , wherein using the features k w , k h , s, v, and m to determine if the particular candidate red-eye object is a mouth includes:
comparing k h to k w , and comparing k h to a threshold t kh and if k h >k w and k h >t kh , then a vertical condition is decided for the particular candidate red-eye object and feature m is extracted based on a vertical mouth condition.