Real-time body segmentation system
View Patent ↗In a human feature recognition system that is intended to provide substantially real-time recognition of body segments, various methods and structures are provided to facilitate real-time recognition with reduced computation requirements, including a face detection module employing an active boosting procedure and a lazy boosting procedure on a hybrid cascade structure, a human body segmentation module and a boundary matting module. The hybrid cascade structure is in the form of a tree where one type of node represents a strong classifier learned from active boosting, another type of classifier is obtained by low-computation-load lazy boosting, and weak classifiers are obtained from the previous layers.
1. A method for recognizing characteristics and identity of a selected human body part in order to recognize a human, the method comprising:
capturing, by a computer system as a dataset an image of at least a portion of the human body having a selected human body part, the human body being shown in a sequence of video frames, the portion of the human body being distinguished thereby from a background;
identifying, by the computer system, a sub-dataset corresponding to the selected body part from the dataset representing the image by subjecting the dataset to a plurality of features-based rejection tests;
segmenting, by the computer system, the sub-dataset for the selected body part according to identified sub-dataset segments corresponding to selected identities;
matching, by the computer system, the sub-dataset segments to said selected identities; and
reporting, by the computer system, the known identities as an output to an output device;
wherein said segmenting step employs active boosting, said active boosting characterized by importance weights sampling, a Kullback-Leibler confidence map, a mean-shift based region partition, and active feature selection.
2. The method according to claim 1 wherein the selected body part is the human face and the features based rejection test employs a region variance set based rejection classifier.
3. The method according to claim 2 wherein the features based rejection test further employs, as a further classifier, a feature set constructed of differences between only low-low non-normalized Haar transform coefficients by computing Fast Walsh-Hadamard transforms.
4. The method according to claim 3 wherein the feature set is formed of a plurality of brick-like features combined by addition, subtraction, and absolute value operations.
5. The method according to claim 1 wherein said feature set comprises a plurality of linear combinations of between two and four rectangular blocks that are used to generate features representing Haar transform coefficients.
6. The method according to claim 5 wherein feature set classifying comprises:
performing, by the computer system, the active boosting procedure on each two-block feature;
recording, by the computer system, a classification error for each boosting procedure; thereafter
selecting, by the computer system, the two-block feature with lowest error; by
sorting the two-block features according to errors in order; and
selecting a plurality of top two-block features;
thereafter
composing, by the computer system, four-block features from the top two-block features; thereafter
performing, by the computer system, the boosting procedure on each of the four-block features;
recording, by the computer system, classification error for the boosting procedure; thereafter
comparing, by the computer system, the classification errors of the four-block features with the classification errors of the top two-block feature; and
selecting, by the computer system, that feature with the lowest overall error.
7. The method according to claim 1 wherein said active feature selection comprises searching for features using a coarse criterion, thereafter searching for features using a fine criterion.
8. A method for recognizing characteristics and identity of a selected human body part in order to recognize a human, the method comprising:
capturing, by a computer system, as a dataset an image of at least a portion of the human body having a selected human body part, the human body being shown in a sequence of video frames, the portion of the human body being distinguished thereby from a background;
identifying, by the computer system, a sub-dataset corresponding to the selected body art from the dataset representing the image by subjecting the dataset to a plurality of features-based rejection tests;
segmenting, by the computer system, the sub-dataset for the selected body part according to identified sub-dataset segments corresponding to selected identities;
matching, by the computer system, the sub-dataset segments to said selected identities; and
reporting, by the computer system, the known identities as an output to an output device;
wherein said segmenting step includes employing foreground information of a previous image frame for current computation, and updating foreground information in preparation for a subsequent frame in order to promote fast multi-level segmentation; and
wherein foreground information is updated according to the following functions:
μ
l
F
(
n
)
=
1
W
∑
k
μ
k
F
(
n
)
exp
(
-
1
2
[
μ
k
F
(
n
)
-
μ
l
F
(
1
)
]
T
∑
k
-
1
[
μ
k
F
(
n
)
-
μ
l
F
(
1
)
]
)
σ
l
F
(
n
)
=
1
W
∑
k
σ
k
F
(
n
)
exp
(
-
1
2
[
σ
k
F
(
n
)
-
σ
l
F
(
1
)
]
T
∑
k
-
1
[
σ
k
F
(
n
)
-
σ
l
F
(
1
)
]
)
With
:
W
=
∑
k
exp
(
-
1
2
[
σ
k
F
(
n
)
-
σ
l
F
(
1
)
]
T
∑
k
-
1
[
σ
k
F
(
n
)
-
σ
l
F
(
1
)
]
)
where W is the normalizing factor, μ i F (n) and σ i F (n) denote the mean and variance of i th components of the foreground for the frame n.
9. A method for recognizing characteristics and identity of a selected human body part in order to recognize a human, the method comprising:
capturing, by a computer system, as a dataset an image of at least a portion of the human body having a selected human body part, the human body being shown in a sequence of video frames, the portion of the human body being distinguished thereby from a background;
identifying, by the computer system, a sub-dataset corresponding to the selected body part from the dataset representing the image by subjecting the dataset to a plurality of features-based rejection tests;
segmenting, by the computer system, the sub-dataset for the selected body part according to identified sub-dataset segments corresponding to selected identities;
matching, by the computer system, the sub-dataset segments to said selected identities; and
reporting, by the computer system, the known identities as an output to an output device;
wherein said segmentation step includes automatically generating trimaps using energy minimization.
10. A system for recognizing humans through recognition of characteristics and identity of a selected human body part in order to recognize a human, the system comprising:
code in form of tangible nontransitory computer readable media for capturing as a dataset an image of at least a portion of the human body having a selected human body part, the human body being shown in a sequence of video frames, the portion of the human body being distinguished thereby from a background;
code in form of tangible nontransitory computer readable media for identifying a sub-dataset corresponding to the selected body part from the dataset representing the image by subjecting the dataset to at plurality of features-based rejection tests;
code in form of tangible nontransitory computer readable media for segmenting the sub-dataset for the selected body part according to identified sub-dataset segments corresponding to selected identities;
code in form of tangible nontransitory computer readable media for matching the sub-dataset segments to said selected identities; and
code in form of tangible nontransitory computer readable media for reporting the known identities as an output to an output device;
wherein the selected body part is the human face and the features based rejection test employs a region variance feature set as a first classifier, the feature set being formulated from integral images used for lighting correction during input scanning and including code in form of tangible nontransitory computer readable media for constructing, as a second classifier, a feature set of differences between only low-low non-normalized Haar transform coefficients.
11. The system according to claim 10 including code in form of tangible nontransitory computer readable media for feature set classifying comprising:
code in form of tangible nontransitory computer readable media for performing an active boosting procedure on each two-block feature;
code in form of tangible nontransitory computer readable media for recording a classification error for each boosting procedure;
code in form of tangible nontransitory computer readable media selecting the two-block feature with lowest error; by
sorting the two-block features according to errors in order; and
selecting a plurality of top two-block features;
code in form of tangible nontransitory computer readable media for composing four-block features from the top two-block features; thereafter
code in form of tangible nontransitory computer readable media for performing the boosting procedure on each of the four-block features;
code in form of tangible nontransitory computer readable media for recording classification error for the boosting procedure;
code in form of tangible nontransitory computer readable media for comparing the classification errors of the four-block features with the classification errors of the top two-block feature; and
code in form of tangible nontransitory computer readable media for selecting that feature with the lowest overall error.
12. A system for recognizing humans through recognition of characteristics and identity of a selected human body part in order to recognize a human, the system comprising:
code in form of tangible nontransitory computer readable media for capturing as a dataset an image of at least a portion of the human body having a selected human body part, the human body being shown in a sequence of video frames, the portion of the human body being distinguished thereby from a background;
code in form of tangible nontransitory computer readable media for identifying a sub-dataset corresponding to the selected body part from the dataset representing the image by subjecting the dataset to at plurality of features-based rejection tests;
code in form of tangible nontransitory computer readable media for segmenting the sub-dataset for the selected body part according to identified sub-dataset segments corresponding to selected identities;
code in form of tangible nontransitory computer readable media for matching the sub-dataset segments to said selected identities; and
code in form of tangible nontransitory computer readable media for reporting the known identities as an output to an output device;
wherein said segmenting code includes code for employing foreground information of a previous image frame for current computation, and updating foreground information in preparation for a subsequent frame in order to promote fast multi-level segmentation and wherein said foreground information is updated according to the following functions:
μ
l
F
(
n
)
=
1
W
∑
k
μ
k
F
(
n
)
exp
(
-
1
2
[
μ
k
F
(
n
)
-
μ
l
F
(
1
)
]
T
∑
k
-
1
[
μ
k
F
(
n
)
-
μ
l
F
(
1
)
]
)
σ
l
F
(
n
)
=
1
W
∑
k
σ
k
F
(
n
)
exp
(
-
1
2
[
σ
k
F
(
n
)
-
σ
l
F
(
1
)
]
T
∑
k
-
1
[
σ
k
F
(
n
)
-
σ
l
F
(
1
)
]
)
With
:
W
=
∑
k
exp
(
-
1
2
[
σ
k
F
(
n
)
-
σ
l
F
(
1
)
]
T
∑
k
-
1
[
σ
k
F
(
n
)
-
σ
l
F
(
1
)
]
)
where W is the normalizing factor, μ i F (n) and σ i F (n) denote the mean and variance of i th components of the foreground for the frame n.
13. A system for recognizing humans through recognition of characteristics and identity of a selected human body part in order to recognize a human, the system comprising:
code in form of tangible nontransitory computer readable media for capturing as a dataset an image of at least a portion of the human body having a selected human body part, the human body being shown in a sequence of video frames, the portion of the human body being distinguished thereby from a background;
code in form of tangible nontransitory computer readable media for identifying a sub-dataset corresponding to the selected body part from the dataset representing the image by subjecting the dataset to at plurality of features-based rejection tests;
code in form of tangible nontransitory computer readable media for segmenting the sub-dataset for the selected body part according to identified sub-dataset segments corresponding to selected identities;
code in form of tangible nontransitory computer readable media for matching the sub-dataset segments to said selected identities; and
code in form of tangible nontransitory computer readable media for reporting the known identities as an output to an output device;
wherein said segmentation code includes code in form of tangible nontransitory computer readable media for automatically generating trimaps using energy minimization.