Confidence weighted classifier combination for multi-modal identification
Techniques are disclosed for multi-modal identification that utilize a classifier combination framework. One embodiment of the present invention provides a multi-modal identification system that includes a collection of classifiers that classify feature streams derived from audio and/or video sources. A classifier combination scheme is used to combine the classifier outputs having varying degrees of confidence, but in a robust way by using a confidence-based weighting scheme that operates on a “per-class” basis, rather than (or in addition to) the traditional “per-classifier” basis. The system can be distributed across several machines running independent feature classifiers on the subscription basis.
1. A multi-class classifier combination system comprising:
at least one processor;
a plurality of multi-class classifiers, each classifier executing on one of the at least one processor, each classifier for classifying a target feature captured in at least one feature stream; and
a combination module for combining classifier outputs into a joint prediction, wherein each multi-class classifier prediction is weighted in accordance with a per-class weighting scheme prior to combining, wherein the per-class weighting scheme carried out by the combination module is in accordance with
P
s
≈
∑
λ
w
λ
[
∑
ω
~
P
λ
(
ω
❘
ω
~
)
P
λ
(
ω
~
❘
x
)
]
,
where P S is the joint prediction, x is a set of features in a given scene, P λ ({tilde over (ω)}|x) is the prediction of the individual classifier, w λ is a per-classifier weight, and confidence measure, P λ (ω|{tilde over (ω)},x), is approximated by its projection, P λ (ω|{tilde over (ω)}), wherein ω is a true class label and {tilde over (ω)} is an output of an individual multi-class classifier.
2. The system of claim 1 further comprising:
a labeling subsystem for labeling stored feature streams accessible to the system, in accordance with a user selected labeling scheme.
3. The system of claim 1 further comprising:
a storage for storing feature streams; and
a database manager for generating and storing for each feature stream a record including at least one of a time stamp, a file name, recording conditions, and current system parameters in the storage.
4. The system of claim 1 further comprising:
a data logging subsystem for generating feature streams including at least one audio stream and one video stream from a target scene.
5. The system of claim 4 wherein the data logging subsystem includes a detector that triggers generation of feature streams in response to detecting a target entity being present in the target scene.
6. The system of claim 1 wherein a classifier is trained on a subset of training data, and then a confidence measure is computed based on the remaining subset of the training data, and the per-class weighting scheme carried out by the combination module includes weighting the classifier output by the resulting confidence measure.
7. The system of claim 1 wherein the per-class weighting scheme carried out by the combination module is based on using a confidence measure to weigh each classifier output, the confidence measure derived from a confusion matrix that represents an empirical value of the distribution of intrinsic error of the classifier on a given data set.
8. A method for multi-class classifier combination using predictions of a plurality of multi-class classifiers, the method comprising using a processor to perform the steps of:
weighting each multi-class classifier prediction in accordance with a per-class weighting scheme; and
combining the weighted predictions from two or more multi-class classifiers into a joint prediction, wherein the per-class weighting scheme is in accordance with
P
s
≈
∑
λ
w
λ
[
∑
ω
~
P
λ
(
ω
❘
ω
~
)
P
λ
(
ω
~
❘
x
)
]
,
where P S is the joint prediction, x is a set of features in a given scene, P λ ({tilde over (ω)}|x) is the prediction of the individual classifier, w λ is a per-classifier weight, and confidence measure, P λ (ω|{tilde over (ω)},x), is approximated by its projection, P λ (ω|{tilde over (ω)}), wherein ω is a true class label and {tilde over (ω)} is an output of an individual multi-class classifier.
9. The method of claim 8 further comprising:
storing feature streams; and
generating and storing a record for each feature stream including at least one of a time stamp, a file name, recording conditions, and current system parameters in the storage.
10. The method of claim 8 further comprising the preliminary steps of:
generating feature streams including at least one audio stream and one video stream from a target scene;
classifying a first target feature captured in at least one feature stream using a first multi-class classifier; and
classifying a second target feature captured in at least one feature stream using a second multi-class classifier.
11. The method of claim 10 wherein generating feature streams is triggered in response to detecting a target entity being present in the target scene.
12. The method of claim 8 wherein the per-class weighting scheme is based on using a confidence measure to weigh each classifier output, the confidence measure derived from a confusion matrix that represents an empirical value of the distribution of intrinsic error of the classifier on a given data set.
13. The method of claim 8 further comprising the preliminary steps of:
training at least one of the multi-class classifiers on a subset of training data; and
computing a confidence measure based on the remaining subset of the training data;
wherein the per-class weighting scheme includes weighting the at least one classifier output by the resulting confidence measure.
14. A multi-class classifier combination system comprising:
a means for classifying a first target feature captured in at least one feature stream using a first multi-class classifier;
a means for classifying a second target feature captured in at least one feature stream using a second multi-class classifier;
a means for weighting each multi-class classifier prediction in accordance with a per-class weighting scheme;
a means for combining the weighted predictions from the first and second multi-class classifiers into a joint prediction, wherein the per-class weighting scheme is in accordance with
P
s
≈
∑
λ
w
λ
[
∑
ω
~
P
λ
(
ω
❘
ω
~
)
P
λ
(
ω
~
❘
x
)
]
,
where P S is the joint prediction, x is a set of features in a given scene, P λ ({tilde over (ω)}|x) is the prediction of the individual classifier, w 80 is a per-classifier weight, and confidence measure, P λ (ω|{tilde over (ω)},x), is approximated by its projection, P λ (ω|{tilde over (ω)}), wherein ω is a true class label and {tilde over (ω)} is an output of an individual multi-class classifier; and
a processor for executing the means for combining the weighted predictions from the first and second multi-class classifiers into a joint prediction.
15. A computer-readable medium encoded with instructions, that when executed by a processor, cause the processor to carry out a multi-class classifier combination process using predictions of a plurality of multi-class classifiers, the process comprising:
weighting each multi-class classifier prediction in accordance with a per-class weighting scheme; and
combining the weighted predictions from two or more multi-class classifiers into a joint prediction, wherein the per-class weighting scheme is in accordance with
P
s
≈
∑
λ
w
λ
[
∑
ω
~
P
λ
(
ω
❘
ω
~
)
P
λ
(
ω
~
❘
x
)
]
,
where P S is the joint prediction, x is a set of features in a given scene, P λ ({tilde over (ω)}|x) is the prediction of the individual classifier, w 80 is a per-classifier weight, and confidence measure, P λ (ω|{tilde over (ω)},x), is approximated by its projection, P λ (ω|{tilde over (ω)}), wherein ω is a true class label and {tilde over (ω)} is an output of an individual multi-class classifier.
16. A multi-class classifier combination system comprising:
at least one processor;
a plurality of multi-class classifiers, each classifier executing on one of the at least one processor, each classifier for classifying a target feature captured in at least one feature stream; and
a combination module for combining classifier outputs into a joint prediction, wherein each multi-class classifier prediction is weighted in accordance with a per-class weighting scheme prior to combining, wherein the per-class weighting scheme carried out by the combination module is in accordance with
P
P
(
ω
❘
x
)
=
1
Z
∏
λ
[
∑
ω
~
P
λ
(
ω
❘
ω
~
)
P
λ
(
ω
~
❘
x
)
]
,
where P P is the joint prediction using a product combination rule, x is a set of features in a given scene, P λ ({tilde over (ω)}|x) is the prediction of the individual classifier, Z is a normalizing constant, and confidence measure, P λ (ω|{tilde over (ω)},x), is approximated by its projection, P λ (ω|{tilde over (ω)}), wherein ω is a true class label and {tilde over (ω)} is an output of an individual multi-class classifier.
17. A method for multi-class classifier combination using predictions of a plurality of multi-class classifiers, the method comprising using a processor to perform the steps of:
weighting each multi-class classifier prediction in accordance with a per-class weighting scheme; and
combining the weighted predictions from two or more multi-class classifiers into a joint prediction, wherein the per-class weighting scheme is in accordance with
P
P
(
ω
❘
x
)
=
1
Z
∏
λ
[
∑
ω
~
P
λ
(
ω
❘
ω
~
)
P
λ
(
ω
~
❘
x
)
]
,
where P P is the joint prediction using a product combination rule, x is a set of features in a given scene, P λ ({tilde over (ω)}|x) is the prediction of the individual classifier, Z is a normalizing constant, and confidence measure, P λ (ω|{tilde over (ω)},x), is approximated by its projection, P λ (ω|{tilde over (ω)}), wherein ω is a true class label and {tilde over (ω)} is an output of an individual multi-class classifier.
18. A computer-readable medium encoded with instructions, that when executed by a processor, cause the processor to carry out a multi-class classifier combination process using predictions of a plurality of multi-class classifiers, the process comprising:
weighting each multi-class classifier prediction in accordance with a per-class weighting scheme; and
combining the weighted predictions from two or more multi-class classifiers into a joint prediction wherein the per-class weighting scheme is in accordance with
P
P
(
ω
❘
x
)
=
1
Z
∏
λ
[
∑
ω
~
P
λ
(
ω
❘
ω
~
)
P
λ
(
ω
~
❘
x
)
]
,
where P P is the joint prediction using a product combination rule, x is a set of features in a given scene, P λ ({tilde over (ω)}|x) is the prediction of the individual classifier, Z is a normalizing constant, and confidence measure, P λ (ω|{tilde over (ω)},x), is approximated by its projection, P λ (ω|{tilde over (ω)}), wherein ω is a true class label and {tilde over (ω)} is an output of an individual multi-class classifier.