Training ensemble models to improve performance in the presence of unreliable base classifiers
A system and method for training base classifiers in a boosting algorithm includes optimally training base classifiers considering an unreliability model, and then using a scheme with an aggregator decoder that reverse-flips inputs using inter-classifier redundancy introduced in training.
1. A method of training a machine learning model, comprising:
determining a probability of an unreliable base classifier output for a plurality of base classifiers of an ensemble model, wherein the probability of an unreliable base classifier output is greater than a threshold value;
training the plurality of base classifiers in a sequence of training iterations to minimize an expected loss function, wherein:
each training iteration of the sequence of training iterations is associated with one base classifier of the plurality of base classifiers, and
the expected loss function accounts for the probability of an unreliable base classifier output;
determining a plurality of base classifier weights for the plurality of base classifiers, wherein each base classifier weight of the plurality of base classifier weights is associated with one base classifier of the plurality of base classifiers; and
updating the ensemble model with the plurality of base classifier weights, wherein the expected loss function comprises an expected exponential loss function, wherein:
the probability of an unreliable base classifier output η comprises the probability of a straggling base classifier,
the expected exponential loss function for each training iteration t is Ē=Σ i=1 n w t (t) ·[(1−η)esp{−y 1 α t h t (x i )}+η],
w i (t) is a data point weight for data point {x i , y i }, and
α t is a base classifier weight of the plurality of base classifier weights associated with a base classifier h t of the plurality of base classifiers.
2. The method of claim 1 , further comprising:
determining an updated base classifier weight for a subset of base classifiers of the plurality of base classifiers,
wherein each base classifier of the subset of base classifiers comprises a non-straggling base classifier.
3. The method of claim 1 , further comprising: performing a classification with an aggregator of the ensemble model based on a non-straggling subset of base classifiers of the plurality of base classifiers.
4. The method of claim 3 , wherein:
the classification is performed according to:
y
^
=
sign
(
∑
t
=
1
t
∈
A
T
α
t
h
t
(
x
)
)
,
A is the non-straggling subset of base classifiers in the plurality of base classifiers T, and
α t is a base classifier weight of the plurality of base classifier weights associated with a base classifier h t of the plurality of base classifiers.
5. A method of training a machine learning model, comprising:
determining a probability of an unreliable base classifier output for a plurality of base classifiers of an ensemble model, wherein the probability of an unreliable base classifier output is greater than a threshold value;
training the plurality of base classifiers in a sequence of training iterations to minimize an expected loss function, wherein:
each training iteration of the sequence of training iterations is associated with one base classifier of the plurality of base classifiers, and
the expected loss function accounts for the probability of an unreliable base classifier output;
determining a plurality of base classifier weights for the plurality of base classifiers, wherein each base classifier weight of the plurality of base classifier weights is associated with one base classifier of the plurality of base classifiers; and
updating the ensemble model with the plurality of base classifier weights, wherein the expected loss function comprises an expected exponential loss function, wherein:
the probability of an unreliable base classifier output ε comprises the probability of a flipping base classifier,
the expected exponential loss function for each training iteration t is Ē=Σ i=1 m Π j=1 t [(1−ε)exp{−y i α j h j (x i )}+εexp{y t α j h j (x i )}],
w l (t) is a data point weight for data point x i , and
α t is a base classifier weight of the plurality of base classifier weights associated with a base classifier h t of the plurality of base classifiers.
6. A method of training a machine learning model, comprising:
determining a probability of an unreliable base classifier output for a plurality of base classifiers of an ensemble model, wherein the probability of an unreliable base classifier output is greater than a threshold value;
training the plurality of base classifiers in a sequence of training iterations to minimize an expected loss function, wherein:
each training iteration of the sequence of training iterations is associated with one base classifier of the plurality of base classifiers, and
the expected loss function accounts for the probability of an unreliable base classifier output;
determining a plurality of base classifier weights for the plurality of base classifiers, wherein each base classifier weight of the plurality of base classifier weights is associated with one base classifier of the plurality of base classifiers; and
updating the ensemble model with the plurality of base classifier weights, wherein determining the plurality of base classifier weights for the plurality of base classifiers, comprises: for each training iteration t, setting a base classifier weight α t for a base classifier h t of the plurality of base classifiers according to:
α
t
=
1
2
ln
(
∑
i
=
1
m
w
i
(
t
)
I
(
h
t
(
x
i
)
=
y
i
∑
i
=
1
m
w
i
(
t
)
I
(
h
t
(
x
i
)
≠
y
i
)
.
7. A method of training a machine learning model, comprising:
determining a probability of an unreliable base classifier output for a plurality of base classifiers of an ensemble model, wherein the probability of an unreliable base classifier output is greater than a threshold value;
training the plurality of base classifiers in a sequence of training iterations to minimize an expected loss function, wherein:
each training iteration of the sequence of training iterations is associated with one base classifier of the plurality of base classifiers, and
the expected loss function accounts for the probability of an unreliable base classifier output;
determining a plurality of base classifier weights for the plurality of base classifiers, wherein each base classifier weight of the plurality of base classifier weights is associated with one base classifier of the plurality of base classifiers; and
updating the ensemble model with the plurality of base classifier weights, wherein determining the plurality of base classifier weights for the plurality of base classifiers, comprises: for each training iteration t, setting a base classifier weight α t for a base classifier h t of the plurality of base classifiers according to:
α
t
=
1
2
ln
(
∑
i
=
1
m
w
i
(
t
)
[
(
1
-
ε
)
I
(
h
t
(
x
i
)
=
y
i
)
+
ε
I
(
h
t
(
x
i
)
≠
y
i
)
]
∑
i
=
1
m
w
i
(
t
)
[
(
1
-
ε
)
I
(
h
t
(
x
i
)
≠
y
i
)
+
ε
I
(
h
t
(
x
i
)
=
y
i
)
]
)
.
8. The method of claim 7 , further comprising:
performing a classification with an aggregator of the ensemble model according to:
y
^
=
sign
(
∑
t
=
1
t
∈
A
T
α
t
h
t
(
x
)
-
∑
t
=
1
t
∉
A
T
α
t
h
t
(
x
)
)
,
wherein A is a subset of non-flipping base classifiers in the plurality of base classifiers T.
9. A processing system configured to train a machine learning model, comprising:
a memory comprising computer-executable instructions;
one or more processors configured to execute the computer-executable instructions and cause the processing system to:
determine a probability of an unreliable base classifier output for a plurality of base classifiers of an ensemble model, wherein the probability of an unreliable base classifier output is greater than a threshold value;
train the plurality of base classifiers in a sequence of training iterations to minimize an expected loss function, wherein:
each training iteration of the sequence of training iterations is associated with one base classifier of the plurality of base classifiers, and
the expected loss function accounts for the probability of an unreliable base classifier output;
determine a plurality of base classifier weights for the plurality of base classifiers, wherein each base classifier weight of the plurality of base classifier weights is associated with one base classifier of the plurality of base classifiers; and
update the ensemble model with the plurality of base classifier weights, wherein determining the plurality of base classifier weights for the plurality of base classifiers, comprises: for each training iteration t, setting a base classifier weight α t for a base classifier h t of the plurality of base classifiers according to:
α
t
=
1
2
ln
(
∑
i
=
1
m
w
i
(
t
)
I
(
h
t
(
x
i
)
=
y
i
∑
i
=
1
m
w
i
(
t
)
I
(
h
t
(
x
i
)
≠
y
i
)
.
10. The processing system of claim 9 , wherein the expected loss function comprises an expected exponential loss function.
11. The processing system of claim 9 , wherein the one or more processors are further configured to cause the processing system to:
determine an updated base classifier weight for a subset of base classifiers of the plurality of base classifiers,
wherein each base classifier of the subset of base classifiers comprises a non-straggling base classifier.
12. The processing system of claim 11 , wherein the one or more processors are further configured to cause the processing system to: train the plurality of base classifiers of an ensemble model in a sequence of training iterations to minimize the expected loss function.
13. The processing system of claim 12 , wherein each training iteration of the sequence of training iterations is associated with one base classifier of the plurality of base classifiers.
14. The processing system of claim 13 , wherein each base classifier weight of the plurality of base classifier weights is associated with one base classifier of the plurality of base classifiers.
15. The processing system of claim 11 , wherein the one or more processors are further configured to cause the processing system to: train a plurality of redundant base classifiers of the ensemble model based on a plurality of trained base classifiers.
16. The processing system of claim 15 , wherein the one or more processors are further configured to cause the processing system to: determine a plurality of redundant base classifier weights for the plurality of redundant base classifiers.
17. The processing system of claim 16 , wherein each redundant base classifier weight of the plurality of redundant base classifier weights is associated with one redundant base classifier of the plurality of redundant base classifiers.
18. The processing system of claim 17 , wherein the one or more processors are further configured to cause the processing system to: update the ensemble model with the plurality of base classifier weights and the plurality of redundant base classifier weights.
19. The processing system of claim 18 , each redundant base classifier of the plurality of redundant base classifiers is trained based on a unique set of two base classifiers of the plurality of base classifiers to form a triple of classifiers.
20. The processing system of claim 9 , wherein the one or more processors are further configured to cause the processing system to: perform a classification with an aggregator of the ensemble model based on a non-straggling subset of base classifiers of the plurality of base classifiers.
21. The processing system of claim 9 , wherein the one or more processors are further configured to cause the processing system to: determine the probability of a straggling base classifier.
22. The processing system of claim 9 , wherein the one or more processors are further configured to cause the processing system to: determine the probability of a flipping base classifier.
23. The processing system of claim 9 , wherein the one or more processors are further configured to cause the processing system to: synthesize a model for logical programmable gates to implement a tensor array.
24. The processing system of claim 9 , wherein the one or more processors are further configured synthesize a model for logical programmable gates to implement a tensor array.