Sufficiency assessment of machine learning models through maximum deviation
Techniques regarding determining sufficiency of one or more machine learning models are provided. For example, one or more embodiments described herein can comprise a system, which can comprise a memory that can store computer executable components. The system can also comprise a processor, operably coupled to the memory, and that can execute the computer executable components stored in memory. The computer executable components can comprise a measurement component that measures maximum deviation of a supervised learning model from a reference model over a certification set and an analysis component that determines sufficiency of the supervised learning model based at least in part on the maximum deviation.
1 . A system, comprising:
a memory that stores computer executable components including a certification set of machine-readable inputs defining a bounded region of an input feature space over which model behavior is evaluated; and
a processor, operably coupled to the memory, and that executes the computer executable components stored in the memory, wherein the computer executable components comprise:
a measurement component that computationally evaluates, using the stored certification set, a maximum deviation of a supervised learning model from a reference model by exhaustively evaluating and provably bounding deviations over the certification set rather than by sampling or empirical validation;
wherein the supervised learning model comprises a tree ensemble, and the measurement component employs one or more discrete optimization techniques to exploit composition in terms of trees to provide anytime bounds on the maximum deviation;
wherein the measurement component constructs a representation of leaf regions of the tree ensemble, identifies candidate combinations of leaves drawn from respective trees whose feature-space regions are mutually compatible, computes a deviation associated with each candidate combination, and uses bounding techniques to prune candidate combinations to determine the maximum deviation; and
an analysis component that determines sufficiency of the supervised learning model based at least in part on the maximum deviation satisfying a predefined certification criterion, and that outputs a machine-readable sufficiency notification that automatically governs retraining of the supervised learning model within a computing environment, and
wherein the system thereby provides a pre-deployment computational certification of model reliability that is independent of training accuracy or test-set performance;
wherein the system automatically controls deployment of the model into a computing environment as a function of the certification.
2 . The system of claim 1 , wherein the certification set is a subset of an input space of the supervised learning model.
3 . The system of claim 1 , wherein for a bipartite graph, with z nodes representing leaves l of/on one side and L G nodes representing leaves 0m of ƒ 0 on the other, the measurement component defines an edge set ε={(l, m): l ∩ 0m ∩ ≠Ø}; where |ε|≤L 0 L, where ƒ and ƒ 0 are decision trees with L and L 0 leaves respectively, and ε is a bipartite edge set of leaf intersections, the maximum deviation can be computed with |ε| evaluations as
max
x
∈
𝒞
D
(
f
(
x
)
,
f
0
(
x
)
)
=
max
(
l
,
m
)
∈
ℰ
D
(
y
l
,
y
0
m
)
.
4 . The system of claim 1 , wherein the analysis component identifies inputs that lead to a deviation greater than a threshold value.
5 . The system of claim 1 , wherein the analysis component for the supervised learning model ƒ, which is a function mapping an input feature space to an output space , determines sufficiency of the supervised learning model by determining its worst-case deviation from the reference model ƒ 0 : wherein for a measure of deviation D: × ← , where + is the set of non-negative reals, and the certification set ⊆ over which the deviation is maximized, the analysis solves:
max
x
∈
𝒞
D
(
f
(
x
)
,
f
0
(
x
)
)
.
6 . The system of claim 1 , wherein the analysis component determines if deviation of the supervised learning model from the reference model over the certification set exceeds a threshold value, and identifies a set of inputs with deviation greater than the threshold value.
7 . A computer-implemented method, comprising:
measuring, by a processor, a maximum deviation metric of a supervised learning model from a reference model by exhaustively evaluating and provably bounding deviations over a certification set of machine-readable inputs stored in memory, the certification set defining a bounded subset of an input feature space over which model behavior is evaluated;
wherein the supervised learning model comprises a tree ensemble, and the measurement component employs one or more discrete optimization techniques to exploit composition in terms of trees to provide anytime bounds on the maximum deviation;
wherein the measurement component constructs a representation of leaf regions of the tree ensemble, identifies candidate combinations of leaves drawn from respective trees whose feature-space regions are mutually compatible, computes a deviation associated with each candidate combination, and uses bounding techniques to prune candidate combinations to determine the maximum deviation; and
using the processor to determine sufficiency of the supervised learning model based at least in part on whether the maximum deviation metric satisfies a predefined certification criterion, and generating, by the processor, a machine-readable sufficiency notification that is consumed by an automated control process to selectively initiate retraining of the supervised learning model within a computing environment;
wherein the method provides a pre-deployment computational certification of model reliability independently of training loss, validation accuracy, or empirical test-set performance of the supervised learning model;
wherein the system automatically controls deployment of the model into a computing environment as a function of the certification.
8 . The computer-implemented method of claim 7 , wherein the certification set is a subset of an input space of the supervised learning model.
9 . The computer-implemented method of claim 7 , wherein for a bipartite graph, with L nodes representing leaves l of ƒ on one side and L 0 nodes representing leaves 0m of ƒ 0 on the other, the processor defines an edge set ε={(l,m): l ∩ 0m ∩ ≠Ø}; where |ε|≤L 0 L, where ƒ and ƒ 0 are decision trees with L and L 0 leaves respectively, and ε is a bipartite edge set of leaf intersections, computes the maximum deviation with |ε| evaluations as
max
x
∈
𝒞
D
(
f
(
x
)
,
f
0
(
x
)
)
=
max
(
l
,
m
)
∈
ℰ
D
(
y
l
,
y
0
m
)
.
10 . The computer-implemented method of claim 7 , further comprising:
identifying inputs that lead to deviation greater than a threshold.
11 . The computer-implemented method of claim 7 , wherein the processor for the supervised learning model ƒ, which is a function mapping an input feature space to an output space , determines sufficiency of the supervised learning model by determining its worst-case deviation from the reference model ƒ 0 : wherein for a measure of deviation D: × + where , is the set of non-negative reals, and the certification set ⊆ over which the deviation is maximized, the analysis solves:
max
x
∈
𝒞
D
(
f
(
x
)
,
f
0
(
x
)
)
.
12 . The computer-implemented method of claim 7 , wherein the processor determines if deviation of the supervised learning model from the reference model over the certification set exceeds a threshold value, and identifies a set of inputs with deviation greater than the threshold value.
13 . A non-transitory computer program product for determining sufficiency of a supervised learning model, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to:
measure a maximum deviation metric of the supervised learning model from a reference model by exhaustively evaluating and provably bounding deviations over a certification set of machine-readable inputs stored in memory, the certification set corresponding to a bounded subset of an input feature space over which model behavior is evaluated;
wherein the supervised learning model comprises a tree ensemble, and the measurement component employs one or more discrete optimization techniques to exploit composition in terms of trees to provide anytime bounds on the maximum deviation;
wherein the measurement component constructs a representation of leaf regions of the tree ensemble, identifies candidate combinations of leaves drawn from respective trees whose feature-space regions are mutually compatible, computes a deviation associated with each candidate combination, and uses bounding techniques to prune candidate combinations to determine the maximum deviation; and
determine sufficiency of the supervised learning model based at least in part on whether the maximum deviation metric satisfies a predefined certification criterion, and generate a machine-readable sufficiency notification configured to be consumed by an automated model-governance process to selectively trigger retraining of the supervised learning model within a computing environment;
wherein the program instructions implement a pre-deployment computational certification of model reliability independently of training loss, validation accuracy, or empirical test-set performance of the supervised learning model;
wherein the system automatically controls deployment of the model into a computing environment as a function of the certification.
14 . The computer program product of claim 13 , wherein the certification set is a subset of an input space of the supervised learning model.
15 . The computer program product of claim 13 , wherein the program instructions further cause the processor to for a bipartite graph, with L nodes representing leaves L l of ƒ on one side and L 0 nodes representing leaves 0m of ƒ 0 on the other, define an edge set ε={(l,m): ∩ 0m ∩ ≠Ø}; where [ε]≤L 0 L, where ƒ and ƒ 0 are decision trees with L and L 0 leaves respectively, and ε is a bipartite edge set of leaf intersections, and compute the maximum deviation with [ε] evaluations as
max
x
∈
𝒞
D
(
f
(
x
)
,
f
0
(
x
)
)
=
max
(
l
,
m
)
∈
ℰ
D
(
y
l
,
y
0
m
)
.
16 . The computer program product of claim 13 , wherein the program instructions further cause the processor to identify inputs that lead to a deviation greater than a threshold value.
17 . The computer program product of claim 13 , wherein the program instructions further cause the processor to for the supervised learning model ƒ, which is a function mapping an input feature space to an output space , determine sufficiency of the supervised learning model by determining its worst-case deviation from the reference model ƒ 0 : wherein for a measure of deviation D: × + where + is the set of non-negative reals, and the certification set ⊆ over which the deviation is maximized, the analysis solves:
max
x
∈
𝒞
D
(
f
(
x
)
,
f
0
(
x
)
)
.
18 . The computer program product of claim 13 , wherein the program instructions further cause the processor to determine if deviation of the supervised learning model from the reference model over the certification set exceeds a threshold value, and identifies a set of inputs with deviation greater than the threshold value.