Machine learning uncertainty quantification and modification
Computer-implemented machines, systems and methods for providing insights about uncertainty of a machine learning model. A method includes determining an uncertainty value associated with a first machine learning model output of a first machine learning model. The method further includes generating a confidence interval for the first machine learning model output associated with an input. The method further includes switching, responsive to the uncertainty value satisfying a threshold, from the first machine learning model to a second machine learning model, the second machine learning model generating a second machine learning model output. The method further includes generating the second machine learning model. The method further includes providing, responsive to the switching, the machine learning output, the uncertainty value, the confidence interval, and the second machine learning output to a user interface.
1. A computer-implemented method, the method comprising:
determining, by one or more programmable processors, an uncertainty value associated with a first machine learning model output of a first machine learning model;
generating, by the one or more programmable processors, a confidence interval for the first machine learning model output associated with an input;
switching, by the one or more programmable processors and responsive to the uncertainty value satisfying a threshold, from the first machine learning model to a second machine learning model, the second machine learning model generating a second machine learning model output;
generating, by the one or more programmable processors, the second machine learning model based on the first machine learning model; and
providing, by the one or more programmable processors and responsive to the switching, the first machine learning output, the uncertainty value, the confidence interval, and the second machine learning output to a user interface,
wherein generating the second machine learning model based on the first machine learning model comprises:
constructing hidden layers of the second machine learning model where hidden nodes of the hidden layers are a sparse sub-network of hidden nodes approximating the first machine learning model;
generating perturbed variations of sparse networks of high variance hidden nodes;
removing or prohibiting feature interactions contributing the high variance hidden nodes; and
iterating and training the second machine learning model based on removed and prohibited feature interactions to minimize model variance of the second machine learning model.
2. The method of claim 1 , wherein the uncertainty value is based on an estimate of model predictive variance for the first machine learning model based on an ensemble of architecturally same machine learning models based on a sampling of models based on different training parameters.
3. The method of claim 2 , wherein the model predictive variance for the first machine learning model is based on variance of a finite sum of possible choices of the first machine learning model from a posterior distribution.
4. The method of claim 1 , wherein the confidence interval is based on a parametric statistical method or a non-parametric statistical method.
5. The method of claim 1 , wherein the confidence interval is represented as [max( x −ƒ(c)s,0), min( x +ƒ(c)s,1)], where c is a desired confidence level, x represents sample scores sample mean, s represents sample standard deviation, and ƒ(c) represents an appropriate parametric multiplier.
6. The method of claim 1 , wherein the second machine learning model comprises a stepdown model.
7. The method of claim 6 , wherein the stepdown model has a lower predictive variance than the first machine learning model.
8. The method of claim 2 , wherein the predictive variance is defined as a possible variation in scores for a given input x over possible choices of the first machine learning model.
9. The method of claim 1 , wherein providing the machine learning output, the uncertainty value, the confidence interval, and the second machine learning output comprises transmitting the machine learning output, the uncertainty value, the confidence interval, and the second machine learning output to a display of the user interface.
10. A system comprising:
at least one programmable processor; and
a non-transitory machine-readable medium storing instructions that, when executed by the at least one programmable processor, cause the at least one programmable processor to perform operations comprising:
determining an uncertainty value associated with a first machine learning model output of a first machine learning model;
generating a confidence interval for the first machine learning model output associated with an input;
switching, responsive to the uncertainty value satisfying a threshold, from the first machine learning model to a second machine learning model, the second machine learning model generating a second machine learning model output;
and
providing, responsive to the switching, the first machine learning output, the uncertainty value, the confidence interval, and the second machine learning output to a user interface,
generating the second machine learning model based on the first machine learning model by:
constructing hidden layers of the second machine learning model where hidden nodes of the hidden layers are a sparse sub-network of hidden nodes approximating the first machine learning model;
generating perturbed variations of sparse networks of high variance hidden nodes;
removing or prohibiting feature interactions contributing the high variance hidden nodes; and
iterating and training the second machine learning model based on removed or prohibited feature interactions to minimize model variance of the second machine learning model.
11. The system of claim 10 , wherein the uncertainty value is based on an estimate of model predictive variance for the first machine learning model.
12. The system of claim 10 , wherein the confidence interval is based on a parametric statistical method or a non-parametric statistical method.
13. The system of claim 10 , wherein the confidence interval is represented as
[
max
(
x
¯
-
f
(
c
)
s
,
0
)
,
min
(
x
_
+
f
(
c
)
s
,
1
)
]
,
where c is a desired confidence level, x represents sample scores sample mean, s represents sample standard deviation, and ƒ(c) represents an appropriate parametric multiplier.
14. The system of claim 10 , wherein the second machine learning model comprises a stepdown model.
15. The system of claim 14 , wherein the stepdown model has a lower predictive variance than the first machine learning model.
16. The system of claim 10 , wherein the predictive variance is defined as a possible variation in scores for a given input x over possible choices of the first machine learning model.