Analytic system for graphical interpretability of and improvement of machine learning models

1. A non-transitory computer-readable medium having stored thereon computer-readable instructions that when executed by a computing device cause the computing device to:

read a dataset that includes a plurality of observation vectors, wherein each observation vector of the plurality of observation vectors includes an explanatory variable value for an explanatory variable and a response variable value for a response variable;

train a model using the plurality of observation vectors;

predict a classification value for the response variable for each observation vector of the plurality of observation vectors using a respective explanatory variable value input to the trained model;

divide the plurality of observation vectors into a plurality of overlapping data slices;

separately cluster each data slice of the plurality of overlapping data slices using the predicted classification value to define a set of clusters for the plurality of observation vectors;

compute a number of observations in each cluster of the set of clusters;

compute an accuracy measure for each cluster of the set of clusters based on a comparison between the predicted classification value and a respective response variable value of each observation vector of the plurality of observation vectors;

compute a number of overlapping observations between each pair of clusters of the set of clusters; and

present a cluster connectivity graph on a display, wherein the cluster connectivity graph includes a node for each cluster of the set of clusters, wherein a size of each node is determined from the computed number of observations, wherein a fill-pattern of each node is determined from the computed accuracy measure, wherein a connector line between each pair of nodes is determined from the computed number of overlapping observations.

2. The non-transitory computer-readable medium of claim 1 , wherein the model is a machine learning model selected from the group consisting of a decision tree model, a factorization machine model, a forest model, a gradient boosting tree model, a neural network model, and a support vector machine model.

3. The non-transitory computer-readable medium of claim 1 , wherein the classification value is predicted based on a probability value output from the trained model.

4. The non-transitory computer-readable medium of claim 1 , wherein the classification value is predicted based on a probability value output from a layer of the trained model.

5. The non-transitory computer-readable medium of claim 1 , wherein dividing the plurality of observation vectors comprises:

sorting the plurality of observation vectors in order based on the response variable value; and

splitting the sorted plurality of observation vectors into a predefined number of data slices based on a predefined percentage overlap between data slices.

6. The non-transitory computer-readable medium of claim 5 , wherein the predefined percentage overlap is greater than zero.

7. The non-transitory computer-readable medium of claim 1 , wherein the accuracy measure is computed based on a predefined objective function that provides a measure of model error.

8. The non-transitory computer-readable medium of claim 1 , wherein the computed accuracy measure is a ratio relative to an overall accuracy measure.

9. The non-transitory computer-readable medium of claim 1 , wherein a width is defined for each connector line, wherein the width is greater than or equal to zero.

10. The non-transitory computer-readable medium of claim 1 , wherein the fill-pattern is a color selected based on the computed accuracy measure for a respective cluster of the node.

11. The non-transitory computer-readable medium of claim 1 , wherein the computer-readable instructions further cause the computing device to:

determine a poor performing cluster of the set of clusters based on the computed accuracy measure;

select a subset of observation vectors included in the determined poor performing cluster;

compare the explanatory variable value of the selected subset of observation vectors to the explanatory variable value of the plurality of observation vectors;

determine a model improvement based on the comparison, wherein the model improvement splits the plurality of observation vectors into a first set of observation vectors and a second set of observation vectors based on a split value of the explanatory variable;

train a first model using the first set of observation vectors;

train a second model using the second set of observation vectors; and

output the trained first model, the trained second model, and the split value of the explanatory variable.

12. The non-transitory computer-readable medium of claim 11 , wherein the poor performing cluster is determined automatically when the computed accuracy measure is less than a threshold accuracy measure.

13. The non-transitory computer-readable medium of claim 11 , wherein the poor performing cluster is determined as a result of an interaction by a user with the presented cluster connectivity graph.

14. The non-transitory computer-readable medium of claim 11 , wherein the split value is determined automatically.

15. The non-transitory computer-readable medium of claim 11 , wherein the split value is determined based on a presented comparison.

16. A computing device comprising:

a processor; and

a non-transitory computer-readable medium operably coupled to the processor, the computer-readable medium having computer-readable instructions stored thereon that, when executed by the processor, cause the computing device to

read a dataset that includes a plurality of observation vectors, wherein each observation vector of the plurality of observation vectors includes an explanatory variable value for an explanatory variable and a response variable value for a response variable;

train a model using the plurality of observation vectors;

predict a classification value for the response variable for each observation vector of the plurality of observation vectors using a respective explanatory variable value input to the trained model;

divide the plurality of observation vectors into a plurality of overlapping data slices;

separately cluster each data slice of the plurality of overlapping data slices using the predicted classification value to define a set of clusters for the plurality of observation vectors;

compute a number of observations in each cluster of the set of clusters;

compute an accuracy measure for each cluster of the set of clusters based on a comparison between the predicted classification value and a respective response variable value of each observation vector of the plurality of observation vectors;

compute a number of overlapping observations between each pair of clusters of the set of clusters; and

present a cluster connectivity graph on a display, wherein the cluster connectivity graph includes a node for each cluster of the set of clusters, wherein a size of each node is determined from the computed number of observations, wherein a fill-pattern of each node is determined from the computed accuracy measure, wherein a connector line between each pair of nodes is determined from the computed number of overlapping observations.

17. A method provides a visualization that summarizes a performance of a machine learning model to support model improvement and understanding, the method comprising:

reading, by a computing device, a dataset that includes a plurality of observation vectors, wherein each observation vector of the plurality of observation vectors includes an explanatory variable value for an explanatory variable and a response variable value for a response variable;

training, by the computing device, a model using the plurality of observation vectors;

predicting, by the computing device, a classification value for the response variable for each observation vector of the plurality of observation vectors using a respective explanatory variable value input to the trained model;

dividing, by the computing device, the plurality of observation vectors into a plurality of overlapping data slices;

separately clustering, by the computing device, each data slice of the plurality of overlapping data slices using the predicted classification value to define a set of clusters for the plurality of observation vectors;

computing, by the computing device, a number of observations in each cluster of the set of clusters;

computing, by the computing device, an accuracy measure for each cluster of the set of clusters based on a comparison between the predicted classification value and a respective response variable value of each observation vector of the plurality of observation vectors;

computing, by the computing device, a number of overlapping observations between each pair of clusters of the set of clusters; and

presenting, by the computing device, a cluster connectivity graph on a display, wherein the cluster connectivity graph includes a node for each cluster of the set of clusters, wherein a size of each node is determined from the computed number of observations, wherein a fill-pattern of each node is determined from the computed accuracy measure, wherein a connector line between each pair of nodes is determined from the computed number of overlapping observations.

18. The method of claim 17 , wherein the model is a machine learning model selected from the group consisting of a decision tree model, a factorization machine model, a forest model, a gradient boosting tree model, a neural network model, and a support vector machine model.

19. The method of claim 17 , wherein the classification value is predicted based on a probability value output from the trained model.

20. The method of claim 17 , wherein the classification value is predicted based on a probability value output from a layer of the trained model.

21. The method of claim 17 , wherein dividing the plurality of observation vectors comprises:

sorting, by the computing device, the plurality of observation vectors in order based on the response variable value; and

splitting, by the computing device, the sorted plurality of observation vectors into a predefined number of data slices based on a predefined percentage overlap between data slices.

22. The method of claim 21 , wherein the predefined percentage overlap is greater than zero.

23. The method of claim 17 , wherein the accuracy measure is computed based on a predefined objective function that provides a measure of model error.

24. The method of claim 17 , wherein the computed accuracy measure is a ratio relative to an overall accuracy measure.

25. The method of claim 17 , wherein a width is defined for each connector line, wherein the width is greater than or equal to zero.

26. The method of claim 17 , wherein the fill-pattern is a color selected based on the computed accuracy measure for a respective cluster of the node.

27. The method of claim 17 , further comprising:

determining, by the computing device, a poor performing cluster of the set of clusters based on the computed accuracy measure;

selecting, by the computing device, a subset of observation vectors included in the determined poor performing cluster;

comparing, by the computing device, the explanatory variable value of the selected subset of observation vectors to the explanatory variable value of the plurality of observation vectors;

determining, by the computing device, a model improvement based on the comparison, wherein the model improvement splits the plurality of observation vectors into a first set of observation vectors and a second set of observation vectors based on a split value of the explanatory variable;

training, by the computing device, a first model using the first set of observation vectors;

training, by the computing device, a second model using the second set of observation vectors; and

outputting, by the computing device, the trained first model, the trained second model, and the split value of the explanatory variable.

28. The method of claim 27 , wherein the poor performing cluster is determined as a result of an interaction by a user with the presented cluster connectivity graph.

29. The method of claim 27 , wherein the poor performing cluster is determined automatically when the computed accuracy measure is less than a threshold accuracy measure.

30. The method of claim 27 , wherein the split value is determined automatically.