Measuring performance in cognitive networks
In one embodiment, a device uses a quality of experience model to make predictions comprising probabilities of a quality of experience metric for an online application accessed via a network being acceptable. The device determines, for each of a subset of the predictions having lowest probabilities of acceptable user experience from amongst the predictions, whether that prediction corresponds to a negative experience rating by a user of the online application. The device computes a negative retrieval accuracy for the quality of experience model based on a count of negative experience ratings associated with the subset and a size of the subset. The device uses the negative retrieval accuracy to detect an issue in the network.
1. A method comprising:
using, by a device, a quality of experience model to make predictions comprising probabilities of a quality of experience metric for an online application accessed via a network being acceptable;
determining, by the device and for each of a subset of the predictions having lowest probabilities of acceptable user experience from amongst the predictions, whether that prediction corresponds to a negative experience rating by a user of the online application;
computing, by the device, a negative retrieval accuracy for the quality of experience model based on a count of negative experience ratings associated with the subset and a size of the subset; and
using, by the device, the negative retrieval accuracy to detect an issue in the network.
2. The method as in claim 1 , wherein the quality of experience model makes the predictions based on telemetry data captured by the network.
3. The method as in claim 1 , further comprising:
providing, by the device, the negative retrieval accuracy for presentation by a user interface.
4. The method as in claim 1 , wherein a size of the subset of the predictions differs from a total amount of the predictions whose probabilities indicate negative user experiences.
5. The method as in claim 1 , further comprising:
computing a robust ranking accuracy metric in part by randomly selecting pairs of samples associated with different subsets of the predictions, wherein the pairs of samples have different percentages of positive user experience labels.
6. The method as in claim 5 , further comprising:
using, by the device, the robust ranking accuracy metric to override a routing decision in the network.
7. The method as in claim 5 , further comprising:
providing the robust ranking accuracy metric to a path computation element or via a Border Gateway Protocol message in the network.
8. The method as in claim 5 , wherein the device computes the robust ranking accuracy metric in part by performing two-sample Mann-Whitney U testing on the pairs.
9. The method as in claim 1 , further comprising:
varying a size of the subset to compute different precision and recall metrics for the quality of experience model;
computing gain values for the different precision and recall metrics; and
computing a precision recall gain metric as an average of the gain values for the different precision and recall metrics.
10. The method as in claim 9 , wherein the precision recall gain metric is used to detect the issue in the network.
11. An apparatus, comprising:
one or more network interfaces;
a processor coupled to the one or more network interfaces and configured to execute one or more processes; and
a memory configured to store a process that is executable by the processor, the process when executed configured to:
use a quality of experience model to make predictions comprising probabilities of a quality of experience metric for an online application accessed via a network being acceptable;
determine, for each of a subset of the predictions having lowest probabilities of acceptable user experience from amongst the predictions, whether that prediction corresponds to a negative experience rating by a user of the online application;
compute a negative retrieval accuracy for the quality of experience model based on a count of negative experience ratings associated with the subset and a size of the subset; and
use the negative retrieval accuracy to detect an issue in the network.
12. The apparatus as in claim 11 , wherein the quality of experience model makes the predictions based on telemetry data captured by the network.
13. The apparatus as in claim 11 , wherein the process when executed is further configured to:
provide the negative retrieval accuracy for presentation by a user interface.
14. The apparatus as in claim 11 , wherein a size of the subset of the predictions differs from a total amount of the predictions whose probabilities indicate negative user experiences.
15. The apparatus as in claim 11 , wherein the process when executed is further configured to:
compute a robust ranking accuracy metric in part by randomly selecting pairs of samples associated with different subsets of the predictions, wherein the pairs of samples have different percentages of positive user experience labels.
16. The apparatus as in claim 15 , wherein the process when executed is further configured to:
use the robust ranking accuracy metric to override a routing decision in the network.
17. The apparatus as in claim 15 , wherein the process when executed is further configured to:
provide the robust ranking accuracy metric to a path computation element or via a Border Gateway Protocol message in the network.
18. The apparatus as in claim 15 , wherein the apparatus computes the robust ranking accuracy metric in part by performing two-sample Mann-Whitney U testing on the pairs.
19. The apparatus as in claim 11 , wherein the process when executed is further configured to:
vary a size of the subset to compute different precision and recall metrics for the quality of experience model;
compute gain values for the different precision and recall metrics; and
compute a precision recall gain metric as an average of the gain values for the different precision and recall metrics.
20. A tangible, non-transitory, computer-readable medium storing program instructions that cause a device to execute a process comprising:
using, by the device, a quality of experience model to make predictions comprising probabilities of a quality of experience metric for an online application accessed via a network being acceptable;
determining, by the device and for each of a subset of the predictions having lowest probabilities of acceptable user experience from amongst the predictions, whether that prediction corresponds to a negative experience rating by a user of the online application;
computing, by the device, a negative retrieval accuracy for the quality of experience model based on a count of negative experience ratings associated with the subset and a size of the subset; and
using, by the device, the negative retrieval accuracy to detect an issue in the network.