On-premise machine learning model selection in a network assurance service
In one embodiment, a network assurance service uses a first machine-learning based model that is locally deployed to a network to assess a set of input features comprising measurements from the network. The service monitors, locally in the network, performance of the first machine learning-based model. The service determines that the monitored performance of the first machine learning-based model does not meet one or more performance requirements associated with the network. The service selects a second machine learning-based model for deployment to the network, based on the one or more performance requirements associated with the network and on the set of input features of the first machine learning-based model. The service deploys the selected second machine learning-based model to the network as a replacement for the first machine learning-based model.
1 . A method comprising:
using, by a network assurance service, a first machine learning-based model that is locally deployed to a network to assess a set of input features comprising measurements indicative of behavior of the network;
monitoring, by the service and locally in the network, performance of the first machine learning-based model according to its resource consumption;
determining, by the service, that the monitored performance of the first machine learning-based model does not meet one or more performance requirements associated with the network based on an assessment that the resource consumption of the first machine learning-based model exceeds a maximum allowed resource consumption;
disabling, by the service, one or more optional input features of the first machine learning-based model, based on the determination that the monitored performance of the first machine learning-based model does not meet the one or more performance requirements associated with the network, wherein the one or more optional input features are selected based on a feature classification stored for the first machine learning-based model that associates each input feature with an estimated resource consumption value;
selecting, by the service and only after determining that the monitored performance of the first machine learning-based model still does not meet set-meet the one or more performance requirements associated with the network after disabling the one or more optional input features according to the feature classification until either the one or more performance requirements are satisfied or no additional optional input features remain, a second machine learning-based model from among a set of machine learning-based models for deployment to the network; and
deploying, by the service, the selected second machine learning-based model to the network as a replacement for the first machine learning-based model.
2 . The method as in claim 1 , wherein the second machine learning-based model is trained based on measurements from one or more other networks different than the network where the first machine learning-based model is locally deployed.
3 . The method as in claim 1 , wherein monitoring the performance of the first machine learning-based model comprises:
receiving relevancy feedback from a user interface regarding alerts raised by the first machine learning-based model and sent to the user interface; and
using the feedback to compute a precision or recall for the first machine learning-based model.
4 . The method as in claim 1 , wherein the second machine learning-based model is selected based on a determination by the service that the second machine learning-based model consumes fewer resources than the first machine learning-based model.
5 . The method as in claim 1 , further comprising:
deploying, by the service, the first machine learning-based model to the network, to test whether the performance of the first machine learning-based model satisfies the performance requirements associated with the network.
6 . The method as in claim 5 , further comprising:
selecting, by the service, the first machine learning-based model for deployment to the network from among the set of machine learning-based models, based on the first machine learning-based model requiring the most resources for execution from among the set of machine learning-based models.
7 . The method as in claim 1 , wherein monitoring the performance of the first machine learning-based model comprises:
receiving feedback regarding outputs of the first machine learning-based model from a network security system; and
using the feedback to determine the performance of the first machine learning-based model.
8 . The method as in claim 1 , wherein the second machine learning-based model is selected by a cloud-based selection engine, the method further comprising:
sending an indication of the monitored performance of the first machine learning-based model to the cloud-based selection engine.
9 . An apparatus, comprising:
one or more network interfaces;
a processor coupled to the network interfaces and configured to execute one or more processes; and
a memory configured to store a process executable by the processor, the process when executed configured to:
use a first machine learning-based model that is locally deployed to a network to assess a set of input features comprising measurements indicative of behavior of the network;
monitor, locally in the network, performance of the first machine learning-based model according to its resource consumption;
determine that the monitored performance of the first machine learning-based model does not meet one or more performance requirements associated with the network based on an assessment that the resource consumption of the first machine learning-based model exceeds a maximum allowed resource consumption;
disable one or more optional input features of the first machine learning-based model, based on the determination that the monitored performance of the first machine learning-based model does not meet the one or more performance requirements associated with the network, wherein the one or more optional input features are selected based on a feature classification stored for the first machine learning-based model that associates each input feature with an estimated resource consumption value;
select, only after determining that the monitored performance of the first machine learning-based model still does not meet set-meet the one or more performance requirements associated with the network after disabling the one or more optional input features according to the feature classification until either the one or more performance requirements are satisfied or no additional optional input features remain, a second machine learning-based model from among a set of machine learning-based models for deployment to the network; and
deploy the selected second machine learning-based model to the network as a replacement for the first machine learning-based model.
10 . The apparatus as in claim 9 , wherein the second machine learning-based model is trained based on measurements from one or more other networks different than the network where the first machine learning-based model is locally deployed.
11 . The apparatus as in claim 9 , wherein the apparatus monitors the performance of the first machine learning-based model by:
receiving relevancy feedback from a user interface regarding alerts raised by the first machine learning-based model and sent to the user interface; and
using the feedback to compute a precision or recall for the first machine learning-based model.
12 . The apparatus as in claim 9 , wherein the second machine learning-based model is selected based on a determination by the apparatus that the second machine learning-based model consumes fewer resources than the first machine learning-based model.
13 . The apparatus as in claim 9 , wherein the process when executed is further configured to:
deploy the first machine learning-based model to the network, to test whether the performance of the first machine learning-based model satisfies the performance requirements associated with the network.
14 . The apparatus as in claim 13 , wherein the process when executed is further configured to:
select the first machine learning-based model for deployment to the network from among the set of machine learning-based models, based on the first machine learning-based model requiring the most resources for execution from among the set of machine learning-based models.
15 . The apparatus as in claim 9 , wherein the apparatus monitors the performance of the first machine learning-based model by:
receiving feedback regarding outputs of the first machine learning-based model from a network security system; and
using the feedback to determine the performance of the first machine learning-based model.
16 . The apparatus as in claim 9 , wherein the second machine learning-based model is selected by a cloud-based selection engine, and wherein the process when executed is further configured to:
send an indication of the monitored performance of the first machine learning-based model to the cloud-based selection engine.
17 . A tangible, non-transitory, computer-readable medium storing program instructions that cause a network assurance service to execute a process comprising:
using, by the network assurance service, a first machine learning-based model that is locally deployed to a network to assess a set of input features comprising measurements indicative of behavior of the network;
monitoring, by the service and locally in the network, performance of the first machine learning-based model according to its resource consumption;
determining, by the service, that the monitored performance of the first machine learning-based model does not meet one or more performance requirements associated with the network based on an assessment that the resource consumption of the first machine learning-based model exceeds a maximum allowed resource consumption;
disabling, by the service, one or more optional input features of the first machine learning-based model, based on the determination that the monitored performance of the first machine learning-based model does not meet the one or more performance requirements associated with the network, wherein the one or more optional input features are selected based on a feature classification stored for the first machine learning-based model that associates each input feature with an estimated resource consumption value;
selecting, by the service and only after determining that the monitored performance of the first machine learning-based model still does not meet the one or more performance requirements associated with the network after disabling the one or more optional input features according to the feature classification until either the one or more performance requirements are satisfied or no additional optional input features remain, a second machine learning-based model from among a set of machine learning-based models for deployment to the network; and
deploying, by the service, the selected second machine learning-based model to the network as a replacement for the first machine learning-based model.
18 . The tangible, non-transitory, computer-readable medium as in claim 17 , wherein the second machine learning-based model is trained based on measurements from one or more other networks different than the network where the first machine learning-based model is locally deployed.