Computer network monitoring and control using a fine-tuned language model
In one implementation, a device generates a response to user input using a first language model. The device determines that the response is an erroneous response. The device generates a resolution to the erroneous response using a teacher language model. The device updates the first language model using the resolution from the teacher language model.
1 . A method comprising:
generating, by a device, a response to user input using a first language model;
determining, by the device, that the response is an erroneous response;
making, by the device, a determination whether to delay querying a teacher language model regarding the erroneous response until after handling of the user input has completed;
querying, by the device and after handling of the user input has completed when the determination is to delay, the teacher language model based on the erroneous response in order to obtain a resolution to the erroneous response;
generating, by the device and based on an output from querying the teacher language model regarding the erroneous response, a resolution to the erroneous response including a corrected version of the erroneous response; and
updating, by the device, the first language model using the resolution from the teacher language model.
2 . The method as in claim 1 , further comprising:
blocking, by the device, the first language model from providing the erroneous response to a user interface associated with the user input.
3 . The method as in claim 1 , wherein determining that the response is an erroneous response comprises:
detecting a syntax error or incorrect parameter within the response.
4 . The method as in claim 1 , wherein generating the resolution to the erroneous response using the teacher language model comprises:
receiving, at the device, feedback from an expert regarding the resolution.
5 . The method as in claim 1 , wherein the determination whether to delay querying the teacher language model is based on one or more of an identification of the user or a current load for the teacher model.
6 . The method as in claim 1 , wherein the response comprises a chart or plot of a network generated by the first language model based on networking telemetry from that network.
7 . The method as in claim 1 , wherein the response comprises a command line interface (CLI) command for a networking element in a network.
8 . The method as in claim 1 , wherein the response comprises one or more of an analysis of a packet trace from a network or an application programming interface (API) query.
9 . The method as in claim 1 , wherein updating the first language model includes using the resolution to augment subsequent contexts provided to the first language model.
10 . The method as in claim 1 , wherein the first language model is configured to perform a monitoring or control action in a computer network.
11 . An apparatus, comprising:
one or more network interfaces;
a processor coupled to the one or more network interfaces and configured to execute one or more processes; and
a memory configured to store a process that is executable by the processor, the process when executed configured to:
generate a response to user input using a first language model;
determine that the response is an erroneous response;
make a determination whether to delay querying a teacher language model regarding the erroneous response until after handling of the user input has completed;
query, after handling of the user input has completed when the determination is to delay, the teacher language model based on the erroneous response in order to obtain a resolution to the erroneous response;
generate, based on an output from querying the teacher language model regarding the erroneous response, a resolution to the erroneous response including a corrected version of the erroneous response; and
update the first language model using the resolution from the teacher language model.
12 . The apparatus as in claim 11 , wherein the process when executed is further configured to:
block the first language model from providing the erroneous response to a user interface associated with the user input.
13 . The apparatus as in claim 11 , wherein the apparatus determines that the response is an erroneous response by:
detecting a syntax error or incorrect parameter within the response.
14 . The apparatus as in claim 11 , wherein the apparatus generates the resolution to the erroneous response using the teacher language model by:
receiving feedback from an expert regarding the resolution.
15 . The apparatus as in claim 11 , wherein the teacher language model is larger in size than that of the first language model.
16 . The apparatus as in claim 11 , wherein the response comprises a chart or plot of a network generated by the first language model based on networking telemetry from that network.
17 . The apparatus as in claim 11 , wherein the response comprises a command line interface (CLI) command for a networking element in a network.
18 . The apparatus as in claim 11 , wherein the response comprises an analysis of a packet trace from a network.
19 . The apparatus as in claim 11 , wherein the response comprises an application programming interface (API) query.
20 . A tangible, non-transitory, computer-readable medium storing program instructions that cause a device to execute a process comprising:
generating, by the device, a response to user input using a first language model;
determining, by the device, that the response is an erroneous response;
making, by the device, a determination whether to delay querying a teacher language model regarding the erroneous response until after handling of the user input has completed;
querying, by the device and after handling of the user input has completed when the determination is to delay, the teacher language model based on the erroneous response in order to obtain a resolution to the erroneous response;
generating, by the device and based on an output from querying the teacher language model regarding the erroneous response, a resolution to the erroneous response including a corrected version of the erroneous response; and
updating, by the device, the first language model using the resolution from the teacher language model.