Evaluation framework for LLM-based network troubleshooting and monitoring agents
In one implementation, a device uses a large language model-based agent to perform a task in a network, to obtain a first result. The device executes code in the network to perform the task, to obtain a second result. The device makes a comparison between the first result and the second result. The device provides, based in part on the comparison, one or more performance metrics for the large language model-based agent for display.
1 . A method comprising:
using, by a device, a large language model-based agent to perform a task in a network, to obtain a first result related to a first detected state of the network;
causing execution of predetermined code to perform the task, to obtain a second result related to a second detected state of the network, the execution of the predetermined code including communicating with one or more network devices in the network, the predetermined code being associated with input previously provided by one or more users;
making, by the device, a comparison between the first result and the second result; and
providing, by the device and based in part on the comparison, one or more performance metrics for the large language model-based agent for display.
2 . The method as in claim 1 , wherein using the large language model-based agent to perform the task in the network comprises:
instructing the large language model-based agent to use a particular large language model to perform the task.
3 . The method as in claim 1 , wherein using the large language model-based agent to perform the task in the network comprises:
configuring the large language model-based agent using a parameter that controls at least one of: a prompting strategy used by the large language model-based agent to perform the task or a degree of randomness of a large language model used by the large language model-based agent to perform the task.
4 . The method as in claim 1 , wherein using the large language model-based agent to perform the task in the network comprises:
configuring the large language model-based agent using a parameter that controls at least one of: a maximum number of tokens the large language model-based agent can use to perform the task or a maximum number of steps the large language model-based agent is allowed to perform to complete the task.
5 . The method as in claim 1 , further comprising:
sending, by the device, a request for a subject matter expert to answer a questionnaire; and
translating, by the device, answers to the questionnaire into the task and the predetermined code.
6 . The method as in claim 1 , wherein the task comprises at least one of: troubleshooting an issue in the network, monitoring the network, or configuring a portion of the network.
7 . The method as in claim 1 , wherein the task has an associated difficulty rating specified by a user.
8 . The method as in claim 1 , wherein using the large language model-based agent to perform the task in the network comprises:
sending a text-based question to the large language model-based agent.
9 . The method as in claim 1 , further comprising:
using reinforcement learning to train the large language model-based agent, based on the comparison.
10 . The method as in claim 1 , wherein the one or more performance metrics comprise at least one of: an efficacy metric, a recall metric, a precision metric, an error rate, or a response time.
11 . An apparatus, comprising:
one or more network interfaces;
a processor coupled to the one or more network interfaces; and
a memory configured to store instructions that, when executed by the processor, configure the processor to:
use a large language model-based agent to perform a task in a network, to obtain a first result related to a first detected state of the network;
cause execution of predetermined code to perform the task, to obtain a second result related to a second detected state of the network, the execution of the predetermined code including communicating with one or more network devices in the network, the predetermined code being associated with input previously provided by one or more users;
make a comparison between the first result and the second result; and
provide, based in part on the comparison, one or more performance metrics for the large language model-based agent for display.
12 . The apparatus as in claim 11 , wherein the processor is configured to use the large language model-based agent to perform the task in the network by:
instructing the large language model-based agent to use a particular large language model to perform the task.
13 . The apparatus as in claim 11 , wherein the processor is further configured to:
configure the large language model-based agent using a parameter that controls at least one of: a prompting strategy used by the large language model-based agent to perform the task or a degree of randomness of a large language model used by the large language model-based agent to perform the task.
14 . The apparatus as in claim 11 , wherein the processor is further configured to:
configure the large language model-based agent using a parameter that controls at least one of: a maximum number of tokens the large language model-based agent can use to perform the task or a maximum number of steps the large language model-based agent is allowed to perform to complete the task.
15 . The apparatus as in claim 11 , wherein the processor is further configured to:
send a request for a subject matter expert to answer a questionnaire; and
translate answers to the questionnaire into the task and the predetermined code.
16 . The apparatus as in claim 11 , wherein the task comprises at least one of: troubleshooting an issue in the network, monitoring the network, or configuring a portion of the network.
17 . The apparatus as in claim 11 , wherein the task has an associated difficulty rating specified by a user.
18 . The apparatus as in claim 11 , wherein the processor is configured to use the large language model-based agent to perform the task in the network by:
sending a text-based question to the large language model-based agent.
19 . The apparatus as in claim 11 , wherein the processor is further configured to:
use reinforcement learning to train the large language model-based agent, based on the comparison.
20 . A tangible, non-transitory, computer-readable medium storing program instructions that cause a device to execute a process comprising:
using, by the device, a large language model-based agent to perform a task in a network, to obtain a first result related to a first detected state of the network;
causing execution of predetermined code to perform the task, to obtain a second result related to a second detected state of the network, the execution of the predetermined code including communicating with one or more network devices in the network, the predetermined code being associated with input previously provided by one or more users;
making, by the device, a comparison between the first result and the second result; and
providing, by the device and based in part on the comparison, one or more performance metrics for the large language model-based agent for display.