Hybrid artificial intelligence chat system with contextual routing between language model and decision tree flows

1 . A method, performed at a computer system comprising a processor and a computer-readable medium, comprising:

receiving a query from a user client device that is associated with a user;

identifying, by a routing large language model (LLM), an intent of the user based in part on context associated with the query;

scoring each of a plurality of predefined decision tree flows based on the intent;

selecting, by the routing LLM and based in part on the intent, a response model from the plurality of predefined decision tree flows and a LLM chatbot, wherein selecting the response model from the plurality of predefined decision tree flows and the LLM chatbot comprises:

identifying a first predefined decision tree flow, of the plurality of predefined decision tree flows, with a highest score,

determining that the score of the first predefined decision tree flow is below a threshold score,

generating, by the routing LLM, a message requesting additional information for the query,

providing the message to the user client device,

receiving a message from the user client device that includes additional information,

identifying, by the routing LLM, an updated intent of the query based in part on the context and the additional information from the message,

scoring each of the plurality of predefined decision tree flows based on the updated intent,

identifying a second predefined decision tree flow, of the plurality of predefined decision tree flows, with a highest score, and

responsive to determining that the score of the second predefined decision tree flow is above a threshold score, selecting the second predefined decision tree flow as the response model;

using the selected response model to generate a response to the query; and

providing the response to the user client device, wherein providing the response to the user client device causes the user client device to display the response.

2 . The method of claim 1 , wherein identifying, by the routing LLM, the intent of the user based in part on the context of the query comprises:

retrieving data associated with the user;

generating, by the routing LLM, a message requesting additional information for the query, the generating based in part on the data and the query;

providing the message to the user client device;

receiving a message from the user client device that includes additional information; and

identifying, by the routing LLM, the intent of the query based on the context and based on the additional information from the user client device.

3 . The method of claim 1 , further comprising:

scoring each of the plurality of predefined decision tree flows based on the intent,

wherein selecting, by the routing LLM, the response model from the plurality of predefined decision tree flows and the LLM chatbot based in part on the intent, further comprises:

identifying a first predefined decision tree flow, of the plurality of predefined decision tree flows, with a highest score,

if the score of the first predefined decision tree flow is above a threshold score, selecting the first predefined decision tree flow as the response model, and

if the score of the first predefined decision tree flow is below the threshold score, selecting the LLM chatbot as the response model.

4 . The method of claim 1 , wherein the first predefined decision tree flow is the second predefined decision tree flow.

5 . The method of claim 1 , wherein using the selected response model to generate the response to the query further comprises:

retrieving order data associated with the user; and

generating, by the selected response model, the response based in part on the order data and the intent.

6 . The method of claim 1 , wherein the routing LLM was trained by:

accessing a set of training examples that includes training service data for a plurality of training queries;

applying the routing LLM to the set of training examples to generate a training output corresponding to a set of selections of training response models for at least some of the training queries;

back-propagating one or more error terms obtained from one or more loss functions to update a set of parameters of the routing LLM, and one or more of the error terms are based on a difference between a label applied to a test interaction of the set of training examples and the set of selections of training response models; and

stopping the back-propagation after the one or more loss functions satisfy one or more criteria.

7 . The method of claim 1 , further comprising:

receiving feedback from the user client device on the response;

generating additional training examples that include service data associated with different queries including the query; and

fine tuning the routing LLM using the additional training examples.

8 . The method of claim 1 , wherein each predefined decision tree flow of the plurality of predefined decision tree flows is designed to perform one or more of: cancelling an order, rescheduling an order, or tracking a package.

9 . A computer program product comprising a non-transitory computer readable storage medium having instructions encoded thereon that, when executed by a processor of a computer system, cause the computer system to perform steps comprising:

receiving a query from a user client device that is associated with a user;

identifying, by a routing large language model (LLM), an intent of the user based in part on context associated with the query;

scoring each of a plurality of predefined decision tree flows based on the intent;

selecting, by the routing LLM and based in part on the intent, a response model from the plurality of predefined decision tree flows and a LLM chatbot, wherein selecting the response model from the plurality of predefined decision tree flows and the LLM chatbot comprises:

identifying a first predefined decision tree flow, of the plurality of predefined decision tree flows, with a highest score,

determining that the score of the first predefined decision tree flow is below a threshold score,

generating, by the routing LLM, a message requesting additional information for the query,

providing the message to the user client device,

receiving a message from the user client device that includes additional information,

identifying, by the routing LLM, an updated intent of the query based in part on the context and the additional information from the message,

scoring each of the plurality of predefined decision tree flows based on the updated intent,

identifying a second predefined decision tree flow, of the plurality of predefined decision tree flows, with a highest score, and

responsive to determining that the score of the second predefined decision tree flow is above a threshold score, selecting the second predefined decision tree flow as the response model;

using the selected response model to generate a response to the query; and

providing the response to the user client device, wherein providing the response to the user client device causes the user client device to display the response.

10 . The computer program product of claim 9 , wherein the encoded instructions for identifying, by the routing LLM, the intent of the user based in part on the context of the query cause the computer system to perform steps comprising:

retrieving data associated with the user;

generating, by the routing LLM, a message requesting additional information for the query, the generating based in part on the data and the query;

providing the message to the user client device;

receiving a message from the user client device that includes additional information; and

identifying, by the routing LLM, the intent of the query based on the context and based on the additional information from the user client device.

11 . The computer program product of claim 9 , further comprising encoded instructions that when executed cause the computer system to perform steps comprising:

scoring each of the plurality of predefined decision tree flows based on the intent,

wherein the encoded instructions for selecting, by the routing LLM, the response model from the plurality of predefined decision tree flows and the LLM chatbot based in part on the intent cause the computer system to perform steps comprising:

identifying a first predefined decision tree flow, of the plurality of predefined decision tree flows, with a highest score,

if the score of the first predefined decision tree flow is above a threshold score, selecting the first predefined decision tree flow as the response model, and

if the score of the first predefined decision tree flow is below the threshold score, selecting the LLM chatbot as the response model.

12 . The computer program product of claim 9 , wherein

the first predefined decision tree flow is the second predefined decision tree flow.

13 . The computer program product of claim 9 , wherein the encoded instructions for using the selected response model to generate a response to the query cause the computer system to perform steps comprising:

retrieving order data associated with the user; and

generating, by the selected response model, the response based in part on the order data and the intent.

14 . The computer program product of claim 9 , wherein the routing LLM was trained by:

accessing a set of training examples that includes training service data for a plurality of training queries;

applying the routing LLM to the set of training examples to generate a training output corresponding to a set of selections of training response models for at least some of the training queries;

back-propagating one or more error terms obtained from one or more loss functions to update a set of parameters of the routing LLM, and one or more of the error terms are based on a difference between a label applied to a test interaction of the set of training examples and the set of selections of training response models; and

stopping the back-propagation after the one or more loss functions satisfy one or more criteria.

15 . The computer program product of claim 9 , further comprising encoded instructions that when executed cause the computer system to perform steps comprising:

receiving feedback from the user client device on the response;

generating additional training examples that include service data associated with different queries including the query; and

fine tuning the routing LLM using the additional training examples.

16 . The computer program product of claim 9 , wherein each predefined decision tree flow of the plurality of predefined decision tree flows is designed to perform one or more of: cancelling an order, rescheduling an order, or tracking a package.

17 . A computer system comprising:

a processor; and

a non-transitory computer readable storage medium having instructions encoded thereon that, when executed by the processor, cause the computer system to perform steps comprising:

receiving a query from a user client device that is associated with a user,

identifying, by a routing large language model (LLM), an intent of the user based in part on context associated with the query,

scoring each of a plurality of predefined decision tree flows based on the intent,

selecting, by the routing LLM and based in part on the intent, a response model from the plurality of predefined decision tree flows and a LLM chatbot, wherein selecting the response model from the plurality of predefined decision tree flows and the LLM chatbot comprises:

identifying a first predefined decision tree flow, of the plurality of predefined decision tree flows, with a highest score,

determining that the score of the first predefined decision tree flow is below a threshold score,

generating, by the routing LLM, a message requesting additional information for the query,

providing the message to the user client device,

receiving a message from the user client device that includes additional information,

identifying, by the routing LLM, an updated intent of the query based in part on the context and the additional information from the message,

scoring each of the plurality of predefined decision tree flows based on the updated intent,

identifying a second predefined decision tree flow, of the plurality of predefined decision tree flows, with a highest score, and

responsive to determining that the score of the second predefined decision tree flow is above a threshold score, selecting the second predefined decision tree flow as the response model;

using the selected response model to generate a response to the query, and

providing the response to the user client device, wherein providing the response to the user client device causes the user client device to display the response.

18 . The computer system of claim 17 , wherein the encoded instructions for identifying, by the routing LLM, the intent of the user based in part on the context of the query cause the computer system to perform steps comprising:

retrieving data associated with the user;

generating, by the routing LLM, a message requesting additional information for the query, the generating based in part on the data and the query;

providing the message to the user client device;

receiving a message from the user client device that includes additional information; and

identifying, by the routing LLM, the intent of the query based on the context and based on the additional information from the user client device.

19 . The computer system of claim 17 , further comprising encoded instructions that when executed cause the computer system to perform steps comprising:

scoring each of the plurality of predefined decision tree flows based on the intent,

wherein the encoded instructions for selecting, by the routing LLM, the response model from the plurality of predefined decision tree flows and the LLM chatbot based in part on the intent cause the computer system to perform steps comprising:

identifying a first predefined decision tree flow, of the plurality of predefined decision tree flows, with a highest score,

if the score of the first predefined decision tree flow is above a threshold score, selecting the first predefined decision tree flow as the response model, and

if the score of the first predefined decision tree flow is below the threshold score, selecting the LLM chatbot as the response model.

20 . Computer system of claim 17 , wherein each predefined decision tree flow of the plurality of predefined decision tree flows is designed to perform one or more of: cancelling an order, rescheduling an order, or tracking a package.