Model-agnostic visualizations using linear programming approximation

1 . A method of visually presenting a representation of the effect of language elements on outcome of a conversation C between a user and a virtual assistant to a representative, the method comprising:

generating a graphical user interface and displaying a conversation C on the graphical user interface to the representative;

identifying a set of turns T 1 , . . . , T N corresponding to the conversation C;

creating a set of perturbations to be applied to data representing a set of textual language elements in conversation C between a user and a virtual assistant, wherein the virtual assistant uses a large language model;

applying the set of perturbations to the data;

determining a weight w i >0 for each of the turns T i in the set of turns such that, given any perturbation P from the set of perturbations, a linear combination of the weights of the turns affected by perturbation P approximates a probability of escalation;

formulating a plurality of linear programming approximations using results from the perturbations and solving the plurality of linear programming approximations to generate solutions;

displaying on the graphical user interface a visual presentation to the representative based on the solutions and the probability of escalation, whereby visual aspects of the visual presentation represent relative possibility that one of the perturbations will change a classification of the conversation, the possibility based on the solutions, wherein the large language model was trained using one or more previous perturbations and solutions.

2 . The method of claim 1 , wherein, for each perturbation, a respective linear programming approximation takes the weights into account to approximate the classification of the conversation.

3 . The method of claim 1 , wherein the visual representation comprises a visual scale indicative of the importance of the textual language elements to the classifier outcome, the visual scale comprising a color indicator, wherein color and intensity of the color indicator reflects relative importance of the classifier outcome for each of the textual language elements.

4 . The method of claim 1 , wherein the visual representation takes into account relative weights of each perturbation.

5 . One or more non-transitory computer-readable storage media storing computer-readable instructions that, when execute, instruct one or more processors to perform operations, comprising:

generating a graphical user interface and displaying a conversation C on the graphical user interface to a representative;

identifying a set of turns T 1 , . . . , T N corresponding to the conversation C;

creating a set of perturbations to be applied to data representing a set of textual language elements in conversation C between a user and a virtual assistant, wherein the virtual assistant uses a large language model;

applying the set of perturbations to the data;

determining a weight w i >0 for each of the turns Ti in the set of turns such that, given any perturbation P from the set of perturbations, a linear combination of the weights of the turns affected by perturbation P approximates a probability of escalation;

formulating a plurality of linear programming approximation using results from the perturbations and solving the plurality of linear programming approximations to generate solutions; and

displaying on the graphical user interface a visual presentation to a representative based on the solutions and the probability of escalation, whereby visual aspects of the visual presentation represent relative possibility that one of the perturbations will change a classification of the conversation, the possibility based on the solutions, wherein the large language model was trained using one or more previous perturbations and solutions.

6 . The one or more non-transitory computer-readable storage media of claim 5 , wherein, for each perturbation, a respective linear programming approximation takes the weights into account to approximate the classification of the conversation.

7 . The one or more non-transitory computer-readable storage media of claim 5 , wherein the visual representation comprises a visual scale indicative of the importance of the textual language elements to the classifier outcome, the visual scale comprising a color indicator, wherein color and intensity of the color indicator reflects relative importance of the classifier outcome for each of the textual language elements.

8 . The one or more non-transitory computer-readable storage media of claim 5 ,

wherein the visual representation takes into account relative weights of each perturbation.

9 . A method of identifying effect of language elements on an outcome of a conversation between a user and a virtual assistant, the method comprising, comprising:

generating a graphical user interface and displaying a conversation C on the graphical user interface to the user;

for each turn Ti of the conversation C =(T 1 , . . . , T N ) between the user and the virtual assistant, wherein the virtual assistant uses a large language model:

removing data representing the turn Ti from a conversation dataset representing the conversation to produce a perturbed dataset, the conversation having a plurality of dialogue turns and the conversation having a classification of escalation or non-escalation before removing the data;

applying a classifier to the perturbed dataset; and

upon determining that the perturbed dataset has a different classification than the conversation dataset, displaying on the graphical interface a visual indication that the turn Ti represented by the removed data is important to an outcome of applying the classifier, wherein the large language model was trained using one or more previous perturbations and solutions.

10 . The method of claim 9 , the steps further comprising:

displaying on the graphical user interface a visual scale indicative of the importance of the turns represented by the removed data to outcome of applying the classifier.

11 . The method of claim 10 , the steps further comprising the visual scale comprising a color indicator, wherein color and intensity of the color indicator reflects relative importance of the classifier outcome for each of the turns.

12 . A non-transitory computer-readable storage media having stored therein software executable to perform steps, comprising:

generating a graphical user interface and displaying a conversation C between a user and a virtual assistant on the graphical user interface to the user, wherein the virtual assistant uses a large language model;

for each turn Ti of a conversation C =(T 1 , . . . , T N ) between the user and the virtual assistant:

removing data representing the turn Ti from a conversation dataset representing the conversation to produce a perturbed dataset, the conversation having a plurality of dialogue turns and the conversation having a classification of escalation or non-escalation before removing the data;

applying a classifier to the perturbed dataset; and

upon determining that the perturbed dataset has a different classification than the conversation dataset, displaying on the graphical interface a visual indication that the turn Ti represented by the removed data is important to an outcome of applying the classifier, wherein the large language model was trained using one or more previous perturbations and solutions.

13 . The non-transitory computer-readable storage media of claim 12 , the steps further comprising:

displaying on a graphical user interface a visual scale indicative of the importance of the turns represented by the removed data to outcome of applying the classifier.

14 . The non-transitory computer-readable storage media of claim 13 , the steps further comprising the visual scale comprising a color indicator, wherein color and intensity of the color indicator reflects relative importance of the classifier outcome for each of the turns.