Dialog flow evaluation

1. A method, comprising:

receiving information corresponding to a dialog flow between a virtual assistant and a user, wherein the information comprises dialog logs from previous conversations between the virtual assistant and users;

identifying, from the received information, dialog states, wherein each dialog state corresponds to a possible response by the virtual assistant, wherein the identifying comprises analyzing the dialog logs and identifying responses provided by the virtual assistant;

classifying a set of the identified dialog states as grounded dialog states, wherein the classifying comprises (i) receiving scoring input representing a desirability for each of the grounded dialog states and (ii) generating a score vector, for each of the grounded dialog states, based upon the received scoring input for each of the grounded dialog states, wherein the classifying comprises classifying more than one of the dialog states as a grounded dialog state and wherein each of the grounded dialog states is separated by at least one other dialog state, wherein the grounded dialog state comprises a dialog state having a scoring input provided by another user;

assigning a score vector to each of the identified dialog states other than the grounded dialog states, wherein the assigning a score vector comprises extrapolating the score vector of a grounded dialog state to dialog states that the grounded dialog state is dependent upon; and

computing an aggregate score vector for the dialog flow by weighting the score vectors for each of the identified dialog states within the dialog flow and averaging the weighted dialog state score vectors, wherein the aggregate score vector identifies an overall desirability of the dialog flow.

2. The method of claim 1 , wherein the information comprises a dialog model corresponding to the virtual assistant.

3. The method of claim 1 , wherein the identifying dialog states comprises analyzing the dialog logs and classifying each response by the virtual assistant within the conversation to a dialog state.

4. The method of claim 1 , wherein the desirability of a dialog state is based upon a response by a user to an output provided by the virtual assistant.

5. The method of claim 1 , wherein the generating a score vector comprises combining a feature of the dialog state with the scoring input into a vector.

6. The method of claim 5 , wherein a feature of a dialog state represents a probability of a predetermined response by the virtual assistant.

7. The method of claim 1 , wherein the extrapolating comprises determining a statistical causality of the dialog states that the grounded dialog state is dependent upon on the dependent grounded dialog state.

8. The method of claim 1 , wherein the weighting the score vectors for each dialog state is based upon a frequency of occurrence of the dialog state.

9. An apparatus, comprising:

at least one processor; and

a computer readable storage medium having computer readable program code embodied therewith and executable by the at least one processor, the computer readable program code comprising:

computer readable program code configured to receive information corresponding to a dialog flow between a virtual assistant and a user, wherein the information comprises dialog logs from previous conversations between the virtual assistant and users;

computer readable program code configured to identify, from the received information, dialog states, wherein each dialog state corresponds to a possible response by the virtual assistant, wherein the identifying comprises analyzing the dialog logs and identifying responses provided by the virtual assistant;

computer readable program code configured to classify a set of the identified dialog states as grounded dialog states, wherein the classifying comprises (i) receiving scoring input representing a desirability for each of the grounded dialog states and (ii) generating a score vector, for each of the grounded dialog states, based upon the received scoring input for each of the grounded dialog states, wherein the classifying comprises classifying more than one of the dialog states as a grounded dialog state and wherein each of the grounded dialog states is separated by at least one other dialog state, wherein the grounded dialog state comprises a dialog state having a scoring input provided by another user;

computer readable program code configured to assign a score vector to each of the identified dialog states other than the grounded dialog states, wherein the assigning a score vector comprises extrapolating the score vector of a grounded dialog state to dialog states that the grounded dialog state is dependent upon; and

computer readable program code configured to compute an aggregate score vector for the dialog flow by weighting the score vectors for each of the identified dialog states within the dialog flow and averaging the weighted dialog state score vectors, wherein the aggregate score vector identifies an overall desirability of the dialog flow.

10. A computer program product, comprising:

a computer readable storage medium having computer readable program code embodied therewith, the computer readable program code executable by a processor and comprising:

computer readable program code configured to receive information corresponding to a dialog flow between a virtual assistant and a user, wherein the information comprises dialog logs from previous conversations between the virtual assistant and users;

computer readable program code configured to identify, from the received information, dialog states, wherein each dialog state corresponds to a possible response by the virtual assistant, wherein the identifying comprises analyzing the dialog logs and identifying responses provided by the virtual assistant;

computer readable program code configured to classify a set of the identified dialog states as grounded dialog states, wherein the classifying comprises (i) receiving scoring input representing a desirability for each of the grounded dialog states and (ii) generating a score vector, for each of the grounded dialog states, based upon the received scoring input for each of the grounded dialog states, wherein the classifying comprises classifying more than one of the dialog states as a grounded dialog state and wherein each of the grounded dialog states is separated by at least one other dialog state, wherein the grounded dialog state comprises a dialog state having a scoring input provided by another user;

computer readable program code configured to assign a score vector to each of the identified dialog states other than the grounded dialog states, wherein the assigning a score vector comprises extrapolating the score vector of a grounded dialog state to dialog states that the grounded dialog state is dependent upon; and

computer readable program code configured to compute an aggregate score vector for the dialog flow by weighting the score vectors for each of the identified dialog states within the dialog flow and averaging the weighted dialog state score vectors, wherein the aggregate score vector identifies an overall desirability of the dialog flow.

11. The computer program product of claim 10 , wherein the identifying dialog states comprises analyzing the dialog logs and classifying each response by the virtual assistant within the conversation to a dialog state.

12. The computer program product of claim 10 , wherein the desirability of a dialog state is based upon a response by a user to an output provided by the virtual assistant.

13. The computer program product of claim 10 , wherein the generating a score vector comprises combining a feature of the dialog state with the scoring input into a vector.

14. The computer program product of claim 13 , wherein a feature of a dialog state represents a probability of a predetermined response by the virtual assistant.

15. The computer program product of claim 10 , wherein the extrapolating comprises determining a statistical causality of the dialog states that the grounded dialog state is dependent upon on the dependent grounded dialog state.

16. The computer program product of claim 10 , wherein the weighting the score vectors for each dialog state based upon a frequency of occurrence of the dialog state.

17. A method, comprising:

accessing at least one dialog model corresponding to a digital assistant, wherein the at least one dialog model identifies a dialog flow between the digital assistant and a user, wherein the at least one dialog model identifies a plurality of dialog nodes that correspond to possible responses that can be provided by the dialog assistant;

identifying dialog states within the dialog model, wherein the dialog states correspond to responses provided by the digital assistant;

receiving input, from a user, scoring each of at least a portion of the identified dialog states, wherein the score for each dialog state identifies a desirability of the dialog state;

classifying the scored dialog states as grounded dialog states, wherein the classifying comprises classifying more than one of the dialog states as a grounded dialog state and wherein each of the grounded dialog states is separated by at least one other dialog state;

scoring, based upon the score of the grounded dialog states, each of the remaining identified dialog states, wherein the scoring comprises statistically generating a score for each of the remaining identified dialog states based upon the dependence of the grounded dialog states to each of the remaining identified dialog states; and

generating, using the grounded dialog states, an aggregate score for the at least one dialog model by weighting the score of each dialog state based upon a frequency of occurrence of the dialog state and averaging the weighted dialog state scores, wherein the aggregate score identifies an overall desirability of the dialog flow.