Automatic semantic analysis and comparison of chatbot capabilities

1. A computer-implemented method, the method comprising steps of:

obtaining a first chatbot specification associated with a first chatbot, the first chatbot specification comprising a first plurality of characteristics arranged in a plurality of categories;

obtaining a second chatbot specification associated with a second chatbot, the second chatbot specification comprising a second plurality of characteristics arranged in the plurality of categories;

identifying, based at least in part on the first plurality of characteristics and the second plurality of characteristics, one or more differences between the first plurality of characteristics and the second plurality of characteristics for each of the plurality of categories;

quantifying the identified differences for each category against the characteristics of that category, wherein said quantifying comprises determining a rating for each of the first and second chatbots with respect to each category, wherein said determining the rating for each of the first and second chatbots with respect to a given category comprises:

determining a total number of unique characteristics in the given category;

determining a first rating for the first chatbot based at least in part on the ratio of the number of characteristics in the first plurality of characteristics associated with the first chatbot and the total number of unique characteristics; and

determining a second rating for the second chatbot based at least in part on the ratio of the number of characteristics in the second plurality of characteristics associated with the second chatbot and the total number of unique characteristics;

generating a natural language expression corresponding to the identified one or more differences; and

presenting, to a user via a graphical user interface, (i) the generated natural language expression and (ii) the rating for each of the first and second chatbots with respect to each category;

wherein the steps are carried out by at least one computing device.

2. The computer-implemented method of claim 1 , wherein the plurality of categories comprises intents, entities, entity values, and dialog states.

3. The computer-implemented method of claim 1 , wherein said presenting the rating for each of the first and second chatbots with respect to each category to the user via the graphical user interface comprises presenting an indication of which of the first and second chatbots has a higher rating in each of the categories.

4. The computer-implemented method of claim 1 , comprising:

obtaining a domain ontology corresponding to a domain of the first and second chatbots, wherein the first and second pluralities of characteristics are mapped to characteristics in the domain ontology.

5. The computer-implemented method of claim 4 , comprising:

determining a difference between the first plurality of characteristics and the characteristics in the domain ontology for each category based on the mapping;

determining a difference between the second plurality of characteristics and the characteristics in the domain ontology for each category based on the mapping;

rating each of the first and second chatbots relative to the domain ontology for each category, based at least in part on (i) the determined difference between the first plurality of characteristics and the characteristics in the domain ontology for each category and (ii) the determined difference between the second plurality of characteristics and the characteristics in the domain ontology for each category; and

presenting the rating of each of the first and second chatbots relative to the domain ontology to the user via the graphical user interface.

6. A computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computing device to cause the computing device to:

obtain a first chatbot specification associated with a first chatbot, the first chatbot specification comprising a first plurality of characteristics arranged in a plurality of categories;

obtain a second chatbot specification associated with a second chatbot, the second chatbot specification comprising a second plurality of characteristics arranged in the plurality of categories;

identify, based at least in part on the first plurality of characteristics and the second plurality of characteristics, one or more differences between the first plurality of characteristics and the second plurality of characteristics for each of the plurality of categories;

quantify the identified differences for each category against the characteristics of that category, wherein said quantifying comprises determining a rating for each of the first and second chatbots with respect to each category, wherein said determining the rating for each of the first and second chatbots with respect to a given category comprises:

determining a total number of unique characteristics in the given category;

determining a first rating for the first chatbot based at least in part on the ratio of the number of characteristics in the first plurality of characteristics associated with the first chatbot and the total number of unique characteristics; and

determining a second rating for the second chatbot based at least in part on the ratio of the number of characteristics in the second plurality of characteristics associated with the second chatbot and the total number of unique characteristics;

generate a natural language expression corresponding to the identified one or more differences; and

present, to a user via a graphical user interface, (i) the generated natural language expression and (ii) the rating for each of the first and second chatbots with respect to each category.

7. The computer program product of claim 6 , wherein said presenting the rating for each of the first and second chatbots with respect to each category to the user via the graphical user interface comprises presenting an indication of which of the first and second chatbots has a higher rating in each of the categories.

8. The computer program product of claim 6 , wherein the program instructions further cause the computing device to obtain a domain ontology corresponding to a domain of the first and second chatbots, wherein the first and second pluralities of characteristics are mapped to characteristics in the domain ontology.

9. The computer program product of claim 8 , wherein the program instructions further cause the computing device to:

determine a difference between the first plurality of characteristics and the characteristics in the domain ontology for each category based on the mapping;

determine a difference between the second plurality of characteristics and the characteristics in the domain ontology for each category based on the mapping;

rate each of the first and second chatbots relative to the domain ontology for each category, based at least in part on (i) the determined difference between the first plurality of characteristics and the characteristics in the domain ontology for each category and (ii) the determined difference between the second plurality of characteristics and the characteristics in the domain ontology for each category; and

present the rating of each of the first and second chatbots relative to the domain ontology to the user via the graphical user interface.

10. A system comprising:

a memory; and

at least one processor operably coupled to the memory and configured for:

obtaining a first chatbot specification associated with a first chatbot, the first chatbot specification comprising a first plurality of characteristics arranged in a plurality of categories;

obtaining a second chatbot specification associated with a second chatbot, the second chatbot specification comprising a second plurality of characteristics arranged in the plurality of categories;

identifying, based at least in part on the first plurality of characteristics and the second plurality of characteristics, one or more differences between the first plurality of characteristics and the second plurality of characteristics for each of the plurality of categories;

quantifying the identified differences for each category against the characteristics of that category, wherein said quantifying comprises determining a rating for each of the first and second chatbots with respect to each category, wherein said determining the rating for each of the first and second chatbots with respect to a given category comprises:

determining a total number of unique characteristics in the given category;

determining a first rating for the first chatbot based at least in part on the ratio of the number of characteristics in the first plurality of characteristics associated with the first chatbot and the total number of unique characteristics; and

determining a second rating for the second chatbot based at least in part on the ratio of the number of characteristics in the second plurality of characteristics associated with the second chatbot and the total number of unique characteristics;

generating a natural language expression corresponding to the identified one or more differences; and

presenting, to a user via a graphical user interface, (i) the generated natural language expression and (ii) the rating for each of the first and second chatbots with respect to each category.

11. The system of claim 10 ,

wherein presenting the rating for each of the first and second chatbots with respect to each category to the user via the graphical user interface comprises presenting an indication of which of the first and second chatbots has a higher rating in each of the categories.

12. The system of claim 10 , wherein the at least one processor is further configured for obtaining a domain ontology corresponding to a domain of the first and second chatbots, wherein the first and second pluralities of characteristics are mapped to characteristics in the domain ontology.

13. The system of claim 12 , wherein the at least one processor is further configured for:

determining a difference between the first plurality of characteristics and the characteristics in the domain ontology for each category based on the mapping;

determining a difference between the second plurality of characteristics and the characteristics in the domain ontology for each category based on the mapping;

rating each of the first and second chatbots relative to the domain ontology for each category, based at least in part on (i) the determined difference between the first plurality of characteristics and the characteristics in the domain ontology for each category and (ii) the determined difference between the second plurality of characteristics and the characteristics in the domain ontology for each category; and

presenting the rating of each of the first and second chatbots relative to the domain ontology to the user via the graphical user interface.