IP Library › Granted Patent US 11,722,598
Granted Patent B2
US 11,722,598 · App. 17/896,024 · Granted Aug 8, 2023

System and methods for an automated chatbot testing platform

Inventors: Geoff Willshire (Greenslopes, AU); Florian Treml (Klausen-Leopoldsdorf, AU); Christoph Börner (Vienna, AU)
Assignee: CYARA SOLUTIONS PTY LTD
H04M3/493G06F11/3684G06F11/3692G10L15/22H04M3/24H04M3/4938H04M7/1295
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,722,598
App. No.
17/896,024
Granted
Aug 8, 2023
Kind
B2
Abstract

A system and method for automated chatbot testing to provide training and quality assurance of conversational artificial intelligence systems, comprising a chatbot testing administrator interface which allows chatbot makers to define what a chatbot is supposed to do, create test scripts to test the performance of the chatbot, and review the results of the chatbot tests, a chatbot testing server which provides and interface between chatbot testing agents and the administrator interface, instantiates chatbot agents and distributes them across available hardware and runs testing programs which activate, configure, and deactivate chatbot testing agents as needed. A plurality of chatbot agents may be running in parallel to provide automated testing based upon test script configuration.

Claims (55)

1. A system for enhanced automated response systems testing, comprising:

a chatbot testing server comprising a first plurality of programming instructions stored in a memory of, and operating on a processor of, a computing device, wherein the first plurality of programming instructions, when operating on the processor, cause the computing device to:

retrieve a chatbot test case;

wherein the chatbot test case comprises a conversation flow and configuration data;

select a chatbot connector based on the configuration data;

initialize a chatbot agent based on the configuration data;

wherein the chatbot agent is initialized on an available hardware component;

use the selected chatbot connector to connect the chatbot agent with an enterprise chatbot;

execute the chatbot test case by performing the conversation flow between the connected chatbot agent and the enterprise chatbot;

monitor the results of the test case to indicate points of failure in the enterprise chatbot;

wherein the points of failure occur when a response from the enterprise chatbot does not match the conversation flow of the test script; and

send the results of the chatbot test case to a user interface for user review.

2. The system of claim 1 , wherein the user interface is an administrator interface comprising a second plurality of programming instructions stored in the memory of, and operating on the processor of, the computing device, wherein the second plurality of programming instructions, when operating on the processor, cause the computing device to:

allow a user to create a chatbot test case;

store the created chatbot test case in a database;

receive the results of the chatbot test case from the chatbot testing server; and

display the results of the chatbot test case.

3. The system of claim 1 , wherein the chatbot connector is selected from a list comprising a plurality of pre-configured chatbot connectors.

4. The system of claim 1 , wherein the chatbot connector is a user defined chatbot connector.

5. The system of claim 1 , wherein the test script further comprises one or more asserters and one or more logic hooks.

6. The system of claim 1 , wherein the conversational flow comprises user utterances and chatbot utterances.

7. The system of claim 6 , wherein the user utterances and the chatbot utterances are selected from respective utterance lists.

8. The system of claim 1 , further comprising a chatbot crawler comprising a third plurality of programming instructions stored in the memory of, and operating on the processor of, the computing device, wherein the third plurality of programming instructions, when operating on the processor, cause the computing device to:

connect to the enterprise chatbot and traverse the conversation flow detecting one or more conversation paths;

for each detected conversation path generate a path flow chart and a conversation;

after all conversation paths in the conversation flow have been detected, generate a conversation tree flow chart;

display the conversation tree flow chart at the user interface; and

produce a plurality of chatbot test cases, wherein each test case is based at least in part on the conversation flow and the conversation of a detected conversation path.

9. A method for enhanced automated response systems testing, comprising the steps of:

retrieving a chatbot test case;

wherein the chatbot test case comprises a conversation flow and configuration data;

selecting a chatbot connector based on the configuration data;

initializing a chatbot agent based on the configuration data;

wherein the chatbot agent is initialized on an available hardware component;

using the selected chatbot connector to connect the chatbot agent with an enterprise chatbot;

executing the chatbot test case by performing the conversation flow between the connected chatbot agent and the enterprise chatbot;

monitoring the results of the test case to indicate points of failure in the enterprise chatbot;

wherein the points of failure occur when a response from the enterprise chatbot does not match the conversation flow of the test script; and

sending the results of the chatbot test case to a user interface for user review.

10. The method of claim 9 , wherein the user interface is an administrator interface configured to perform the following additional steps:

allowing a user to create a chatbot test case;

storing the created chatbot test case in a database;

receiving the results of the chatbot test case from the chatbot testing server; and

displaying the results of the chatbot test case.

11. The method of claim 9 , wherein the chatbot connector is selected from a list comprising a plurality of pre-configured chatbot connectors.

12. The method of claim 9 , wherein the chatbot connector is a user defined chatbot connector.

13. The method of claim 9 , wherein the test script further comprises one or more asserters and one or more logic hooks.

14. The method of claim 9 , wherein the conversational flow comprises user utterances and chatbot utterances.

15. The method of claim 14 , wherein the user utterances and the chatbot utterances are selected from respective utterance lists.

16. The method of claim 9 , further comprising the steps of:

connecting to the enterprise chatbot and traverse the conversation flow detecting one or more conversation paths;

for each detected conversation path generating a path flow chart and a conversation;

after all conversation paths in the conversation flow have been detected, generating a conversation tree flow chart;

displaying the conversation tree flow chart at the user interface; and

producing a plurality of chatbot test cases, wherein each test case is based at least in part on the conversation flow and the conversation of a detected conversation path.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 15, 2023
From: WILLSHIRE, GEOFF
To: CYARA SOLUTIONS PTY LTD
Reel/Frame 063646/0583 →
Continuity (6)
Continuation In Part 16985652 · Aug 5, 2020
Continuation In Part 16379084 · Apr 9, 2019
Continuation 15091556 · Apr 5, 2016
Continuation In Part 14590972 · Jan 6, 2015
Provisional Application 63354616 · Jun 22, 2022
Related Publication 20220407960A1 · Dec 22, 2022
Cited By (3)
US 12,225,157 US 12,530,282 US 12,712,830