IP Library Granted Patent US 11,928,010
Granted Patent B2
US 11,928,010 · App. 17/490,267 · Granted Mar 12, 2024

Extracting and selecting feature values from conversation logs of dialogue systems using predictive machine learning models

Inventors: Sergey Zeltyn (Haifa, IL); Avi Yaeli (Ramot Menashe, IL)
Assignee: International Business Machines Corporation
G06F11/079G06F18/2113G06F18/2115G06F18/214G06N20/00
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,928,010
App. No.
17/490,267
Granted
Mar 12, 2024
Kind
B2
Abstract

An example system includes a processor that can receive conversation logs of a dialogue system to be analyzed. The processor can train a predictive machine learning model using a training set of the conversation logs on a selected feature to obtain feature values with associated importance values. The processor can select a number of feature values using a significance score calculated based on the associated importance values. The processor can generate an interactive user interface including the selected number of feature values.

Claims (32)

1. A system, comprising a processor to:

receive conversation logs of a dialogue system to be analyzed;

train a predictive machine learning model using a training set of the conversation logs on a selected feature to obtain feature values with associated importance values;

select a plurality of the feature values using a significance score calculated based on the associated importance values; and

generate an interactive user interface comprising the selected plurality of feature values.

2. The system of claim 1 , wherein the processor is to train a predictive model on an interaction between two selected features, and obtain pairs of feature values for the interaction and an importance value associated with the interaction, wherein the selected plurality of feature values comprises a feature value for the interaction.

3. The system of claim 1 , wherein the processor is to display a root cause analysis setting user interface and receive a root cause analysis setting comprising a selected failure type, a selected flow, a selected subset of conversation steps, a selected feature, and a selected interaction type.

4. The system of claim 1 , wherein the processor is to apply frequency filtering on the feature values.

5. The system of claim 1 , wherein the calculated significance score is further calculated based on accuracy of a corresponding prediction model, wherein the accuracy of the corresponding prediction model is calculated by inputting a subset of the conversation logs not used for the training into the trained prediction model.

6. The system of claim 1 , wherein the calculated significance score is further calculated based on a cardinality of the feature values.

7. The system of claim 1 , wherein the processor is to apply statistical significance filtering on the feature values.

8. A computer-implemented method, comprising:

receiving, via a processor, a plurality of conversation logs, a feature and an interaction type to be analyzed;

training, via the processor, a predictive machine learning model on a subset of the plurality of conversation logs to obtain feature values with associated importance values for the feature, wherein the predictive machine learning model is trained with respect to a different feature to be analyzed;

selecting, via the processor, a plurality of feature values using a significance score calculated based on the associated importance values; and

generating, via the processor, an interactive user interface comprising the selected plurality of feature values.

9. The computer-implemented method of claim 8 , further comprising receiving a selected failure type to be analyzed.

10. The computer-implemented method of claim 8 , further comprising training a predictive machine learning model to obtain feature values with associated importance values for an interaction between two features based on the interaction type.

11. The computer-implemented method of claim 8 , comprising extracting feature values corresponding to interactions between two features.

12. The computer-implemented method of claim 8 , comprising generating a conversations transcript comprising relevant portions of conversation logs in response to detecting a selection of a significance bar chart in the interactive user interface.

13. The computer-implemented method of claim 8 , comprising generating a most significant n-gram chart in response to detecting a selection of a significance bar chart in the interactive user interface.

14. The computer-implemented method of claim 8 , further comprising receiving a blacklist comprising a subset of the feature values and executing the predictive machine learning model without the subset of the feature values to generate a second set of feature values with associated importance values.

15. A computer program product for generating interactive user interfaces, the computer program product comprising a computer-readable storage medium having program code embodied therewith, the program code executable by a processor to cause the processor to:

receive conversation logs, a selected feature, and an interaction type, to be analyzed;

train a predictive machine learning model on a subset of the conversation logs for the selected feature to obtain feature values with associated importance values;

select a plurality of the feature values using a significance score calculated based on the associated importance values; and

generate an interactive user interface comprising the selected plurality of feature values.

16. The computer program product of claim 15 , further comprising program code executable by the processor to receive a plurality of selected features and train a plurality of predictive machine learning models to obtain feature values with associated importance values for each of the plurality of selected features, wherein the selected plurality of feature values comprise feature values associated with different features.

17. The computer program product of claim 15 , further comprising program code executable by the processor to receive a plurality of selected features and a selected interaction between the plurality of selected features and train a predictive machine learning model to obtain pairs of feature values with associated importance values for the selected interaction between two of the selected features.

18. The computer program product of claim 15 , further comprising program code executable by the processor to calculate the significance score based on an accuracy of a corresponding prediction model and a cardinality of the feature values.

19. The computer program product of claim 15 , further comprising program code executable by the processor to generate a conversations transcript comprising relevant portions of conversation logs in response to detecting a selection of a significance bar chart in the interactive user interface.

20. The computer program product of claim 15 , further comprising program code executable by the processor to generate a most significant n-gram chart in response to detecting a selection of a significance bar chart in the interactive user interface.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 12, 2021
From: ZELTYN, SERGEY; YAELI, AVI
To: INTERNATIONAL BUSINESS MACHINES CORPORATION
Reel/Frame 057770/0120 →
Continuity (1)
Related Publication 20230097628A1 · Mar 30, 2023