Granted Patent B2

US 12,585,677 · App. 18/124,589 · Granted Mar 24, 2026

Inventors: Sara Rosenthal (Spring Valley, NY); Odellia Boni (Giva'at Ela, IL); Michal Shmueli-Scheuer (Tel-aviv, IL); Ora Peled Nakash (Ramat David, IL)

Assignee: International Business Machines Corporation

G06F16/3329G06F16/3322

Abstract

Claims (118)

1 . A computer-implemented method comprising, automatically:

receiving data comprising: a question provided by a user, a list that constitutes a direct answer to the question, and an introductory text to the list;

using a first machine learning model to classify the introductory text as redundant or nonredundant, based on at least one of the question and the list, wherein the first machine learning model is a language model, wherein:

redundant introductory text is introductory text that conveys information which is included, explicitly or implicitly, in at least one of the question and the list; and

nonredundant introductory text is introductory text that conveys information which is not included, explicitly or implicitly, in at least one of the question and the list;

using a second machine learning model to classify the list as belonging to a certain list type out of multiple list types, based on the list, wherein the second machine learning model is another language model;

providing to the user:

(a) the introductory text, only if the introductory text has been classified as nonredundant,

(b) all or only a subset of items of the list,

(c) an indication as to a number of non-provided items of the list or a number of all items of the list, if only a subset of the items is being provided in (b), and

(d) a description of the certain list type; and

training the first machine learning model by:

obtaining multiple samples, each comprising a question, a list, and an introductory text;

for each of the samples:

computing a similarity measure between (i) at least one of: the respective question and the respective list, and (ii) the respective introductory text, and

weakly labeling the respective introductory text as redundant or nonredundant, based on the similarity measure associated with the respective introductory text and on a predetermined similarity measure threshold; and

adapting a first language model based on the samples and the weak labels of the introductory texts to form the first machine learning model.

2 . The computer-implemented method of claim 1 , further comprising training the second machine learning model by:

defining each of multiple question patterns as corresponding to one of the multiple list types;

obtaining multiple samples, each comprising a question, an introductory text, and a list;

for each of the samples:

matching the respective question with one of the question patterns, and

weakly labeling the respective list and the respective introductory text as belonging to the list type which corresponds to the matched question pattern; and

adapting a language model based on the samples and the weak labels of the lists and of the introductory texts.

3 . The computer-implemented method of claim 1 , further comprising training the second machine learning model by:

defining each of multiple introductory text patterns as corresponding to one of the multiple list types;

obtaining multiple samples, each comprising an introductory text and a list;

for each of the samples:

matching the respective introductory text with one of the introductory text patterns, and

weakly labeling the respective list as belonging to the list type which corresponds to the matched introductory text pattern; and

adapting a language model based on the samples and the weak labels of the lists.

4 . The computer-implemented method of claim 1 , wherein the multiple list types comprise at least some of:

a sequence, which is a list of interdependent items with a meaningful order;

a ranking, which is a list of independent items with a meaningful order;

a catalog, which is a list of independent items without a meaningful order, wherein each of the items is an optional answer to the question; and

an itemization, which is a list of independent items without a meaningful order, wherein all the items are necessary to answer the question.

5 . The computer-implemented method of claim 1 , wherein:

when only a subset of the items is being provided in (b), the method further comprises providing to the user a hyperlink to a document which contains all the items; and

when all the items are being provided in (b), the method further comprises providing to the user an indication that all the items are being provided.

6 . A system comprising:

(i) at least one hardware processor; and

(ii) a non-transitory computer-readable storage medium having program code embodied therewith, the program code executable by said at least one hardware processor to, automatically:

receive data comprising: a question provided by a user, a list that constitutes a direct answer to the question, and an introductory text to the list,

use a first machine learning model to classify the introductory text as redundant or nonredundant, based on at least one of the question and the list, wherein the first machine learning model is a language model,

use a second machine learning model to classify the list as belonging to a certain list type out of multiple list types, based on the list, wherein the second machine learning model is another language model,

provide to the user:

(a) the introductory text, only if the introductory text has been classified as nonredundant,

(b) all or only a subset of items of the list,

(c) an indication as to a number of non-provided items of the list or a number of all items of the list, if only a subset of the items is being provided in (b), and

(d) a description of the certain list type; and

train the second machine learning model by:

defining each of multiple question patterns as corresponding to one of the multiple list types;

obtaining multiple samples, each comprising a question, an introductory text, and a list;

for each of the samples:

matching the respective question with one of the question patterns, and

weakly labeling the respective list and the respective introductory text as belonging to the list type which corresponds to the matched question pattern; and

adapting a first language model based on the samples and the weak labels of the lists and of the introductory texts to form the second machine learning model.

7 . The system of claim 6 , wherein:

redundant introductory text is introductory text that conveys information which is included, explicitly or implicitly, in at least one of: the question and the list; and

nonredundant introductory text is introductory text that conveys information which is not included, explicitly or implicitly, in at least one of: the question and the list.

8 . The system of claim 7 , wherein the program code is further executable to train the first machine learning model by:

obtaining multiple samples, each comprising a question, a list, and an introductory text;

for each of the samples:

computing a similarity measure between (i) at least one of: the respective question and the respective list, and (ii) the respective introductory text, and

weakly labeling the respective introductory text as redundant or nonredundant, based on the similarity measure associated with the respective introductory text and on a predetermined similarity measure threshold; and

adapting a language model based on the samples and the weak labels of the introductory texts.

9 . The system of claim 6 , wherein the training of the second machine learning model further comprises:

defining each of multiple introductory text patterns as corresponding to one of the multiple list types;

obtaining multiple additional samples, each comprising an introductory text and a list;

for each of the additional samples:

matching the respective introductory text with one of the introductory text patterns, and

weakly labeling the respective list as belonging to the list type which corresponds to the matched introductory text pattern; and

further adapting the first language model based on the additional samples and the additional weak labels of the lists to help for the second machine learning model.

10 . The system of claim 6 , wherein the multiple list types comprise at least some of:

a sequence, which is a list of interdependent items with a meaningful order;

a ranking, which is a list of independent items with a meaningful order;

a catalog, which is a list of independent items without a meaningful order, wherein each of the items is an optional answer to the question; and

an itemization, which is a list of independent items without a meaningful order, wherein all the items are necessary to answer the question.

11 . The system of claim 6 , wherein:

when only a subset of the items is being provided in (b), the program code further is executable to automatically provide to the user a hyperlink to a document which contains all the items; and

when all the items are being provided in (b), the program code further is executable to automatically provide to the user an indication that all the items are being provided.

12 . A computer program product comprising a non-transitory computer-readable storage medium having program code embodied therewith, the program code executable by at least one hardware processor to, automatically:

receive data comprising: a question provided by a user, a list that constitutes a direct answer to the question, and an introductory text to the list;

use a first machine learning model to classify the introductory text as redundant or nonredundant, based on at least one of the question and the list, wherein the first machine learning model is a language model,

use a second machine learning model to classify the list as belonging to a certain list type out of multiple list types, based on the list, wherein the second machine learning model is another language model,

provide to the user:

(a) the introductory text, only if the introductory text has been classified as nonredundant,

(b) all or only a subset of items of the list,

(c) an indication as to a number of non-provided items of the list or a number of all items of the list, if only a subset of the items is being provided in (b), and

(d) a description of the certain list type; and

train the second machine learning model by:

defining each of multiple question patterns as corresponding to one of the multiple list types;

obtaining multiple samples, each comprising a question, an introductory text, and a list;

for each of the samples:

matching the respective question with one of the question patterns, and

weakly labeling the respective list and the respective introductory text as belonging to the list type which corresponds to the matched question pattern; and

adapting a first language model based on the samples and the weak labels of the lists and of the introductory texts to form the second machine learning model.

13 . The computer program product of claim 12 , wherein:

redundant introductory text is introductory text that conveys information which is included, explicitly or implicitly, in at least one of: the question and the list; and

nonredundant introductory text is introductory text that conveys information which is not included, explicitly or implicitly, in at least one of: the question and the list.

14 . The computer program product of claim 13 , wherein the program code is further executable to train the first machine learning model by:

obtaining multiple samples, each comprising a question, a list, and an introductory text;

for each of the samples:

computing a similarity measure between (i) at least one of: the respective question and the respective list, and (ii) the respective introductory text, and

weakly labeling the respective introductory text as redundant or nonredundant, based on the similarity measure associated with the respective introductory text and on a predetermined similarity measure threshold; and

adapting a language model based on the samples and the weak labels of the introductory texts.

15 . The computer program product of claim 12 , wherein the program code is further executable to further train the second machine learning model by:

defining each of multiple question patterns as corresponding to one of the multiple list types;

obtaining multiple additional samples, each comprising a question, an introductory text, and a list;

for each of the additional samples:

matching the respective question with one of the question patterns, and

weakly labeling the respective list and the respective introductory text as belonging to the list type which corresponds to the matched question pattern; and

further adapting the first language model based on the additional samples and the additional weak labels of the lists and of the introductory texts to help form the second machine learning model.

16 . The computer program product of claim 12 , wherein the multiple list types comprise at least some of:

a sequence, which is a list of interdependent items with a meaningful order;

a ranking, which is a list of independent items with a meaningful order;

a catalog, which is a list of independent items without a meaningful order, wherein each of the items is an optional answer to the question; and

an itemization, which is a list of independent items without a meaningful order, wherein all the items are necessary to answer the question.

Assignments (1)

Continuity (1)

References Cited (32)