Automatic generation of clarification questions for conversational search

1. A computer-implemented method comprising the following automated steps:

obtaining multiple dialogs between users and agents, wherein:

each of the dialogs comprises messages exchanged between one of the users and one of the agents, and

one of the messages of each of the dialogs comprises a reference to a solution document provided by the respective agent;

for each of the dialogs, operating a search engine to retrieve a text passage, relevant to at least one of the messages of the respective dialog, from the respective solution document; and

training a machine learning language model to generate a new clarification question given at least one new message and a multiple new text passages, wherein the training is based on a training set which comprises, for each of the dialogs: said at least one of the messages of the respective dialog, and the text passage retrieved for the respective dialog.

2. The computer-implemented method of claim 1 , wherein:

said at least one of the messages comprises the respective clarification question of each of the dialogs, such that:

for each of the dialogs, the text passage retrieved by the search engine is relevant at least to the respective clarification question, and

the training set comprises, for each of the dialogs, the respective clarification question.

3. The computer-implemented method of claim 2 , wherein:

said at least one of the messages further comprises an answer of the respective user to the respective clarification question of each of the dialogs, such that:

for each of the dialogs, the text passage retrieved by the search engine is relevant also to the respective answer, and

the training set further comprises, for each of the dialogs, the respective answer.

4. The computer-implemented method of claim 1 , wherein:

said at least one of the messages comprises the respective clarification question of each of the dialogs, and all messages preceding the respective clarification question in each of the dialogs, such that:

for each of the dialogs, the text passage retrieved by the search engine is relevant to the respective clarification question and all the messages preceding the respective clarification question, and

the training set comprises, for each of the dialogs, the respective clarification question and all the messages preceding the respective clarification question.

5. The computer-implemented method of claim 1 , further comprising the following automated steps:

receiving said at least one new message from a new user;

operating a search engine to retrieve new text passages relevant to said at least one new message from a corpus containing the solution documents or other solution documents;

separately applying the trained machine learning language model to each of multiple sets of input, each of the sets of input comprising said at least one new message and a different one of the new text passages, to generate a candidate clarification question for each of the sets of input; and

presenting the new user with a selected clarification question out of the candidate clarification questions.

6. The computer-implemented method of claim 5 , further comprising the following automated steps:

calculating a relevancy score for each of the candidate clarification questions,

wherein the selected clarification question presented to the user is the candidate clarification question having the highest score.

7. The computer-implemented method of claim 5 , further comprising the following automated steps:

receiving, in response to the presentation of the selected clarification question, a new answer from the user; and

using at least one of the clarification question and the new answer to retrieve one or more additional text passages from the corpus, to satisfy an information need of the user as reflected by said at least one new message and the new answer.

8. The computer-implemented method of claim 7 , further comprising the following automated steps:

repeating:

(a) said receiving, said operating, said separately applying, and said presenting of claim 5 , and

(b) said receiving of claim 7 ,

with the new answer instead of or in addition to said at least one new message; and

ceasing said repeating upon receiving an indication that an information need of the user is satisfied.

9. The computer-implemented method of claim 1 , wherein said automated steps are executed by at least one hardware processor of the computer in which the method is implemented.

10. A system comprising:

(a) at least one hardware processor; and

(b) a non-transitory computer-readable storage medium having program code embodied therewith, the program code executable by said at least one hardware processor to, automatically:

obtain multiple dialogs between users and agents, wherein:

each of the dialogs comprises messages exchanged between one of the users and one of the agents, and

one of the messages of each of the dialogs comprises a reference to a solution document provided by the respective agent,

for each of the dialogs, operate a search engine to retrieve a text passage, relevant to at least one of the messages of the respective dialog, from the respective solution document, and

train a machine learning language model to generate a new clarification question given at least one new message and a multiple new text passages, wherein the training is based on a training set which comprises, for each of the dialogs: said at least one of the messages of the respective dialog, and the text passage retrieved for the respective dialog.

11. The system of claim 10 , wherein:

said at least one of the messages comprises the respective clarification question of each of the dialogs, such that:

for each of the dialogs, the text passage retrieved by the search engine is relevant at least to the respective clarification question, and

the training set comprises, for each of the dialogs, the respective clarification question.

12. The system of claim 11 , wherein:

said at least one of the messages further comprises an answer of the respective user to the respective clarification question of each of the dialogs, such that:

for each of the dialogs, the text passage retrieved by the search engine is relevant also to the respective answer, and

the training set further comprises, for each of the dialogs, the respective answer.

13. The system of claim 10 , wherein:

said at least one of the messages comprises the respective clarification question of each of the dialogs, and all messages preceding the respective clarification question in each of the dialogs, such that:

for each of the dialogs, the text passage retrieved by the search engine is relevant to the respective clarification question and all the messages preceding the respective clarification question, and

the training set comprises, for each of the dialogs, the respective clarification question and all the messages preceding the respective clarification question.

14. The system of claim 10 , wherein the program code is further executable by said at least one hardware processor to:

receive said at least one new message from a new user;

operate a search engine to retrieve new text passages relevant to said at least one new message from a corpus containing the solution documents or other solution documents;

separately apply the trained machine learning language model to each of multiple sets of input, each of the sets of input comprising said at least one new message and a different one of the new text passages, to generate a candidate clarification question for each of the sets of input; and

present the new user with a selected clarification question out of the candidate clarification questions.

15. The system of claim 14 , wherein the program code is further executable by said at least one hardware processor to:

calculate a relevancy score for each of the candidate clarification questions,

wherein the selected clarification question presented to the user is the candidate clarification question having the highest score.

16. The system of claim 14 , wherein the program code is further executable by said at least one hardware processor to:

receive, in response to the presentation of the selected clarification question, a new answer from the user; and

use at least one of the clarification question and the new answer to retrieve one or more additional text passages from the corpus, to satisfy an information need of the user as reflected by said at least one new message and the new answer.

17. The system of claim 16 , wherein the program code is further executable by said at least one hardware processor to:

repeat:

(a) said receive, said operate, said separately apply, and said present of claim 14 , and

(b) said receive of claim 16 ,

with the new answer instead of or in addition to said at least one new message; and

cease said repeating upon receiving an indication that an information need of the user is satisfied.

18. A computer program product comprising a non-transitory computer-readable storage medium having program code embodied therewith, the program code executable by at least one hardware processor to, automatically:

obtain multiple dialogs between users and agents, wherein:

each of the dialogs comprises messages exchanged between one of the users and one of the agents, and

one of the messages of each of the dialogs comprises a reference to a solution document provided by the respective agent;

for each of the dialogs, operate a search engine to retrieve a text passage, relevant to at least one of the messages of the respective dialog, from the respective solution document; and

train a machine learning language model to generate a new clarification question given at least one new message and a multiple new text passages, wherein the training is based on a training set which comprises, for each of the dialogs: said at least one of the messages of the respective dialog, and the text passage retrieved for the respective dialog.

19. The computer program product of claim 18 , wherein:

said at least one of the messages comprises the respective clarification question of each of the dialogs, such that:

for each of the dialogs, the text passage retrieved by the search engine is relevant at least to the respective clarification question, and

the training set comprises, for each of the dialogs, the respective clarification question.

20. The computer program product of claim 19 , wherein:

said at least one of the messages further comprises an answer of the respective user to the respective clarification question of each of the dialogs, such that:

for each of the dialogs, the text passage retrieved by the search engine is relevant also to the respective answer, and

the training set further comprises, for each of the dialogs, the respective answer.