Semantic cluster formation in deep learning intelligent assistants
Enhanced techniques and circuitry are presented herein for providing responses to user questions from among digital documentation sources spanning various documentation formats, versions, and types. One example includes a method comprising receiving a user question directed to subject having a documentation corpus, determining a set of passages of the documentation corpus related to the user question, ranking the set of passages according to relevance to the user question, forming semantic clusters comprising sentences extracted from ranked ones of the set of passages according to sentence similarity, and providing a response to the user question based at least on a selected semantic cluster.
1. A method comprising:
in response to a question directed to a subject of a documentation corpus that describes operations or features of a software platform or service, extracting a textual answer comprising one or more sentences from one or more top ranked passages of the documentation corpus;
extracting images comprising screenshots of user interfaces of the software platform or service from one or more pages of the documentation corpus from which the textual answer was extracted;
identifying, from the images, one or more selected images within a threshold similarity to the textual answer; and
providing the textual answer and the or more selected images as a multimodal answer to the question.
2. The method of claim 1 , wherein the screenshots of the user interfaces of the software platform or service extracted from the documentation corpus visually represent user interface tabs or dialog boxes, and wherein identifying the one or more selected images identifies one or more of the screenshots of the user interface tabs or the dialog boxes within the threshold similarity to the textual answer.
3. The method of claim 1 , wherein identifying the one or more selected images comprises quantifying similarity between the textual answer and one or more of a heading, caption, title, or optical character recognition (OCR) results associated with at least one image of the images.
4. The method of claim 1 , wherein identifying the one or more selected images comprises quantifying similarity between the textual answer and a sentence encoding of a heading or title associated with least one image of the images.
5. The method of claim 1 , wherein identifying the one or more selected images comprises, for least one image of the images, using optical character recognition (OCR) to generate representations of extracted words in the at least one image, and quantifying similarity between the textual answer and the representations of the extracted words.
6. The method of claim 1 , wherein identifying the one or more selected images comprises, for least one image of the images, quantifying similarity between the textual answer and a weighted average of embeddings of words that are optically extracted from the at least one image and weighted by corresponding inverse term frequency in the documentation corpus.
7. The method of claim 1 , wherein identifying the one or more selected images comprises, for least one image of the images:
generating a plurality of scores quantifying similarity to the textual answer;
generating a combined score based on the plurality of scores; and
determining to whether to include the least one image in the one or more selected images based on the combined score.
8. One or more computer readable storage media storing computer-useable instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations comprising:
extracting, in response to a question and from one or more pages of a documentation corpus that describes operations or features of a software platform or service, a textual answer comprising one or more sentences from one or more top ranked passages of the documentation corpus;
extracting images comprising screenshots of user interfaces of the software platform or service from the one or more pages from which the textual answer was extracted;
identifying, from the images, one or more selected images within a threshold similarity to the textual answer; and
triggering presentation the textual answer and the or more selected images as a multimodal answer to the question.
9. The one or more computer readable storage media of claim 8 , wherein the screenshots of the user interfaces of the software platform or service extracted from the documentation corpus visually represent user interface tabs or dialog boxes, and wherein identifying the one or more selected images identifies one or more of the screenshots of the user interface tabs or the dialog boxes within the threshold similarity to the textual answer.
10. The one or more computer readable storage media of claim 8 , wherein identifying the one or more selected images comprises quantifying similarity between the textual answer and one or more of a heading, caption, title, or optical character recognition (OCR) results associated with least one image of the images.
11. The one or more computer readable storage media of claim 8 , wherein identifying the one or more selected images comprises quantifying similarity between the textual answer and a sentence encoding of a heading or title associated with least one image of the images.
12. The one or more computer readable storage media of claim 8 , wherein identifying the one or more selected images comprises, for least one image of the images, using optical character recognition (OCR) to generate representations of extracted words in the least one image, and quantifying similarity between the textual answer and the representations of the extracted words.
13. The one or more computer readable storage media of claim 8 , wherein identifying the one or more selected images comprises, for at least one image of the images, quantifying similarity between the textual answer and a weighted average of embeddings of words that are optically extracted from the at least one image and weighted by corresponding inverse term frequency in the documentation corpus.
14. The one or more computer readable storage media of claim 8 , wherein identifying the one or more selected images comprises, for least one image of the images:
generating a plurality of scores quantifying similarity to the textual answer;
generating a combined score based on the plurality of scores; and
determining to whether to include the least one image in the one or more selected images based on the combined score.
15. A computer system comprising one or more processors and memory configured to provide computer program instructions to the one or more processors, the computer program instructions comprising:
a response generator configured to extract, in response to a question directed to a subject of a documentation corpus that describes operations or features of a software platform or service, a textual answer comprising one or more sentences from one or more top ranked passages of the documentation corpus; and
an image embellishment element configured to:
extract images comprising screenshots of user interfaces of the software platform or service from one or more pages of the documentation corpus from which the textual answer was extracted; and
identify, from the images, one or more selected images within a threshold similarity to the textual answer,
wherein the response generator is configured to provide the textual answer and the or more selected images as a multimodal answer to the question.
16. The computer system of claim 15 , wherein the screenshots of the user interfaces of the software platform or service extracted from the documentation corpus visually represent user interface tabs or dialog boxes, and wherein identifying the one or more selected images identifies one or more of the screenshots of the user interface tabs or the dialog boxes within the threshold similarity to the textual answer.
17. The computer system of claim 15 , wherein the image embellishment element is configured to identify the one or more selected images based on quantifying similarity between the textual answer and one or more of a heading, caption, title, or optical character recognition (OCR) results associated with least one image of the images.
18. The computer system of claim 15 , wherein the image embellishment element is configured to identify the one or more selected images based on quantifying similarity between the textual answer and a sentence encoding of a heading or title associated with least one image of the images.
19. The computer system of claim 15 , wherein the image embellishment element is configured to identify the one or more selected images based on, for least one image of the images, using optical character recognition (OCR) to generate representations of extracted words in the least one image, and quantifying similarity between the textual answer and the representations of the extracted words.
20. The computer system of claim 15 , wherein the image embellishment element is configured to identify the one or more selected images based on, for at least one image of the images, quantifying similarity between the textual answer and a weighted average of embeddings of words that are optically extracted from the at least one image and weighted by corresponding inverse term frequency in the documentation corpus.