Intelligent online personal assistant with multi-turn dialog based on visual search
Systems, methods, and computer program products for identifying a relevant candidate product in an electronic marketplace. Embodiments perform a visual similarity comparison between candidate product image visual content and input query image visual content, process formal and informal natural language user inputs, and coordinate aggregated past user interactions with the marketplace stored in a knowledge graph. Visually similar items and their corresponding product categories, aspects, and aspect values can determine suggested candidate products without discernible delay during a multi-turn user dialog. The user can then refine the search for the most relevant items available for purchase by providing responses to machine-generated prompts that are based on the initial search results from visual, voice, and/or text inputs. An intelligent online personal assistant can thus guide a user to the most relevant candidate product more efficiently than existing search tools.
1. A method comprising:
receiving, from a client device, an input query image and a non-image input;
assembling a ranked list of candidate products based on a visual similarity measure between the input query image and each candidate product in the ranked list;
providing, for each candidate product in the ranked list, a corresponding resolved knowledge graph information to a knowledge graph comprising aggregate historical electronic marketplace user interaction information;
detecting, based on the knowledge graph, a mismatch between the ranked list of candidate products and the non-image input;
outputting an instruction to cause display of a user prompt on a graphical user interface of a client device requesting specified user input in a multi-turn dialog;
receiving, from the client device via the multi-turn dialog, the specified user input;
selecting, based on the specified user input, unresolved knowledge graph information in the knowledge graph for the one or more candidate products from the ranked list;
repeating the previous operations until a stopping condition occurs; and
outputting a filtered list of candidate products selected by the user via the multi-turn dialog.
2. The method of claim 1 , further comprising:
generating, using a neural network, an image signature for each of a plurality of candidate product images and the input query image, the respective image signatures semantically representing respective visual content features.
3. The method of claim 2 , further comprising:
calculating, using one or more processors, the visual similarity measure between each candidate product image and the input query image based on the respective corresponding image signatures.
4. The method of claim 2 , wherein the image signature comprises a binary hash of a number of floating point numbers semantically representing the visual content features.
5. The method of claim 1 , wherein the user prompt comprises at least one of:
a number of selectable candidate product images for a subset of highest ranked candidate products in the ranked list, and a number of questions requesting natural language input data describing the unresolved knowledge graph information.
6. The method of claim 1 , wherein the resolved knowledge graph information is determined from both visual searches and non-visual searches.
7. The method of claim 1 , wherein the stopping condition comprises at least one of:
a detection of a user intent change, a submission of an entirely new query, and a final user selection of a candidate product.
8. A non-transitory computer-readable storage medium having embedded therein a set of instructions which, when executed by one or more processors of a computer, causes the computer to execute operations comprising:
receiving, from a client device, an input query image and a non-image input;
assembling a ranked list of candidate products based on a visual similarity measure between the input query image and each candidate product in the ranked list;
providing, for each candidate product in the ranked list, a corresponding resolved knowledge graph information to a knowledge graph comprising aggregate historical electronic marketplace user interaction information;
detecting, based on the knowledge graph, a mismatch between the ranked list of candidate products and the non-image input;
outputting an instruction to cause display of a user prompt on a graphical user interface of a client device requesting specified user input in a multi-turn dialog;
receiving, from the client device via the multi-turn dialog, the specified user input;
selecting, based on the specified user input, unresolved knowledge graph information in the knowledge graph for the one or more candidate products from the ranked list;
repeating the previous operations until a stopping condition occurs; and
outputting a filtered list of candidate products selected by the user via the multi-turn dialog.
9. The medium of claim 8 , further comprising:
generating, using a neural network, an image signature for each of a plurality of candidate product images and the input query image, the respective image signatures semantically representing respective visual content features.
10. The medium of claim 9 , further comprising:
calculating, using one or more processors, the visual similarity measure between each candidate product image and the input query image based on the respective corresponding image signatures.
11. The medium of claim 9 , wherein the image signature comprises a binary hash of a number of floating point numbers semantically representing the visual content features.
12. The medium of claim 8 , wherein the user prompt comprises at least one of:
a number of selectable candidate product images for a subset of highest ranked candidate products in the ranked list, and a number of questions requesting natural language input data describing the unresolved knowledge graph information.
13. The medium of claim 8 , wherein the resolved knowledge graph information is determined from both visual searches and non-visual searches.
14. The medium of claim 8 , wherein the stopping condition comprises at least one of:
a detection of a user intent change, a submission of an entirely new query, and a final user selection of a candidate product.
15. A system comprising:
a memory comprising instructions; and
one or more hardware-based computer processors, wherein the instructions, when executed by the one or more computer processors, cause the one or more computer processors to perform operations comprising:
receiving, from a client device, an input query image and a non-image input;
assembling a ranked list of candidate products based on a visual similarity measure between the input query image and each candidate product in the ranked list;
providing, for each candidate product in the ranked list, a corresponding resolved knowledge graph information to a knowledge graph comprising aggregate historical electronic marketplace user interaction information;
detecting, based on the knowledge graph, a mismatch between the ranked list of candidate products and the non-image input;
outputting an instruction to cause display of a user prompt on a graphical user interface of a client device requesting specified user input in a multi-turn dialog;
receiving, from the client device via the multi-turn dialog, the specified user input;
selecting, based on the specified user input, unresolved knowledge graph information in the knowledge graph for the one or more candidate products from the ranked list;
repeating the previous operations until a stopping condition occurs; and
outputting a filtered list of candidate products selected by the user via the multi-turn dialog.
16. The system of claim 15 , further comprising:
generating, using a neural network, an image signature for each of a plurality of candidate product images and the input query image, the respective image signatures semantically representing respective visual content features.
17. The system of claim 16 , wherein the image signature comprises a binary hash of a number of floating point numbers semantically representing the visual content features.
18. The system of claim 15 , wherein the user prompt comprises at least one of:
a number of selectable candidate product images for a subset of highest ranked candidate products in the ranked list, and a number of questions requesting natural language input data describing the unresolved knowledge graph information.
19. The system of claim 15 , wherein the resolved knowledge graph information is determined from both visual searches and non-visual searches.
20. The system of claim 15 , wherein the stopping condition comprises at least one of:
a detection of a user intent change, a submission of an entirely new query, and a final user selection of a candidate product.