Multimodal entity and coreference resolution for assistant systems

1. A method comprising:

accessing visual data from a client system associated with a user, wherein the visual data comprises images portraying one or more objects;

receiving, from the client system, a user request, wherein the user request comprises a coreference to a target object;

resolving the coreference to the target object from among the one or more objects;

resolving the target object to a specific entity; and

sending, to the client system, instructions for providing a response to the user request, wherein the response comprises attribute information about the specific entity.

2. The method of claim 1 , further comprising:

analyzing, by a computer vision module, the visual data to identify the one or more objects portrayed in the images;

parsing, by a natural-language understanding (NLU) module, the user request to identify an intent of the user request and references to one or more of the objects; and

updating a dialog state to include the identified objects and the identified references.

3. The method of claim 2 , further comprising:

receiving gesture or gaze information from the user; and

updating the dialog state to include the received gesture or gaze information.

4. The method of claim 2 , wherein resolving the coreference to the target object from among the one or more objects comprises combining additional information with the dialog state.

5. The method of claim 4 , wherein the additional information comprises a level of interestingness of one or more of the objects.

6. The method of claim 4 , wherein the additional information comprises information of a user profile of the user.

7. The method of claim 4 , wherein the additional information comprises a model of past interactions with one or more of the objects by other users.

8. The method of claim 1 , further comprising:

analyzing the visual data to identify the one or more objects portrayed in the images;

assigning respective object identifiers to one or more of the identified objects; and

storing one or more of the object identifiers as entities in a dialog state tracker.

9. The method of claim 8 , wherein resolving the coreference to the target object from among the one or more objects comprises:

analyzing the dialog state tracker to identify one or more of the entities; and

identifying a result set of one or more of the identified entities based on a context of the user.

10. The method of claim 9 , wherein resolving the target object to a specific entity comprises:

selecting the specific entity from among the identified entities of the result set.

11. The method of claim 8 , further comprising:

in response to receiving the user request, generating, by a scene understanding engine, relational information between identified objects.

12. The method of claim 1 , further comprising:

accessing a knowledge graph; and

retrieving the attribute information about the specific entity from the knowledge graph.

13. The method of claim 1 , wherein one or more of the objects of the visual data are virtual objects in a virtual reality environment displayed to the user via the client system.

14. The method of claim 1 , further comprising:

receiving, from the client system, a second user request to perform a task with respect to the specific entity;

executing the task; and

sending, the client system, instructions for providing a second response to the user comprising results of the executed task.

15. The method of claim 1 , wherein resolving the coreference to the target object from among the one or more objects comprises:

analyzing, by a scene understanding engine, the visual data to identify the one or more objects portrayed in the images; and

identifying the target object from among the identified objects based on its position within a field of view of the visual data.

16. The method of claim 1 , wherein resolving the coreference to the target object from among the one or more objects comprises:

sending, to the client system, a prompt for the user to confirm the target object; and

receiving, from the client system, a user response to the prompt identifying the target object.

17. The method of claim 1 , further comprising:

accessing a set of images from the visual data; and

selecting one or more images from among the set of images based on a timing of the user request, wherein the selected images portray the target object.

18. The method of claim 1 , further comprising:

accessing a memory graph associated with the user to retrieve context data associated with the user; and

comparing the attribute information about the specific entity with the context data,

wherein the response further comprises a result of the comparison.

19. One or more computer-readable non-transitory storage media embodying software that is operable when executed to:

access visual data from a client system associated with a user, wherein the visual data comprises images portraying one or more objects;

receive, from the client system, a user request, wherein the user request comprises a coreference to a target object;

resolve the coreference to the target object from among the one or more objects;

resolve the target object to a specific entity; and

send, to the client system, instructions for providing a response to the user request, wherein the response comprises attribute information about the specific entity.

20. A system comprising: one or more processors; and a non-transitory memory coupled to the processors comprising instructions executable by the processors, the processors operable when executing the instructions to:

access visual data from a client system associated with a user, wherein the visual data comprises images portraying one or more objects;

receive, from the client system, a user request, wherein the user request comprises a coreference to a target object;

resolve the coreference to the target object from among the one or more objects;

resolve the target object to a specific entity; and

send, to the client system, instructions for providing a response to the user request, wherein the response comprises attribute information about the specific entity.