Query response using media consumption history
Methods, systems, and apparatus for receiving a natural language query of a user, and environmental data, identifying a media item based on the environmental data, determining an entity type based on the natural language query, selecting an entity associated with the media item that matches the entity type, selecting, from a media consumption database that identifies media items that have been indicated as consumed by the user, one or more media items that have been indicated as consumed by the user and that are associated with the selected entity, and providing a response to the query based on selecting the one or more media items that have been indicated as consumed by the user and that are associated with the selected entity.
1. A computer-implemented method comprising:
receiving (i) a natural language query submitted by a user that requests information relating to a context associated with a prior consumption of a media item, and (ii) ambient environmental data obtained from an environment of a user;
identifying a particular media item based on detecting a match between one or more features of the ambient environmental data obtained from the environment of the user and one or more features of the particular media item;
determining that the particular media item is identified in a media consumption database that identifies media items that are identified as having been previously consumed by the user;
accessing, at the media consumption database, information that (i) identifies a context associated with the previous consumption of the particular media item by the user, and that is (ii) identified as being responsive to the natural language query submitted by the user; and
providing a response to the natural language query submitted by the user that includes at least a portion of the information accessed at the media consumption database that (i) identifies the context associated with the previous consumption of the particular media item by the user, and that is (ii) identified as being responsive to the natural language query submitted by the user.
2. The method of claim 1 , further comprising:
selecting a media item type based on the natural language query submitted by the user; and
identifying the particular media item based on (i) detecting the match between one or more features of the ambient environmental data obtained from the environment of the user and one or more features of the particular media item, and (ii) determining that the particular media item is associated with a media item type that corresponds to the selected media item type.
3. The method of claim 2 , wherein selecting the media item type based on the natural language query submitted by the user comprises:
obtaining a transcription of the natural language query submitted by the user;
comparing the transcription of the natural language query submitted by the user to one or more keyword phrases that are associated with media item types; and
selecting the media item type based on determining that the transcription of the natural language query submitted by the user matches a particular keyword phrase that is associated with the selected media item type.
4. The method of claim 1 , wherein identifying the particular media item based on detecting the match between one or more features of the ambient environmental data obtained from the environment of the user and one or more features of the particular media item comprises:
obtaining audio fingerprints of the ambient environmental data obtained from the environment of the user;
comparing the audio fingerprints of the ambient environmental data obtained from the environment of the user to audio fingerprints of the particular media item; and
determining that the audio fingerprints of the ambient environmental data obtained from the environment of the user match audio fingerprints of the particular media item.
5. The method of claim 1 , wherein accessing the information that (i) identifies a context associated with the previous consumption of the particular media item by the user, and that is (ii) identified as being responsive to the natural language query submitted by the user comprises:
obtaining a transcription of the natural language query submitted by the user;
determining, based on comparing the transcription of the natural language query submitted by the user to one or more keyword phrases that are associated with one or more corresponding information types that each relate to a context associated with consumption of a media item by a user, that the transcription of the natural language query submitted by the user matches a particular keyword phrase associated with a particular information type; and
accessing, at the media consumption database, the information that (i) identifies characteristics associated with the consumption of the particular media item by the user that correspond to the particular information type.
6. The method of claim 1 , wherein accessing the information that (i) identifies a context associated with the previous consumption of the particular media item by the user, and that is (ii) identified as being responsive to the natural language query submitted by the user comprises accessing, at the media consumption database, information that indicates at least one of a location, a time, a date, or a method of consumption of the particular media item by the user.
7. The method of claim 1 , wherein the ambient environmental data obtained from the environment of the user is one of ambient audio data obtained from the environment of the user, image data obtained from the environment of the user, or video data obtained from the environment of the user.
8. A system comprising:
one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:
receiving (i) a natural language query submitted by a user that requests information relating to a context associated with a prior consumption of a media item, and (ii) ambient environmental data obtained from an environment of a user;
identifying a particular media item based on detecting a match between one or more features of the ambient environmental data obtained from the environment of the user and one or more features of the particular media item;
determining that the particular media item is identified in a media consumption database that identifies media items that are identified as having been previously consumed by the user;
accessing, at the media consumption database, information that (i) identifies a context associated with the previous consumption of the particular media item by the user, and that is (ii) identified as being responsive to the natural language query submitted by the user; and
providing a response to the natural language query submitted by the user that includes at least a portion of the information accessed at the media consumption database that (i) identifies the context associated with the previous consumption of the particular media item by the user, and that is (ii) identified as being responsive to the natural language query submitted by the user.
9. The system of claim 8 , wherein the operations further comprise:
selecting a media item type based on the natural language query submitted by the user; and
identifying the particular media item based on (i) detecting the match between one or more features of the ambient environmental data obtained from the environment of the user and one or more features of the particular media item, and (ii) determining that the particular media item is associated with a media item type that corresponds to the selected media item type.
10. The system of claim 9 , wherein selecting the media item type based on the natural language query submitted by the user comprises:
obtaining a transcription of the natural language query submitted by the user;
comparing the transcription of the natural language query submitted by the user to one or more keyword phrases that are associated with media item types; and
selecting the media item type based on determining that the transcription of the natural language query submitted by the user matches a particular keyword phrase that is associated with the selected media item type.
11. The system of claim 8 , wherein identifying the particular media item based on detecting the match between one or more features of the ambient environmental data obtained from the environment of the user and one or more features of the particular media item comprises:
obtaining audio fingerprints of the ambient environmental data obtained from the environment of the user;
comparing the audio fingerprints of the ambient environmental data obtained from the environment of the user to audio fingerprints of the particular media item; and
determining that the audio fingerprints of the ambient environmental data obtained from the environment of the user match audio fingerprints of the particular media item.
12. The system of claim 8 , wherein accessing the information that (i) identifies a context associated with the previous consumption of the particular media item by the user, and that is (ii) identified as being responsive to the natural language query submitted by the user comprises:
obtaining a transcription of the natural language query submitted by the user;
determining, based on comparing the transcription of the natural language query submitted by the user to one or more keyword phrases that are associated with one or more corresponding information types that each relate to a context associated with consumption of a media item by a user, that the transcription of the natural language query submitted by the user matches a particular keyword phrase associated with a particular information type; and
accessing, at the media consumption database, information that (i) identifies characteristics associated with the consumption of the particular media item by the user that correspond to the particular information type.
13. The system of claim 8 , wherein accessing the information that (i) identifies a context associated with the previous consumption of the particular media item by the user, and that is (ii) identified as being responsive to the natural language query submitted by the user comprises accessing, at the media consumption database, information that indicates at least one of a location, a time, a date, or a method of consumption of the particular media item by the user.
14. The system of claim 8 , wherein the ambient environmental data obtained from the environment of the user is one of ambient audio data obtained from the environment of the user, image data obtained from the environment of the user, or video data obtained from the environment of the user.
15. A computer-readable storage device encoded with a computer program, the program comprising instructions that, if executed by one or more computers, cause the one or more computers to perform operations comprising:
receiving (i) a natural language query submitted by a user that requests information relating to a context associated with a prior consumption of a media item, and (ii) ambient environmental data obtained from an environment of a user;
identifying a particular media item based on detecting a match between one or more features of the ambient environmental data obtained from the environment of the user and one or more features of the particular media item;
determining that the particular media item is identified in a media consumption database that identifies media items that are identified as having been previously consumed by the user;
accessing, at the media consumption database, information that (i) identifies a context associated with the previous consumption of the particular media item by the user, and that is (ii) identified as being responsive to the natural language query submitted by the user; and
providing a response to the natural language query submitted by the user that includes at least a portion of the information accessed at the media consumption database that (i) identifies the context associated with the previous consumption of the particular media item by the user, and that is (ii) identified as being responsive to the natural language query submitted by the user.
16. The computer-readable device of claim 15 , the operations further comprising:
selecting a media item type based on the natural language query submitted by the user; and
identifying the particular media item based on (i) detecting the match between one or more features of the ambient environmental data obtained from the environment of the user and one or more features of the particular media item, and (ii) determining that the particular media item is associated with a media item type that corresponds to the selected media item type.
17. The computer-readable device of claim 16 , wherein selecting the media item type based on the natural language query submitted by the user comprises:
obtaining a transcription of the natural language query submitted by the user;
comparing the transcription of the natural language query submitted by the user to one or more keyword phrases that are associated with media item types; and
selecting the media item type based on determining that the transcription of the natural language query submitted by the user matches a particular keyword phrase that is associated with the selected media item type.
18. The computer-readable device of claim 15 , wherein identifying the particular media item based on detecting the match between one or more features of the ambient environmental data obtained from the environment of the user and one or more features of the particular media item comprises:
obtaining audio fingerprints of the ambient environmental data obtained from the environment of the user;
comparing the audio fingerprints of the ambient environmental data obtained from the environment of the user to audio fingerprints of the particular media item; and
determining that the audio fingerprints of the ambient environmental data obtained from the environment of the user match audio fingerprints of the particular media item.
19. The computer-readable device of claim 15 , wherein accessing the information that (i) identifies a context associated with the previous consumption of the particular media item by the user, and that is (ii) identified as being responsive to the natural language query submitted by the user comprises:
obtaining a transcription of the natural language query submitted by the user;
determining, based on comparing the transcription of the natural language query submitted by the user to one or more keyword phrases that are associated with one or more corresponding information types that each relate to a context associated with consumption of a media item by a user, that the transcription of the natural language query submitted by the user matches a particular keyword phrase associated with a particular information type; and
accessing, at the media consumption database, the information that (i) identifies characteristics associated with the consumption of the particular media item by the user that correspond to the particular information type.
20. The computer-readable device of claim 15 , wherein accessing the information that (i) identifies a context associated with the previous consumption of the particular media item by the user, and that is (ii) identified as being responsive to the natural language query submitted by the user comprises accessing, at the media consumption database, information that indicates at least one of a location, a time, a date, or a method of consumption of the particular media item by the user.