Query response using media consumption history
View Patent ↗Methods, systems, and apparatus for receiving a natural language query of a user, and environmental data, identifying a media item based on the environmental data, determining an entity type based on the natural language query, selecting an entity associated with the media item that matches the entity type, selecting, from a media consumption database that identifies media items that have been indicated as consumed by the user, one or more media items that have been indicated as consumed by the user and that are associated with the selected entity, and providing a response to the query based on selecting the one or more media items that have been indicated as consumed by the user and that are associated with the selected entity.
1. A computer-implemented method comprising:
receiving (i) environmental audio data, and (ii) a natural language query from a user that requests identification of a media item that has previously been consumed by the user, wherein the natural language query includes one or more terms that reference an entity type of an entity associated with the environmental audio data, and does not include one or more terms that uniquely identify the entity associated with the environmental audio data;
identifying a particular media item that includes one or more acoustic features that match one or more corresponding acoustic features of the environmental audio data;
identifying, as the entity associated with the environmental audio data, a particular entity (i) that matches the entity type and (ii) that is associated with the particular media item;
selecting, from a media consumption database that identifies media items that are identified as having been previously consumed by the user, one or more previously consumed media items that are identified as having been previously consumed by the user and that are associated with the entity associated with the environmental audio data; and
providing a response to the natural language query that identifies one or more of the previously consumed media items, based on selecting the one or more previously consumed media items that are identified as having been previously consumed by the user and that are associated with the entity associated with the environmental audio data.
2. The method of claim 1 , wherein the environmental audio data is obtained by a microphone associated with a client device and wherein identifying the particular media item that includes one or more acoustic features that match one or more corresponding acoustic features of the environmental audio data comprises:
obtaining audio fingerprints of the environmental audio data;
comparing the audio fingerprints of the environmental audio data to audio fingerprints of one or more candidate media items; and
identifying, from among the one or more candidate media items, a particular media item that includes one or more audio fingerprints that match one or more corresponding audio fingerprints of the environmental audio data.
3. The method of claim 1 , comprising:
obtaining a transcription of the natural language query received from the user;
comparing one or more terms of the transcription of the natural language query received from the user to one or more keyword phrases that are associated with entity types; and
determining that the natural language query includes one or more terms that reference an entity type of an entity based on determining that one or more terms of the transcription of the natural language query received from the user match a particular keyword phrase associated with the entity type.
4. The method of claim 1 , wherein identifying, as the entity associated with the environmental audio data, the particular entity (i) that matches the entity type and (ii) that is associated with the particular media item comprises:
obtaining audio fingerprints of the environmental audio data;
comparing the audio fingerprints of the environmental audio data to audio fingerprints of the particular media item, wherein the audio fingerprints of the particular media item are associated with timestamps corresponding to segments of the particular media item;
identifying, based on comparing the audio fingerprints of the environmental audio data to audio fingerprints of the particular media item, a particular timestamp corresponding to a particular segment of the particular media item; and
identifying, as the entity associated with the environmental audio data, an entity that (i) is associated with the particular timestamp corresponding to the particular segment of the particular media item, and (ii) that corresponds to the entity type.
5. The method of claim 1 , comprising:
determining that the natural language query includes one or more terms that identify a particular characteristic associated with the consumption of the media items that are identified as having been previously consumed by the user;
identifying attributes corresponding to the particular characteristic, for one or more of the previously consumed media items that are identified as having been previously consumed by the user and that are associated with the entity associated with the environmental audio data; and
providing a response to the natural language query that identifies the attributes corresponding to the particular characteristic for one or more of the previously consumed media items that are identified as having been previously consumed by the user and that are associated with the entity associated with the environmental audio data.
6. The method of claim 1 , wherein providing the response to the natural language query comprises providing information for output that identifies one or more of the previously consumed media items that are identified as having been previously consumed by the user and that are associated with the entity associated with the environmental audio data.
7. The method of claim 1 , wherein providing the response to the natural language query comprises providing information for output that indicates whether the media consumption database identifies one or more previously consumed media items that are identified as having been previously consumed by the user and that are associated with the entity associated with the environmental audio data.
8. A system comprising:
one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:
receiving (i) environmental audio data, and (ii) a natural language query from a user that requests identification of a media item that has previously been consumed by the user, wherein the natural language query includes one or more terms that reference an entity type of an entity associated with the environmental audio data, and does not include one or more terms that uniquely identify the entity associated with the environmental audio data;
identifying a particular media item that includes one or more acoustic features that match one or more corresponding acoustic features of the environmental audio data;
identifying, as the entity associated with the environmental audio data, a particular entity (i) that matches the entity type and (ii) that is associated with the particular media item;
selecting, from a media consumption database that identifies media items that are identified as having been previously consumed by the user, one or more previously consumed media items that are identified as having been previously consumed by the user and that are associated with the entity associated with the environmental audio data; and
providing a response to the natural language query that identifies one or more of the previously consumed media items, based on selecting the one or more previously consumed media items that are identified as having been previously consumed by the user and that are associated with the entity associated with the environmental audio data.
9. The system of claim 8 , wherein the environmental audio data is obtained by a microphone associated with a client device and wherein identifying the particular media item that includes one or more acoustic features that match one or more corresponding acoustic features of the environmental audio data comprises:
obtaining audio fingerprints of the environmental audio data;
comparing the audio fingerprints of the environmental audio data to audio fingerprints of one or more candidate media items; and
identifying, from among the one or more candidate media items, a particular media item that includes one or more audio fingerprints that match one or more corresponding audio fingerprints of the environmental audio data.
10. The system of claim 8 , wherein the operations comprise:
obtaining a transcription of the natural language query received from the user;
comparing one or more terms of the transcription of the natural language query received from the user to one or more keyword phrases that are associated with entity types; and
determining that the natural language query includes one or more terms that reference an entity type of an entity based on determining that one or more terms of the transcription of the natural language query received from the user match a particular keyword phrase associated with the entity type.
11. The system of claim 8 , wherein identifying, as the entity associated with the environmental audio data, the particular entity (i) that matches the entity type and (ii) that is associated with the particular media item comprises:
obtaining audio fingerprints of the environmental audio data;
comparing the audio fingerprints of the environmental audio data to audio fingerprints of the particular media item, wherein the audio fingerprints of the particular media item are associated with timestamps corresponding to segments of the particular media item;
identifying, based on comparing the audio fingerprints of the environmental audio data to audio fingerprints of the particular media item, a particular timestamp corresponding to a particular segment of the particular media item; and
identifying, as the entity associated with the environmental audio data, an entity that (i) is associated with the particular timestamp corresponding to the particular segment of the particular media item, and (ii) that corresponds to the entity type.
12. The system of claim 8 , wherein the operations comprise:
determining that the natural language query includes one or more terms that identify a particular characteristic associated with the consumption of the media items that are identified as having been previously consumed by the user;
identifying attributes corresponding to the particular characteristic, for one or more of the previously consumed media items that are identified as having been previously consumed by the user and that are associated with the entity associated with the environmental audio data; and
providing a response to the natural language query that identifies the attributes corresponding to the particular characteristic for one or more of the previously consumed media items that are identified as having been previously consumed by the user and that are associated with the entity associated with the environmental audio data.
13. The system of claim 8 , wherein providing the response to the natural language query comprises providing information for output that identifies one or more of the previously consumed media items that are identified as having been previously consumed by the user and that are associated with the entity associated with the environmental audio data.
14. The system of claim 8 , wherein providing the response to the natural language query comprises providing information for output that indicates whether the media consumption database identifies one or more previously consumed media items that are identified as having been previously consumed by the user and that are associated with the entity associated with the environmental audio data.
15. A computer-readable storage device encoded with a computer program, the program comprising instructions that, if executed by one or more computers, cause the one or more computers to perform operations comprising:
receiving (i) environmental audio data, and (ii) a natural language query from a user that requests identification of a media item that has previously been consumed by the user, wherein the natural language query includes one or more terms that reference an entity type of an entity associated with the environmental audio data, and does not include one or more terms that uniquely identify the entity associated with the environmental audio data;
identifying a particular media item that includes one or more acoustic features that match one or more corresponding acoustic features of the environmental audio data;
identifying, as the entity associated with the environmental audio data, a particular entity (i) that matches the entity type and (ii) that is associated with the particular media item;
selecting, from a media consumption database that identifies media items that are identified as having been previously consumed by the user, one or more previously consumed media items that are identified as having been previously consumed by the user and that are associated with the entity associated with the environmental audio data; and
providing a response to the natural language query that identifies one or more of the previously consumed media items, based on selecting the one or more previously consumed media items that are identified as having been previously consumed by the user and that are associated with the entity associated with the environmental audio data.
16. The computer-readable device of claim 15 , wherein the environmental audio data is obtained by a microphone associated with a client device and wherein identifying the particular media item that includes one or more acoustic features that match one or more corresponding acoustic features of the environmental audio data comprises:
obtaining audio fingerprints of the environmental audio data;
comparing the audio fingerprints of the environmental audio data to audio fingerprints of one or more candidate media items; and
identifying, from among the one or more candidate media items, a particular media item that includes one or more audio fingerprints that match one or more corresponding audio fingerprints of the environmental audio data.
17. The computer-readable device of claim 15 , wherein the operations comprise:
obtaining a transcription of the natural language query received from the user;
comparing one or more terms of the transcription of the natural language query received from the user to one or more keyword phrases that are associated with entity types; and
determining that the natural language query includes one or more terms that reference an entity type of an entity based on determining that one or more terms of the transcription of the natural language query received from the user match a particular keyword phrase associated with the entity type.
18. The computer-readable device of claim 15 , wherein identifying, as the entity associated with the environmental audio data, the particular entity (i) that matches the entity type and (ii) that is associated with the particular media item comprises:
obtaining audio fingerprints of the environmental audio data;
comparing the audio fingerprints of the environmental audio data to audio fingerprints of the particular media item, wherein the audio fingerprints of the particular media item are associated with timestamps corresponding to segments of the particular media item;
identifying, based on comparing the audio fingerprints of the environmental audio data to audio fingerprints of the particular media item, a particular timestamp corresponding to a particular segment of the particular media item; and
identifying, as the entity associated with the environmental audio data, an entity that (i) is associated with the particular timestamp corresponding to the particular segment of the particular media item, and (ii) that corresponds to the entity type.
19. The computer-readable device of claim 15 , wherein the operations comprise:
determining that the natural language query includes one or more terms that identify a particular characteristic associated with the consumption of the media items that are identified as having been previously consumed by the user;
identifying attributes corresponding to the particular characteristic, for one or more of the previously consumed media items that are identified as having been previously consumed by the user and that are associated with the entity associated with the environmental audio data; and
providing a response to the natural language query that identifies the attributes corresponding to the particular characteristic for one or more of the previously consumed media items that are identified as having been previously consumed by the user and that are associated with the entity associated with the environmental audio data.
20. The computer-readable device of claim 15 , wherein providing the response to the natural language query comprises providing information for output that identifies one or more of the previously consumed media items that are identified as having been previously consumed by the user and that are associated with the entity associated with the environmental audio data.