Proactive actions based on audio and body movement
Various implementations disclosed herein include devices, systems, and methods that determine that a user is interested in audio content by determining that a movement (e.g., a user's head bob) has a time-based relationship with detected audio content (e.g., the beat of music playing in the background). Some implementations involve obtaining first sensor data and second sensor data corresponding to a physical environment, the first sensor data corresponding to audio in the physical environment and the second sensor data corresponding to a body movement in the physical environment. A time-based relationship between one or more elements of the audio and one or more aspects of the body movement is identified based on the first sensor data and the second sensor data. An interest in content of the audio is identified based on identifying the time-based relationship. Various actions may be performed proactively based on identifying the interest in the content.
1. A method comprising,
at an electronic device having a processor:
obtaining first sensor data and second sensor data corresponding to a physical environment, the first sensor data corresponding to audio in the physical environment and the second sensor data corresponding to a body movement in the physical environment;
selectively switching to different power states on the electronic device based on multiple triggers identified at the electronic device, where selectively switching through the different power states comprises:
detecting, based on the second sensor data, that the body movement corresponds to a type of body movement indicative of user interest in music;
based on detecting that the body movement corresponds to the type of body movement indicative of user interest in music, selectively triggering performance of audio analysis to determine if music is playing based on the first sensor data;
determining that music is playing based on the audio analysis;
based on determining that music is playing, selectively triggering performance of a comparison of audio elements with one or more aspects of the body movement, based on the first sensor data and the second sensor data;
identifying a time-based relationship between one or more elements of the audio and one or more aspects of the body movement via at least the comparison;
based on identifying the time-based relationship, selectively triggering performance of a source attribute identification process; and
identifying an interest in content of the audio based at least on the source attribute identification process;
determining to wait to provide one or more features based on a user state corresponding to the user being busy; and
providing one or more features based on identifying the interest in the content, wherein a timing of providing the one or more features comprises waiting based on the user state corresponding to the user being busy.
2. The method of claim 1 further comprising, based on identifying the interest in the content, presenting an identification of the content on the electronic device.
3. The method of claim 1 further comprising, based on identifying the interest in the content, presenting text corresponding to words in the content.
4. The method of claim 1 further comprising, based on identifying the interest in the content presenting a selectable option for:
replaying the content;
continuing to experience the content after leaving the physical environment;
purchasing the content;
downloading the content; or
adding the content to a playlist.
5. The method of claim 1 further comprising, based on identifying the interest in the content, identifying a characteristic of the content and identifying additional content based on the identified characteristic.
6. The method of claim 1 further comprising determining to limit providing features associated with the content based on determining the user state.
7. The method of claim 1 , wherein identifying the interest in the content is further based on determining that a voice in the physical environment is singing along with the content.
8. The method of claim 1 , wherein identifying the interest in the content is further based on an identified gaze direction corresponding to a device that is producing the audio or a direction relative to a head of the user corresponding to mental state indicative of user interest.
9. The method of claim 1 , wherein identifying the interest in the content is further based on an identified facial expression.
10. The method of claim 1 , wherein identifying the time-based relationship further comprises determining that a timing of a repeating body motion of the body movement matches a timing of a beat or rhythm of the content.
11. The method of claim 1 , wherein identifying the time-based relationship further comprises determining that lip movement of the body movement matches words of the content and identifying the interest in the content of the audio is based on identifying from the time-based relationship that the user is singing along or lip syncing to the words of the content.
12. The method of claim 1 , wherein identifying the time-based relationship further comprises determining that the body movement is a reaction to an event in the content.
13. The method of claim 1 , wherein the second sensor data corresponding to the body movement comprises:
image sensor data in which a portion of a body moves over time; or
motion sensor data from a motion sensor attached to a portion of the body.
14. The method of claim 1 further comprising identifying that multiple persons are interested in the content of the audio based on identifying time-based relationships using audio and body movement sensor data from the multiple persons in the physical environment.
15. A system comprising:
a non-transitory computer-readable storage medium; and
one or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises program instructions that, when executed on the one or more processors, cause the system to perform operations comprising:
obtaining first sensor data and second sensor data corresponding to a physical environment, the first sensor data corresponding to audio in the physical environment and the second sensor data corresponding to a body movement in the physical environment;
selectively switching to different power states on an electronic device based on multiple triggers identified at the electronic device, where selectively switching through the different power states comprises:
detecting, based on the second sensor data, that the body movement corresponds to a type of body movement indicative of user interest in music;
based on detecting that the body movement corresponds to the type of body movement indicative of user interest in music, selectively triggering performance of audio analysis to determine if music is playing based on the first sensor data;
determining that music is playing based on the audio analysis;
based on determining that music is playing, selectively triggering performance of a comparison of audio elements with one or more aspects of the body movement, based on the first sensor data and the second sensor data;
identifying a time-based relationship between one or more elements of the audio and one or more aspects of the body movement via at least the comparison;
based on identifying the time-based relationship, selectively triggering performance of a source attribute identification process; and
identifying an interest in content of the audio based at least on the source attribute identification process;
determining to wait to provide one or more features based on a user state corresponding to the user being busy; and
providing one or more features based on identifying the interest in the content, wherein a timing of providing the one or more features comprises waiting based on the user state corresponding to the user being busy.
16. The system of claim 15 , wherein the operations further comprise, based on identifying the interest in the content:
presenting an identification of the content on an electronic device;
presenting text corresponding to words in the content;
replaying the content;
continuing to experience the content after leaving the physical environment;
purchasing the content;
downloading the content; or
adding the content to a playlist.
17. The system of claim 15 , wherein identifying the interest in the content is further based on an identified gaze direction detected using images obtained via an image sensor.
18. A non-transitory computer-readable storage medium storing program instructions executable via one or more processors to perform operations comprising:
obtaining first sensor data and second sensor data corresponding to a physical environment, the first sensor data corresponding to audio in the physical environment and the second sensor data corresponding to a body movement in the physical environment;
selectively switching to different power states on an electronic device based on multiple triggers identified at the electronic device, where selectively switching through the different power states comprises:
detecting, based on the second sensor data, that the body movement corresponds to a type of body movement indicative of user interest in music;
based on detecting that the body movement corresponds to the type of body movement indicative of user interest in music, selectively triggering performance of audio analysis to determine if music is playing based on the first sensor data;
determining that music is playing based on the audio analysis;
based on determining that music is playing, selectively triggering performance of a comparison of audio elements with one or more aspects of the body movement, based on the first sensor data and the second sensor data;
identifying a time-based relationship between one or more elements of the audio and one or more aspects of the body movement via at least the comparison;
based on identifying the time-based relationship, selectively triggering performance of a source attribute identification process; and
identifying an interest in content of the audio based at least on the source attribute identification process;
determining to wait to provide one or more features based on a user state corresponding to the user being busy; and
providing one or more features based on identifying the interest in the content, wherein a timing of providing the one or more features comprises waiting based on the user state corresponding to the user being busy.