Personalized, real-time audio processing
An apparatus and method for real-time audio processing employs a gaze detection sensor to detect a direction of a user's gaze and output a gaze signal corresponding to the detected direction of the user's gaze. A digital signal processing unit responds to a plurality of signals corresponding to a plurality of sounds received at the apparatus, and the determined direction of gaze to identify a signal of interest from the plurality of signals using the gaze signal. The signal of interest is processed for output to the user. In embodiments, a microphone array provides the plurality of signals. An imaging sensor may work with either the microphone array or the gaze detection sensor to identify the signal of interest.
1. An apparatus, comprising:
a sensor configured to detect a direction of a gaze of a user and output a first gaze signal corresponding to the detected direction of the gaze of the user;
a digital signal processing unit configured to:
receive the first gaze signal from the sensor;
receive a second gaze signal;
receive a third gaze signal;
determine a priority for each of the first gaze signal, the second gaze signal, and the third gaze signal;
select at least one of the first gaze signal, the second gaze signal, and the third gaze signal based on the priority;
identify, in an electronic data storage, one or more available actions that correspond to the selected at least one of the first gaze signal, the second gaze signal, and the third gaze signal;
select an action from the one or more available actions; and
execute a set of computer-readable instructions that correspond to the action.
2. The apparatus of claim 1 , further comprising a microphone positioned relative to a position of the sensor to provide information indicating directionality of one or more sounds that correspond to the second gaze signal.
3. The apparatus of claim 2 , wherein the second gaze signal is selected based on a determination that the detected direction of the gaze of the user that corresponds to the first gaze signal is in a different direction as compared to the one or more sounds that correspond to the second gaze signal, wherein the action is selected based on the detected direction of the one or more sounds.
4. The apparatus of claim 1 , wherein the first gaze signal is associated with a first priority and the second gaze signal is associated with a second priority, wherein when selecting one of the first gaze signal, the second gaze signal, and the third gaze signal based on the priority, the digital signal processing unit is configured to:
determine that the third gaze signal corresponds to an unwanted signal that corresponds to a third priority that has a lower priority than the first priority for the first gaze signal and the second priority for the second gaze signal; and
deemphasize the third gaze signal in view of the third priority having a lower priority than the first priority and the second priority.
5. The apparatus of claim 4 , wherein deemphasizing the third gaze signal includes removing audio from the third gaze signal.
6. The apparatus of claim 1 , wherein the digital signal processing unit is further configured to determine that a person is speaking in a direction of the user based on one or more of the first gaze signal, the second gaze signal, and the third gaze signal, wherein the action is selected based on a determination that the person is speaking in the direction of the user.
7. The apparatus of claim 1 , further comprising an imaging sensor configured to detect an image, wherein the action is selected based on the image.
8. The apparatus of claim 1 , wherein the third gaze signal includes an identification signal from a sound source, the identification signal including metadata that identifies either the sound source itself, or a person associated with the sound source.
9. A method, comprising:
receiving a first gaze signal from a sensor;
receiving a second gaze signal;
receiving a third gaze signal;
determining a priority for each of the first gaze signal, the second gaze signal, and the third gaze signal;
selecting at least one of the first gaze signal, the second gaze signal, and the third gaze signal based on the priority;
identifying, in an electronic data storage, one or more available actions that correspond to the selected at least one of the first gaze signal, the second gaze signal, and the third gaze signal;
selecting an action from the one or more available actions; and
executing a set of computer-readable instructions that correspond to the action.
10. The method of claim 9 , further comprising receiving information indicating directionality of one or more sounds that correspond to the second gaze signal.
11. The method of claim 10 , wherein the second gaze signal is selected based on a determination that a detected direction of a gaze of a user that corresponds to the first gaze signal is in a different direction as compared to the one or more sounds that correspond to the second gaze signal, wherein the action is selected based on the detected direction of the one or more sounds.
12. The method of claim 9 , wherein the first gaze signal is associated with a first priority and the second gaze signal is associated with a second priority, wherein selecting one of the first gaze signal, the second gaze signal, and the third gaze signal based on the priority includes:
determining that the third gaze signal corresponds to an unwanted signal that corresponds to a third priority that has a lower priority than the first priority for the first gaze signal and the second priority for the second gaze signal; and
deemphasizing the third gaze signal in view of the third priority having a lower priority than the first priority and the second priority.
13. The method of claim 12 , wherein deemphasizing the third gaze signal includes removing audio from the third gaze signal.
14. The method of claim 9 , further comprising determining that a person is speaking in a direction of a user based on one or more of the first gaze signal, the second gaze signal and the third gaze signal, wherein the action is selected based on a determination that the person is speaking in the direction of the user.
15. The method of claim 9 , further comprising detecting an image, wherein the action is selected based on the image.
16. The method of claim 9 , wherein the third gaze signal includes an identification signal from a sound source, the identification signal including metadata that identifies either the sound source itself, or a person associated with the sound source.
17. A non-transitory machine-readable medium having stored thereon machine-readable instructions executable to cause a machine to perform operations comprising:
receive a first gaze signal from a sensor;
receive a second gaze signal;
receive a third gaze signal;
determine a priority for each of the first gaze signal, the second gaze signal, and the third gaze signal;
select at least one of the first gaze signal, the second gaze signal, and the third gaze signal based on the priority;
identify, in an electronic data storage, one or more available actions that correspond to the selected at least one of the first gaze signal, the second gaze signal, and the third gaze signal;
select an action from the one or more available actions; and
execute the action.
18. The non-transitory machine-readable medium of claim 17 , wherein the second gaze signal is selected based on a determination that a detected direction of a gaze of a user that corresponds to the first gaze signal is in a different direction as compared to one or more sounds that correspond to the second gaze signal, wherein the action is selected based on the detected direction of the one or more sounds.
19. The non-transitory machine-readable medium of claim 17 , wherein the first gaze signal is associated with a first priority and the second gaze signal is associated with a second priority, wherein the operation to select one of the first gaze signal, the second gaze signal, and the third gaze signal based on the priority includes:
determine that the third gaze signal corresponds to an unwanted signal that corresponds to a third priority that has a lower priority than the first priority for the first gaze signal and the second priority for the second gaze signal; and
deemphasize the third gaze signal in view of the third priority having a lower priority than the first priority and the second priority.
20. The non-transitory machine-readable medium of claim 17 , the operations further include determine that a person is speaking in a direction of a user based on one or more of the first gaze signal, the second gaze signal and the third gaze signal, wherein the action is selected based on a determination that the person is speaking in the direction of the user.