Personalized, real-time audio processing
An apparatus and method for real-time audio processing employs a gaze detection sensor to detect a direction of a user's gaze and output a gaze signal corresponding to the detected direction of the user's gaze. A digital signal processing unit responds to a plurality of signals corresponding to a plurality of sounds received at the apparatus, and the determined direction of gaze to identify a signal of interest from the plurality of signals using the gaze signal. The signal of interest is processed for output to the user. In embodiments, a microphone array provides the plurality of signals. An imaging sensor may work with either the microphone array or the gaze detection sensor to identify the signal of interest.
1. A computer-implemented method comprising:
detecting, by a device associated with a user, a direction of a gaze of the user;
based at least in part on determining that the direction of the gaze of the user is different from directions of a plurality of sound signals:
receiving one or more image signals from an environment surrounding the user; and
detecting, by the device based on the one or more image signals, a speaker speaking in a direction of the user; and
performing, by the device, a noise cancelling algorithm on a first sound signal, of the plurality of sound signals, arriving from a direction different from a direction of the speaker.
2. The computer-implemented method of claim 1 , further comprising:
determining, from the plurality of sound signals, a second sound signal that corresponds to speech of the speaker; and
emphasizing the second sound signal.
3. The computer-implemented method of claim 2 , wherein emphasizing the second sound signal comprises causing output of the second sound signal using one or more acoustical techniques to amplify the second sound signal.
4. The computer-implemented method of claim 1 , wherein performing the noise cancelling algorithm on the first sound signal comprises generating and outputting an inverted frequency profile of the first sound signal.
5. The computer-implemented method of claim 1 , wherein performing the noise cancelling algorithm on the first sound signal is based at least in part on determining that an audio source associated with the first sound signal is located a threshold number of degrees from a determined centerline between the speaker and the user.
6. The computer-implemented method of claim 1 , wherein determining that the direction of the gaze of the user is different from the directions of the plurality of sound signals comprises:
detecting, using one or more sensors, movement of:
a position of eyes of the user, or
one or more microphones placed on a body of the user; and
comparing directions associated with the detected movement to the directions of the plurality of sound signals.
7. The computer-implemented method of claim 1 , further comprising:
determining, based on comparing the direction of the gaze of the user to the directions of the plurality of sound signals, which of a first sensor or a second sensor to prioritize for use in determining which sound signal, of the plurality of sound signals, to emphasize.
8. The computer-implemented method of claim 1 , further comprising:
receiving one or more gaze signals from a gaze detection sensor, wherein the one or more gaze signals are, by default, used to determine which sound signal, of the plurality of sound signals, to emphasize;
receiving the one or more image signals from an image sensor; and
based at least in part on determining that the direction of the gaze of the user is different from the directions of the plurality of sound signals, ignoring the one or more gaze signals and using the one or more image signals to determine which of the plurality of sound signals to emphasize.
9. A device associated with a user comprising:
at least one processor; and
memory storing instructions that, when executed by the at least one processor, cause the device to:
detect a direction of a gaze of the user;
based at least in part on determining that the direction of the gaze of the user is different from directions of a plurality of sound signals:
receive one or more image signals from an environment surrounding the user; and
detect, based on the one or more image signals, a speaker speaking in a direction of the user; and
perform a noise cancelling algorithm on a first sound signal, of the plurality of sound signals, arriving from a direction different from a direction of the speaker.
10. The device of claim 9 , wherein the instructions, when executed by the at least one processor, further cause the device to:
determine, from the plurality of sound signals, a second sound signal that corresponds to speech of the speaker; and
emphasize the second sound signal.
11. The device of claim 9 , wherein, to perform the noise cancelling algorithm on the first sound signal, the instructions, when executed by the at least one processor, cause the device to generate and output an inverted frequency profile of the first sound signal.
12. The device of claim 9 , wherein the instructions, when executed by the at least one processor, cause the device to perform the noise cancelling algorithm on the first sound signal based at least in part on a determination that an audio source associated with the first sound signal is located a threshold number of degrees from a determined centerline between the speaker and the user.
13. The device of claim 9 , wherein, to determine that the direction of the gaze of the user is different from the directions of the plurality of sound signals, the instructions, when executed by the at least one processor, cause the device to:
detect, using one or more sensors, movement of:
a position of eyes of the user, or
one or more microphones placed on a body of the user; and
compare directions associated with the detected movement to the directions of the plurality of sound signals.
14. The device of claim 9 , wherein the instructions, when executed by the at least one processor, further cause the device to:
determine, based on comparing the direction of the gaze of the user to the directions of the plurality of sound signals, which of a first sensor or a second sensor to prioritize for use in determining which of the plurality of sound signals to emphasize.
15. The device of claim 9 , wherein the instructions, when executed by the at least one processor, further cause the device to:
receive one or more gaze signals from a gaze detection sensor, wherein the one or more gaze signals are, by default, used to determine which of the plurality of sound signals to emphasize;
receive the one or more image signals from an image sensor; and
based at least in part on determining that the direction of the gaze of the user is different from the directions of the plurality of sound signals, ignore the one or more gaze signals and using the one or more image signals to determine which of the plurality of sound signals to emphasize.
16. A non-transitory, computer-readable medium storing instructions that, when executed by at least one processor of a device, cause the device to perform operations comprising:
detecting a direction of a gaze of a user associated with the device;
based at least in part on determining that the direction of the gaze of the user is different from directions of a plurality of sound signals:
receiving one or more image signals from an environment surrounding the user; and
detecting, based on the one or more image signals, a speaker speaking in a direction of the user; and
performing a noise cancelling algorithm on a first sound signal, of the plurality of sound signals, arriving from a direction different from a direction of the speaker.
17. The non-transitory, computer-readable medium of claim 16 , wherein the instructions, when executed by the at least one processor, further cause the device to perform operations comprising:
determining, from the plurality of sound signals, a second sound signal that corresponds to speech of the speaker; and
emphasizing the second sound signal.
18. The non-transitory, computer-readable medium of claim 16 , wherein performing the noise cancelling algorithm on the first sound signal comprises generating and outputting an inverted frequency profile of the first sound signal.
19. The non-transitory, computer-readable medium of claim 16 , wherein performing the noise cancelling algorithm on the first sound signal is based at least in part on determining that an audio source associated with the first sound signal is located a threshold number of degrees from a determined centerline between the speaker and the user.
20. The non-transitory, computer-readable medium of claim 16 , wherein the instructions, when executed by the at least one processor, further cause the device to perform operations comprising:
receiving one or more gaze signals from a gaze detection sensor, wherein the one or more gaze signals are, by default, used to determine which of the plurality of sound signals to emphasize;
receiving the one or more image signals from an image sensor; and
based at least in part on determining that the direction of the gaze of the user is different from the directions of the plurality of sound signals, ignoring the one or more gaze signals and using the one or more image signals to determine which of the plurality of sound signals to emphasize.