Spatial audio head tracker
Example technologies described herein relate to spatial audio on wearable playback devices, such as headphone and earbuds. Such technologies may include forms of binaural rendering of audio, which is played back on the wearable playback devices. These technologies may create or enhance an immersive listening experience which, to the listener, does not sound like you are listening on a wearable device, but are instead listening to a well-tuned, well-place, three-dimensional audio system of loudspeakers in a particular listening environment, such as the listener's actual living room.
1 . A system comprising:
at least one processor; and
at least one non-transitory computer-readable medium comprising program instructions that are executable by the at least one processor such that the system is configured to:
send, via a wireless network interface, instructions to play a localization signal on a non-wearable playback device at a first location in a listening environment;
while a wearable playback device is donned by a user at a second location in the listening environment, capture, via microphones of the wearable playback device; recordings of the localization signal, wherein the microphones comprise first microphones carried on a first earcup of the wearable playback device and second microphones carried on a second earcup of the wearable playback device;
determine respective times-of-arrival of the localization signal at the microphones;
determine pairwise angles-of-arrival for each pair of the microphones that are on opposite earcups such that each pair includes one of the first microphones and one of the second microphones;
determine an ensemble angle-of-arrival based on the determined pairwise angles-of-arrival;
add the determined ensemble angle-of-arrival to a buffer representing estimated user gaze directions;
determine head tracking data based on a rotation of samples from an inertial measurement unit (IMU) to a reference frame centered on the estimated gaze directions stored in the buffer;
during binaural rendering of audio via a binaural renderer, update binaural rendering filters with the determined head tracking data; and
cause the wearable playback device to play back the binaural-rendered audio to simulate playback from at least one virtual source in the listening environment, wherein the at least virtual source comprises the non-wearable playback device at the first location.
2 . The system of claim 1 , wherein the at least one non-transitory computer readable medium further comprises program instructions that are executable by the at least one processor such that the system is configured to determine the respective times-of-arrival of the localization signal at the microphones comprise program instructions that are executable by the at least one processor such that the system is configured to:
estimate impulse responses for each recording of the localization signal; and
detect respective onsets of the localization signal in the estimated impulse responses as the respective times-of-arrival.
3 . The system of claim 1 , wherein the at least one non-transitory computer readable medium further comprises program instructions that are executable by the at least one processor such that the system is configured to determine pairwise angles-of-arrival for each pair of the microphones comprise program instructions that are executable by the at least one processor such that the system is configured to:
determine pairwise time-differences-of-arrival between each pair of microphones; and
determine pairwise angles-of-arrival based on the respective pairwise time-differences-of-arrival.
4 . The system of claim 3 , wherein the at least one non-transitory computer readable medium further comprises program instructions that are executable by the at least one processor such that the system is configured to determine pairwise angles-of-arrival based on the respective pairwise time-differences-of-arrival comprise program instructions that are executable by the at least one processor such that the system is configured to:
adjust geometry of the pairwise angles-of-arrival determination from a default inter-aural distance to an estimated inter-aural distance.
5 . The system of claim 3 , wherein the at least one non-transitory computer readable medium further comprises program instructions that are executable by the at least one processor such that the system is configured to determine pairwise angles-of-arrival for each pair of the microphones comprise program instructions that are executable by the at least one processor such that the system is configured to:
apply one or more discriminators to reject recordings that are outside of thresholds corresponding to the respective discriminator.
6 . The system of claim 1 , wherein the at least one non-transitory computer readable medium further comprises program instructions that are executable by the at least one processor such that the system is configured to determine an ensemble angle-of-arrival based on the determined pairwise angles-of-arrival comprise program instructions that are executable by the at least one processor such that the system is configured to:
apply per-pair weightings to the determined pairwise angles-of-arrival based on at least one confidence metric; and
determine the ensemble angle-of-arrival based on the weighted pairwise angles-of-arrival.
7 . The system of claim 1 , wherein the at least one non-transitory computer readable medium further comprises program instructions that are executable by the at least one processor such that the system is configured to add the determined ensemble angle-of-arrival to the buffer comprise program instructions that are executable by the at least one processor such that the system is configured to:
add the determined ensemble angle-of-arrival to the buffer when the determined ensemble angle-of-arrival is not an outlier relative to additional ensemble angles-of-arrival stored in the buffer.
8 . The system of claim 1 , wherein the at least one non-transitory computer readable medium further comprises program instructions that are executable by the at least one processor such that the system is configured to:
detect that a threshold period of time has elapsed since user gaze direction was last estimated, wherein the program instructions that are executable by the at least one processor such that the system is configured to send the instructions to play the localization signal on the non-wearable playback device comprise program instructions that are executable by the at least one processor such that the system is configured to:
based on the detection that the threshold period of time has elapsed since user gaze direction was last estimated, send the instructions to play the localization signal on the non-wearable playback device.
9 . The system of claim 1 , wherein the at least one non-transitory computer readable medium further comprises program instructions that are executable by the at least one processor such that the system is configured to:
detect that the user has changed position, wherein the program instructions that are executable by the at least one processor such that the system is configured to send the instructions to play the localization signal on the non-wearable playback device comprise program instructions that are executable by the at least one processor such that the system is configured to:
based on the detection that the user has changed position, send the instructions to play the localization signal on the non-wearable playback device.
10 . The system of claim 1 , wherein a first set of estimated gaze directions comprises the estimated gaze directions stored in the buffer, wherein a second set of estimated gaze directions comprises additional estimated gaze directions based on samples from the IMU, and wherein the at least one non-transitory computer readable medium further comprises program instructions that are executable by the at least one processor such that the system is configured to:
based on at least one confidence metric, select the first set of estimated gaze directions from among two or more sets of estimated gaze directions, wherein the two or more sets comprise the first set and the second set.
11 . At least one non-transitory computer-readable medium comprising program instructions that are executable by at least one processor such that a system is configured to:
send, via a wireless network interface, instructions to play a localization signal on a non-wearable playback device at a first location in a listening environment;
while a wearable playback device is donned by a user at a second location in the listening environment, capture, via microphones of the wearable playback device; recordings of the localization signal, wherein the microphones comprise first microphones carried on a first earcup of the wearable playback device and second microphones carried on a second earcup of the wearable playback device;
determine respective times-of-arrival of the localization signal at the microphones;
determine pairwise angles-of-arrival for each pair of the microphones that are on opposite earcups such that each pair includes one of the first microphones and one of the second microphones;
determine an ensemble angle-of-arrival based on the determined pairwise angles-of-arrival;
add the determined ensemble angle-of-arrival to a buffer representing estimated user gaze directions;
determine head tracking data based on a rotation of samples from an inertial measurement unit (IMU) to a reference frame centered on the estimated gaze directions stored in the buffer;
during binaural rendering of audio via a binaural renderer, update binaural rendering filters with the determined head tracking data; and
cause the wearable playback device to play back the binaural-rendered audio to simulate playback from at least one virtual source in the listening environment, wherein the at least virtual source comprises the non-wearable playback device at the first location.
12 . The at least one non-transitory computer-readable medium of claim 11 , wherein the program instructions that are executable by the at least one processor such that the system is configured to determine the respective times-of-arrival of the localization signal at the microphones comprise program instructions that are executable by the at least one processor such that the system is configured to:
estimate impulse responses for each recording of the localization signal; and
detect respective onsets of the localization signal in the estimated impulse responses as the respective times-of-arrival.
13 . The at least one non-transitory computer-readable medium of claim 11 , wherein the program instructions that are executable by the at least one processor such that the system is configured to determine pairwise angles-of-arrival for each pair of the microphones comprise program instructions that are executable by the at least one processor such that the system is configured to:
determine pairwise time-differences-of-arrival between each pair of microphones; and
determine pairwise angles-of-arrival based on the respective pairwise time-differences-of-arrival.
14 . The at least one non-transitory computer-readable medium of claim 13 , wherein the program instructions that are executable by the at least one processor such that the system is configured to determine pairwise angles-of-arrival based on the respective pairwise time-differences-of-arrival comprise program instructions that are executable by the at least one processor such that the system is configured to:
adjust geometry of the pairwise angles-of-arrival determination from a default inter-aural distance to an estimated inter-aural distance.
15 . The at least one non-transitory computer-readable medium of claim 13 , wherein the program instructions that are executable by the at least one processor such that the system is configured to determine pairwise angles-of-arrival for each pair of the microphones comprise program instructions that are executable by the at least one processor such that the system is configured to:
apply one or more discriminators to reject recordings that are outside of thresholds corresponding to the respective discriminator.
16 . The at least one non-transitory computer-readable medium of claim 11 , wherein the program instructions that are executable by the at least one processor such that the system is configured to determine an ensemble angle-of-arrival based on the determined pairwise angles-of-arrival comprise program instructions that are executable by the at least one processor such that the system is configured to:
apply per-pair weightings to the determined pairwise angles-of-arrival based on at least one confidence metric; and
determine the ensemble angle-of-arrival based on the weighted pairwise angles-of-arrival.
17 . The at least one non-transitory computer-readable medium of claim 11 , wherein the program instructions that are executable by the at least one processor such that the system is configured to add the determined ensemble angle-of-arrival to the buffer comprise program instructions that are executable by the at least one processor such that the system is configured to:
add the determined ensemble angle-of-arrival to the buffer when the determined ensemble angle-of-arrival is not an outlier relative to additional ensemble angles-of-arrival stored in the buffer.
18 . The at least one non-transitory computer-readable medium of claim 11 , wherein the at least one non-transitory computer readable medium further comprises program instructions that are executable by the at least one processor such that the system is configured to:
detect that a threshold period of time has elapsed since user gaze direction was last estimated, wherein the program instructions that are executable by the at least one processor such that the system is configured to send the instructions to play the localization signal on the non-wearable playback device comprise program instructions that are executable by the at least one processor such that the system is configured to:
based on the detection that the threshold period of time has elapsed since user gaze direction was last estimated, send the instructions to play the localization signal on the non-wearable playback device.
19 . The at least one non-transitory computer-readable medium of claim 11 , wherein the at least one non-transitory computer readable medium further comprises program instructions that are executable by the at least one processor such that the system is configured to:
detect that the user has changed position, wherein the program instructions that are executable by the at least one processor such that the system is configured to send the instructions to play the localization signal on the non-wearable playback device comprise program instructions that are executable by the at least one processor such that the system is configured to:
based on the detection that the user has changed position, send the instructions to play the localization signal on the non-wearable playback device.
20 . The at least one non-transitory computer-readable medium of claim 11 , wherein a first set of estimated gaze directions comprises the estimated gaze directions stored in the buffer, wherein a second set of estimated gaze directions comprises additional estimated gaze directions based on samples from the IMU, and wherein the at least one non-transitory computer readable medium further comprises program instructions that are executable by the at least one processor such that the system is configured to:
based on at least one confidence metric, select the first set of estimated gaze directions from among two or more sets of estimated gaze directions, wherein the two or more sets comprise the first set and the second set.
21 . A method comprising:
sending, via a wireless network interface, instructions to play a localization signal on a non-wearable playback device at a first location in a listening environment;
while a wearable playback device is donned by a user at a second location in the listening environment, capturing, via microphones of the wearable playback device; recordings of the localization signal, wherein the microphones comprise first microphones carried on a first earcup of the wearable playback device and second microphones carried on a second earcup of the wearable playback device;
determining respective times-of-arrival of the localization signal at the microphones;
determining pairwise angles-of-arrival for each pair of the microphones that are on opposite earcups such that each pair includes one of the first microphones and one of the second microphones;
determining an ensemble angle-of-arrival based on the determined pairwise angles-of-arrival;
adding the determined ensemble angle-of-arrival to a buffer representing estimated user gaze directions;
determining head tracking data based on a rotation of samples from an inertial measurement unit (IMU) to a reference frame centered on the estimated gaze directions stored in the buffer;
during binaural rendering of audio via a binaural renderer, updating binaural rendering filters with the determined head tracking data; and
causing the wearable playback device to play back the binaural-rendered audio to simulate playback from at least one virtual source in the listening environment, wherein the at least virtual source comprises the non-wearable playback device at the first location.