Spatial audio head tracker
Example technologies described herein relate to spatial audio on wearable playback devices, such as headphone and earbuds. Such technologies may include forms of binaural rendering of audio, which is played back on the wearable playback devices. These technologies may create or enhance an immersive listening experience which, to the listener, does not sound like you are listening on a wearable device, but are instead listening to a well-tuned, well-place, three-dimensional audio system of loudspeakers in a particular listening environment, such as the listener's actual living room.
1 . A system comprising:
a network interface;
at least one processor; and
at least one non-transitory computer-readable medium comprising program instructions that are executable by the at least one processor such that the system is configured to:
receive a command to play back audio on a wearable playback device;
after the wearable playback device is donned by a user, determine, via a first gaze direction estimator, first preliminary estimates of user gaze direction, the first preliminary estimates based on one or more time-of-flight measurements of at least one signal propagating from a non-wearable playback device at a first location in a listening environment to the wearable playback device at a second location in the listening environment;
determine, via a second gaze direction estimator, second preliminary estimates of user gaze direction, the second preliminary estimates based on first samples from an inertial measurement unit (IMU);
estimate gaze directions based on fusion of the first gaze direction estimates and the second preliminary estimates;
determine head tracking data based on a rotation of second samples from the IMU to a reference frame centered on the estimated gaze directions;
during binaural rendering of the audio via a binaural renderer, update binaural rendering filters with the determined head tracking data; and
cause the wearable playback device to play back the binaural-rendered audio to simulate playback from at least one virtual source in the listening environment.
2 . The system of claim 1 , wherein the program instructions that are executable by the at least one processor such that the system is configured to estimate the gaze directions based on fusion of the first preliminary estimates and the second preliminary estimates comprise program instructions that are executable by the at least one processor such that the system is configured to:
based on at least one confidence metric, select the estimated gaze directions from among (a) the first preliminary gaze estimates and (b) the second preliminary gaze estimates.
3 . The system of claim 2 , wherein the at least one confidence metric comprises a length of time elapsed since a previous time of flight measurement; and wherein the program instructions that are executable by the at least one processor such that the system is configured to select the estimated gaze directions comprise program instructions that are executable by the at least one processor such that the system is configured to:
select the first preliminary gaze estimates as the estimated gaze directions when the length of time elapsed since the previous time of flight measurement is less than a threshold; and
select the second preliminary gaze estimates as the estimated gaze directions when the length of time elapsed since the previous time of flight measurement is greater than the threshold.
4 . The system of claim 2 , wherein the at least one virtual source comprises the first location when the first preliminary gaze estimates are selected, and wherein the at least one virtual source comprise a third location when the second preliminary gaze estimates are selected.
5 . The system of claim 1 , wherein the program instructions that are executable by the at least one processor such that the system is configured to estimate the gaze directions based on fusion of the first preliminary estimates and the second preliminary estimates comprise program instructions that are executable by the at least one processor such that the system is configured to:
based on at least one confidence metric, apply weightings to at least one of the first preliminary gaze estimates and the second preliminary gaze estimates; and
combine the weighted first preliminary gaze estimates and the second preliminary gaze estimates into the estimated gaze directions.
6 . The system of claim 5 , wherein the at least one confidence metric comprises a length of time elapsed since a previous time of flight measurement; and wherein the program instructions that are executable by the at least one processor such that the system is configured to apply weightings to at least one of the first preliminary gaze estimates and the second preliminary gaze estimates comprise program instructions that are executable by the at least one processor such that the system is configured to:
apply a relatively higher weighting to the first preliminary gaze estimates than the second preliminary gaze estimates when the length of time elapsed since the previous time of flight measurement is less than a threshold length of time; and
apply a relatively lower weighting to the first preliminary gaze estimates than the second preliminary gaze estimates when the length of time elapsed since the previous time of flight measurement is greater than the threshold length of time.
7 . The system of claim 5 , wherein the at least one confidence metric comprises a condition that the user has moved since a previous time of flight measurement; and wherein the program instructions that are executable by the at least one processor such that the system is configured to apply weightings to at least one of the first preliminary gaze estimates and the second preliminary gaze estimates comprise program instructions that are executable by the at least one processor such that the system is configured to:
apply a relatively lower weighting to the first preliminary gaze estimates than the second preliminary gaze estimates when the condition that the user has moved since the previous time of flight measurement is present; and
apply a relatively higher weighting to the first preliminary gaze estimates than the second preliminary gaze estimates when the condition that the user has moved since the previous time of flight measurement is not present.
8 . The system of claim 5 , wherein the at least one confidence metric comprises an IMU drift estimate; and wherein the program instructions that are executable by the at least one processor such that the system is configured to apply weightings to at least one of the first preliminary gaze estimates and the second preliminary gaze estimates comprise program instructions that are executable by the at least one processor such that the system is configured to:
apply a relatively lower weighting to the second preliminary gaze estimates than the first preliminary gaze estimates when the IMU drift estimate is greater than an IMU drift threshold; and
apply a relatively higher weighting to the second preliminary gaze estimates than the first preliminary gaze estimates when the IMU drift estimate is less than the IMU drift threshold.
9 . The system of claim 1 , wherein the program instructions that are executable by the at least one processor such that the system is configured to determine the first preliminary estimates of the user gaze direction comprise program instructions that are executable by the at least one processor such that the system is configured to:
capture, via at least one first microphone and at least one second microphone, a signal emitted by at least one audio transducer of the non-wearable playback device;
estimate respective times-of-arrival at the at least one first microphone and the at least one second microphone;
estimate pairwise angle-of-arrivals from the estimated times-of-arrival; and
calculate the first preliminary estimates from the estimated pairwise angle-of-arrivals.
10 . The system of claim 1 , wherein the program instructions that are executable by the at least one processor such that the system is configured to determine the second preliminary estimate of the user gaze direction comprise program instructions that are executable by the at least one processor such that the system is configured to:
receive quaternion samples from the IMU;
standardize the quaternion samples; and
calculate angular variation in the standardized quaternion samples;
determine that a range of head rotation over a time period is within a threshold; and
when the range of head rotation over the time period is determined to be within the threshold, average the standardized quaternion samples.
11 . The system of claim 1 , wherein the at least one non-transitory computer readable medium further comprises program instructions that are executable by the at least one processor such that the system is configured to:
detect that a threshold period of time has elapsed since a previous time-of-flight measurement; and
send, via a wireless network interface, instructions to trigger a new time-of-flight measurement.
12 . The system of claim 1 , wherein the at least one non-transitory computer readable medium further comprises program instructions that are executable by the at least one processor such that the system is configured to:
detect that the user has changed position; and
send, via a wireless network interface, instructions to trigger a new time-of-flight measurement.
13 . A non-wearable playback device comprising:
a network interface;
at least one processor; and
at least one non-transitory computer-readable medium comprising program instructions that are executable by the at least one processor such that the non-wearable playback device is configured to:
receive a command to play back audio on a wearable playback device;
after the wearable playback device is donned by a user, determine, via a first gaze direction estimator, first preliminary estimates of user gaze direction, the first preliminary estimates based on one or more time-of-flight measurements of at least one signal propagating from the non-wearable playback device at a first location in a listening environment to the wearable playback device at a second location in the listening environment;
determine, via a second gaze direction estimator, second preliminary estimates of user gaze direction, the second preliminary estimates based on first samples from an inertial measurement unit (IMU);
estimate gaze directions based on fusion of the first gaze direction estimates and the second preliminary estimates;
determine head tracking data based on a rotation of second samples from the IMU to a reference frame centered on the estimated gaze directions;
during binaural rendering of the audio via a binaural renderer, update binaural rendering filters with the determined head tracking data; and
cause the wearable playback device to play back the binaural-rendered audio to simulate playback from at least one virtual source in the listening environment.
14 . The non-wearable playback device of claim 13 , wherein the program instructions that are executable by the at least one processor such that the non-wearable playback device is configured to estimate the gaze directions based on fusion of the first preliminary estimates and the second preliminary estimates comprise program instructions that are executable by the at least one processor such that the non-wearable playback device is configured to:
based on at least one confidence metric, select the estimated gaze directions from among (a) the first preliminary gaze estimates and (b) the second preliminary gaze estimates.
15 . The non-wearable playback device of claim 14 , wherein the at least one confidence metric comprises a length of time elapsed since a previous time of flight measurement; and wherein the program instructions that are executable by the at least one processor such that the non-wearable playback device is configured to select the estimated gaze directions comprise program instructions that are executable by the at least one processor such that the non-wearable playback device is configured to:
select the first preliminary gaze estimates as the estimated gaze directions when the length of time elapsed since the previous time of flight measurement is less than a threshold; and
select the second preliminary gaze estimates as the estimated gaze directions when the length of time elapsed since the previous time of flight measurement is greater than the threshold.
16 . The non-wearable playback device of claim 13 , wherein the program instructions that are executable by the at least one processor such that the non-wearable playback device is configured to estimate the gaze directions based on fusion of the first preliminary estimates and the second preliminary estimates comprise program instructions that are executable by the at least one processor such that the non-wearable playback device is configured to:
based on at least one confidence metric, apply weightings to at least one of the first preliminary gaze estimates and the second preliminary gaze estimates; and
combine the weighted first preliminary gaze estimates and the second preliminary gaze estimates into the estimated gaze directions.
17 . The non-wearable playback device of claim 16 , wherein the at least one confidence metric comprises a length of time elapsed since a previous time of flight measurement; and wherein the program instructions that are executable by the at least one processor such that the non-wearable playback device is configured to apply weightings to at least one of the first preliminary gaze estimates and the second preliminary gaze estimates comprise program instructions that are executable by the at least one processor such that the non-wearable playback device is configured to:
apply a relatively higher weighting to the first preliminary gaze estimates than the second preliminary gaze estimates when the length of time elapsed since the previous time of flight measurement is less than a threshold length of time; and
apply a relatively lower weighting to the first preliminary gaze estimates than the second preliminary gaze estimates when the length of time elapsed since the previous time of flight measurement is greater than the threshold length of time.
18 . The non-wearable playback device of claim 16 , wherein the at least one confidence metric comprises a condition that the user has moved since a previous time of flight measurement; and wherein the program instructions that are executable by the at least one processor such that the non-wearable playback device is configured to apply weightings to at least one of the first preliminary gaze estimates and the second preliminary gaze estimates comprise program instructions that are executable by the at least one processor such that the non-wearable playback device is configured to:
apply a relatively lower weighting to the first preliminary gaze estimates than the second preliminary gaze estimates when the condition that the user has moved since the previous time of flight measurement is present; and
apply a relatively higher weighting to the first preliminary gaze estimates than the second preliminary gaze estimates when the condition that the user has moved since the previous time of flight measurement is not present.
19 . The non-wearable playback device of claim 16 , wherein the at least one confidence metric comprises an IMU drift estimate; and wherein the program instructions that are executable by the at least one processor such that the non-wearable playback device is configured to apply weightings to at least one of the first preliminary gaze estimates and the second preliminary gaze estimates comprise program instructions that are executable by the at least one processor such that the non-wearable playback device is configured to:
apply a relatively lower weighting to the second preliminary gaze estimates than the first preliminary gaze estimates when the IMU drift estimate is greater than an IMU drift threshold; and
apply a relatively higher weighting to the second preliminary gaze estimates than the first preliminary gaze estimates when the IMU drift estimate is less than the IMU drift threshold.
20 . The non-wearable playback device of claim 13 , wherein the at least one non-transitory computer readable medium further comprises program instructions that are executable by the at least one processor such that the non-wearable playback device is configured to:
detect a trigger comprising one of (a) elapse of a threshold period of time since a previous time-of-flight measurement and (b) a user change in position; and
based on detection of the trigger, send, via a wireless network interface, instructions to trigger a new time-of-flight measurement.
21 . At least one on-transitory computer-readable medium comprising program instructions that are executable by at least one processor such that a system is configured to: receive a command to play back audio on a wearable playback device;
after the wearable playback device is donned by a user, determine, via a first gaze direction estimator, first preliminary estimates of user gaze direction, the first preliminary estimates based on one or more time-of-flight measurements of at least one signal propagating from a non-wearable playback device at a first location in a listening environment to the wearable playback device at a second location in the listening environment;
determine, via a second gaze direction estimator, second preliminary estimates of user gaze direction, the second preliminary estimates based on first samples from an inertial measurement unit (IMU);
estimate gaze directions based on fusion of the first gaze direction estimates and the second preliminary estimates;
determine head tracking data based on a rotation of second samples from the IMU to a reference frame centered on the estimated gaze directions;
during binaural rendering of the audio via a binaural renderer, update binaural rendering filters with the determined head tracking data; and
cause the wearable playback device to play back the binaural-rendered audio to simulate playback from at least one virtual source in the listening environment.