Head-related transfer function (HRTF) personalization based on captured images of user
A virtual reality (VR) system simulates sounds that a user of the VR system perceives to have originated from sources at desired virtual locations of the VR system. The simulated sounds are generated based on personalized head-related transfer functions (HRTF) of the user that are constructed by applying machine-learned models to a set of anatomical features identified for the user. The set of anatomical features may be identified from images of the user captured by a camera. In one instance, the HRTF is represented as a reduced set of parameters that allow the machine-learned models to capture the variability in HRTF across individual users while being trained in a computationally-efficient manner.
1. A method comprising:
obtaining, by a computing device, a set of anatomical features describing physical characteristics of a user's body relevant to a personalized head-related transfer function (HRTF) of the user, the set of anatomical features identified from one or more images of the user;
constructing, by the computing device, the personalized HRTF of the user from the set of anatomical features of the user; and
providing, by the computing device, the personalized HRTF for generating audio signals using the personalized HRTF.
2. The method of claim 1 , wherein the personalized HRTF is represented by a set of parameters, and constructing the personalized HRTF of the user comprises applying a machine-learned model to the set of anatomical features of the user to generate the set of parameters for the personalized HRTF.
3. The method of claim 2 , wherein the machine-learned model is trained using training data for a plurality of subjects, the training data comprising:
for each of the subjects, a training set of anatomical features obtained from one or more images of the subject, and
for each of the subjects, a training set of parameters that represent a HRTF for the subject.
4. The method of claim 1 , wherein providing the personalized HRTF comprises providing the personalized HRTF to a system that generates the audio signals by applying a sound source signal to the personalized HRTF to generate the audio signals.
5. The method of claim 4 , wherein the system generates the audio signals while the user interacts with a virtual reality (VR) environment, an augmented reality (AR) environment, or a mixed reality (MR) environment.
6. The method of claim 1 , wherein the one or more images are RGB images from a RGB camera.
7. The method of claim 1 , wherein the one or more images are depth images from a depth sensor.
8. The method of claim 1 , wherein the personalized HRTF is provided to a system that applies the personalized HRTF to generate the audio signals, and wherein the computing device is included in a remote server distinct from the system.
9. The method of claim 1 , wherein the computing device is included in a system for simulating a virtual reality (VR) environment, an augmented reality (AR) environment, or a mixed reality (MR) environment the user interacts with.
10. A non-transitory computer-readable medium storing instructions for execution on a processor, the instructions when executed by the processor causing the processor to perform steps comprising:
obtaining a set of anatomical features describing physical characteristics of a user's body relevant to a personalized head-related transfer function (HRTF) of the user, the set of anatomical features identified from one or more images of the user;
constructing the personalized HRTF of the user from the set of anatomical features of the user; and
providing the personalized HRTF for generating audio signals using the personalized HRTF.
11. The non-transitory computer-readable medium of claim 10 , wherein the personalized HRTF is represented by a set of parameters, and constructing the personalized HRTF of the user comprises applying a machine-learned model to the set of anatomical features of the user to generate the set of parameters for the personalized HRTF.
12. The non-transitory computer-readable medium of claim 11 , wherein the machine-learned model is trained using training data for a plurality of subjects, the training data comprising:
for each of the subjects, a training set of anatomical features obtained from one or more images of the subject, and
for each of the subjects, a training set of parameters that represent a HRTF for the subject.
13. The non-transitory computer-readable medium of claim 10 , wherein providing the personalized HRTF comprises providing the personalized HRTF to a system that generates the audio signals by applying a sound source signal to the personalized HRTF.
14. The non-transitory computer-readable medium of claim 13 , wherein the system generates the audio signals while the user interacts with a virtual reality (VR) environment, an augmented reality (AR) environment, or a mixed reality (MR) environment.
15. The non-transitory computer-readable medium of claim 10 , wherein the one or more images are RGB images from a RGB camera.
16. The non-transitory computer-readable medium of claim 10 , wherein the one or more images are depth images from a depth sensor.
17. The non-transitory computer-readable medium of claim 10 , wherein the personalized HRTF is provided to a system at a remote location, and wherein the system applies the personalized HRTF to generate the audio signals.
18. A system comprising:
an interface module configured to receive a set of anatomical features describing physical characteristics of a user's body relevant to a personalized head-related transfer function (HRTF) of the user and identified from images of the user; and
a HRTF generation module configured to construct the personalized HRTF for the user based on the received set of anatomical features by applying a machine-learned model to the set of anatomical features for the user, and configured to provide the personalized HRTF to a device that applies the personalized HRTF to generate audio signals.
19. The system of claim 18 , wherein the personalized HRTF is represented by a set of parameters, and wherein the HRTF generation module is further configured to apply a machine-learned model to the set of anatomical features of the user to construct the personalized HRTF.
20. The system of claim 18 , wherein the system includes the device, and the device is configured to simulate a virtual reality (VR) environment, an augmented reality (AR) environment, or a mixed reality (MR) environment the user interacts with.