Physics-based audio and haptic synthesis
Disclosed herein are systems and methods for generating and presenting virtual audio for mixed reality systems. A method may include determining a collision between a first object and a second object, wherein the first object comprises a first virtual object. A memory storing one or more audio models can be accessed. It can be determined if the one or more audio models stored in the memory comprises an audio model corresponding to the first object. In accordance with a determination that the one or more audio models comprises an audio model corresponding to the first object, an audio signal can be synthesized, wherein the audio signal is based on the collision and the audio model corresponding to the first object, and the audio signal can be presented to a user via a speaker of a head-wearable device. In accordance with a determination that the one or more audio models does not comprise an audio model corresponding to the first object, an acoustic property of the first object can be determined, a custom audio model based on the acoustic property of the first object can be generated, an audio signal can be synthesized, wherein the audio signal is based on the collision and the custom audio model, and the audio signal can be presented, via a speaker of a head-wearable device, to a user.
1. A method comprising:
determining a collision between a first object and a second object, wherein the first object comprises a first virtual object;
accessing a memory storing one or more audio models;
determining if the one or more audio models stored in the memory comprises an audio model corresponding to the first object;
in accordance with a determination that the one or more audio models comprises an audio model corresponding to the first object:
synthesizing a first audio signal, wherein the first audio signal is based on the collision and the audio model corresponding to the first object, and
presenting the first audio signal, via a speaker of a head-wearable device, to a user; and
in accordance with a determination that the one or more audio models does not comprise an audio model corresponding to the first object:
determining an acoustic property of the first object,
generating a custom audio model based on the acoustic property of the first object,
synthesizing a second audio signal, wherein the second audio signal is based on the collision and the custom audio model, and
presenting the second audio signal, via the speaker of the head-wearable device, to the user.
2. The method of claim 1 , wherein the second object comprises a second virtual object.
3. The method of claim 1 , wherein the second object comprises a real object.
4. The method of claim 3 , further comprising:
receiving, via a sensor of the head-wearable device, sensor data associated with the real object;
determining an acoustic characteristic of the second object based on the sensor data; and
generating an audio model corresponding to the second object based on the acoustic characteristic.
5. The method of claim 3 , further comprising:
identifying, via a sensor of the head-wearable device, an aspect of the second object;
retrieving an acoustic characteristic of the second object based on the identified aspect; and
generating an audio model corresponding to the second object based on the acoustic characteristic.
6. The method of claim 3 , further comprising:
receiving, via a sensor of the head-wearable device, sensor data associated with the real object;
determining a first acoustic characteristic of the second object based on the sensor data;
generating a first audio model corresponding to the second object based on the first acoustic characteristic;
identifying, via a sensor of the head-wearable device, an aspect of the second object;
retrieving a second acoustic characteristic of the second object based on the identified aspect;
generating a second audio model corresponding to the second object based on the second acoustic characteristic; and
generating a third audio model corresponding to the second object based on the first audio model corresponding to the second object and the second audio model corresponding to the second object.
7. A system comprising:
one or more processors, wherein the one or more processors are configured to execute a method comprising:
determining a collision between a first object and a second object, wherein the first object comprises a first virtual object;
accessing a memory storing one or more audio models;
determining if the one or more audio models stored in the memory comprises an audio model corresponding to the first object;
in accordance with a determination that the one or more audio models comprises an audio model corresponding to the first object:
synthesizing a first audio signal, wherein the first audio signal is based on the collision and the audio model corresponding to the first object, and
presenting the first audio signal, via a speaker of a head-wearable device, to a user; and
in accordance with a determination that the one or more audio models does not comprise an audio model corresponding to the first object:
determining an acoustic property of the first object,
generating a custom audio model based on the acoustic property of the first object,
synthesizing a second audio signal, wherein the second audio signal is based on the collision and the custom audio model, and
presenting the second audio signal, via the speaker of the head-wearable device, to the user.
8. The system of claim 7 , wherein the second object comprises a second virtual object.
9. The system of claim 7 , wherein the second object comprises a real object.
10. The system of claim 9 , the method further comprising:
receiving, via a sensor of a head-wearable device, sensor data associated with the real object;
determining an acoustic characteristic of the second object based on the sensor data; and
generating an audio model corresponding to the second object based on the acoustic characteristic.
11. The system of claim 9 , the method further comprising:
identifying, via a sensor of the head-wearable device, an aspect of the second object;
retrieving an acoustic characteristic of the second object based on the identified aspect; and
generating an audio model corresponding to the second object based on the acoustic characteristic.
12. The system of claim 9 , the method further comprising:
receiving, via a sensor of the head-wearable device, sensor data associated with the real object;
determining a first acoustic characteristic of the second object based on the sensor data;
generating a first audio model corresponding to the second object based on the first acoustic characteristic;
identifying, via a sensor of the head-wearable device, an aspect of the second object;
retrieving a second acoustic characteristic of the second object based on the identified aspect;
generating a second audio model corresponding to the second object based on the second acoustic characteristic; and
generating a third audio model corresponding to the second object based on the first audio model corresponding to the second object and the second audio model corresponding to the second object.
13. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to execute a method comprising:
determining a collision between a first object and a second object, wherein the first object comprises a first virtual object;
accessing a memory storing one or more audio models;
determining if the one or more audio models stored in the memory comprises an audio model corresponding to the first object;
in accordance with a determination that the one or more audio models comprises an audio model corresponding to the first object:
synthesizing a first audio signal, wherein the first audio signal is based on the collision and the audio model corresponding to the first object, and
presenting the first audio signal, via a speaker of a head-wearable device, to a user; and
in accordance with a determination that the one or more audio models does not comprise an audio model corresponding to the first object:
determining an acoustic property of the first object,
generating a custom audio model based on the acoustic property of the first object,
synthesizing a second audio signal, wherein the second audio signal is based on the collision and the custom audio model, and
presenting the second audio signal, via the speaker of the head-wearable device, to the user.
14. The non-transitory computer-readable medium of claim 13 , wherein the second object comprises a second virtual object.
15. The non-transitory computer-readable medium of claim 13 , wherein the second object comprises a real object.
16. The non-transitory computer-readable medium of claim 15 , the method further comprising:
receiving, via a sensor of the head-wearable device, sensor data associated with the real object;
determining an acoustic characteristic of the second object based on the sensor data; and
generating an audio model corresponding to the second object based on the acoustic characteristic.
17. The non-transitory computer-readable medium of claim 15 , the method further comprising:
identifying, via a sensor of the head-wearable device, an aspect of the second object;
retrieving an acoustic characteristic of the second object based on the identified aspect; and
generating an audio model corresponding to the second object based on the acoustic characteristic.
18. The non-transitory computer-readable medium of claim 15 , the method further comprising:
receiving, via a sensor of the head-wearable device, sensor data associated with the real object;
determining a first acoustic characteristic of the second object based on the sensor data;
generating a first audio model corresponding to the second object based on the first acoustic characteristic;
identifying, via a sensor of the head-wearable device, an aspect of the second object;
retrieving a second acoustic characteristic of the second object based on the identified aspect;
generating a second audio model corresponding to the second object based on the second acoustic characteristic; and
generating a third audio model corresponding to the second object based on the first audio model corresponding to the second object and the second audio model corresponding to the second object.