Transforming computer game audio using impulse response of a virtual 3D space generated by NeRF input to a convolutional reverberation engine
A 3D neural radiance field (NeRF) is used to generate an impulse response (IR) characterization, which can then be input to a convolutional reverberation engine to create an audio experience that reflects the in-game world on a 2.0 stereo speaker system. The NeRF recreates a background geometry and the impulse response of a virtual 3D space generated using NeRF is input to the convolutional reverberation engine to transform game sounds/music to appear as though they are occurring inside the 3D space of the game. The same may be done for the player's real-world space in which the virtual IR and real IR are blended together in real-time, and the real-life player tracked as he moves around the room to create audio as it would sound were the player moving through the virtual space while adjusting for the acoustics of the real space.
1. A device comprising:
at least one computer storage that is not a transitory signal and that comprises instructions executable by at least one processor to:
generate a three dimensional (3D) neural radiance field (NeRF) from at least one image of a virtual space in a computer simulation;
use at least part of the NeRF to generate an impulse response (IR) characterizing sound transmission in the virtual space;
process audio from the computer simulation using the IR; and
play the audio after processing using the IR on at least one speaker.
2. The device of claim 1 , wherein the instructions are executable to process the audio from the computer simulation at least in part using at least one convolutional reverberation engine decoding the IR characterizing sound transmission in the virtual space.
3. The device of claim 1 , comprising the at least one processor.
4. The device of claim 1 , wherein the instructions are executable to:
for at least one object in the virtual space, correlate at least one surface characteristic to at least one acoustic reflection property; and
use the at least one acoustic reflection property to generate the IR.
5. The device of claim 4 , wherein the surface characteristic comprises at least one texture.
6. The device of claim 1 , wherein the at least one speaker comprises a speaker in a stereo speaker system.
7. The device of claim 1 , wherein the instructions are executable to:
access at least one image of a physical space in which the speaker is disposed;
based at least in part on the image of the physical space, generate a physical space IR; and
use the physical space IR to process sound from the computer simulation such that audio from the computer simulation is played as it would sound were a player moving through the virtual space.
8. The device of claim 7 , wherein the instructions are executable to:
based at least in part on the image of the physical space, generate a physical space IR at least in part using a NeRF representing the physical space.
9. An apparatus comprising:
at least one processor programmed with instructions to:
generate a neural radiance field (NeRF) representation of a virtual space from a computer simulation;
using at least one virtual sound source and at least one virtual microphone in the virtual space, generate at least one impulse response (IR) representation of the virtual space;
process audio from the computer simulation at least in part using the IR representation of the virtual space; and
play the audio from the computer simulation on at least one real world (RW) speaker in a RW space.
10. The apparatus of claim 9 , wherein the instructions are executable to process audio from the computer simulation at least in part using at least one convolutional reverberation engine programmed with the IR representation of the virtual space.
11. The apparatus of claim 9 , wherein the instructions are executable to:
for at least one object in the virtual space, correlate at least one surface characteristic of the object to at least one acoustic reflection property; and
use the at least one acoustic reflection property to generate the IR representation of the virtual space.
12. The apparatus of claim 11 , wherein the surface characteristic comprises at least one texture.
13. The apparatus of claim 9 , wherein the at least one RW speaker comprises a speaker in a stereo speaker system.
14. The apparatus of claim 9 , wherein the instructions are executable to:
access at least one image of a physical space in which the RW speaker is disposed;
based at least in part on the image of the physical space, generate a physical space IR; and
use the physical space IR to process sound from the computer simulation such that audio from the computer simulation is played as it would sound were a player moving through the virtual space.
15. The apparatus of claim 14 , wherein the instructions are executable to:
based at least in part on the image of the physical space, generate a physical space IR at least in part using a NeRF representing the physical space.
16. A method comprising:
generating a neural radiance field (NeRF) representation of a virtual space;
based at least in part on the NeRF representation, generating information representing acoustic transmission in the virtual space; and
playing, on at least one speaker, audio processed using the information representing acoustic transmission in the virtual space.
17. The method of claim 16 , wherein the information representing acoustic transmission in the virtual space comprises at least one impulse response (IR) element.
18. The method of claim 17 , comprising processing the audio using at least one convolutional reverberation engine programmed with the IR.
19. The method of claim 16 , wherein the virtual space and the audio are both sourced from a computer simulation.
20. The method of claim 16 , comprising processing the audio using at least one impulse response (IR) representation of a physical space in which the speaker is located.