Manifold learning for sound field estimation
System and methods are provided for estimating the sound field from partial observations. Estimating an acoustic environment for virtual reality and augmented reality applications is a step in the creation of simulated acoustic sound scenes. In particular, the impulse responses of room can be estimated with a generative model. In a teleconferencing scenario with remote participants and a group of participants in a common physical space, giving the remote participants the impression that all other participants are sitting is in the same room acoustically requires filtering the speech of the remote participants with impulse responses estimated at the desired rendering position in the conference room.
1. A computer-implemented method for estimating a sound field for virtual reality or augmented reality, comprising:
receiving, for a first position associated with a near end room, room data comprising (i) input audio data and (ii) target audio data;
generating measurement vector data from the input audio data;
generating initial input vector data for a second position associated with the near end room;
generating input data from (i) the measurement vector data, (ii) the first position, (iii) the initial input vector data, and (iv) the second position;
applying initial filter parameters to the input data that results in filtered data;
generating target data from the target audio data;
determining an estimated loss from the filtered data and the target data;
determining a matrix from a decoder of a trained variational autoencoder;
combining the matrix, the estimated loss, and a step value that results in a tangent vector;
applying the decoder to a point in a tangent space indicated by the tangent vector, wherein the decoder outputs updated filter parameters;
receiving far end audio data;
in response to receiving the far end audio data, substantially in real-time:
generating near end audio data from (i) the far end audio data, (ii) the updated filter parameters, and (iii) the second position; and
outputting the near end audio data.
2. The computer-implemented method of claim 1 , further comprising:
training a machine learning model with training data comprising a plurality of impulse responses for a second room as input training data and a training label,
wherein the training data further comprises, for each impulse response in the plurality of impulse responses, a position relative to a source in the second room, and
wherein training the machine learning model further comprises:
determining a loss and a gradient of a neural network; and
updating, based on the loss and the gradient, a weight of the neural network that results in the trained variational autoencoder.
3. The computer-implemented method of claim 2 , wherein the training data further comprises a room type for the second room, and the input data further comprises a near end room type.
4. The computer-implemented method of claim 1 , wherein generating the near end audio data further comprises:
applying a machine learning model to the far end audio data, wherein the machine learning model outputs de-reverbed audio data.
5. The computer-implemented method of claim 4 , wherein generating the near end audio data further comprises:
determining a near end impulse response from the updated filter parameters at the second position; and
applying the near end impulse response at the second position to the de-reverbed audio data that results in the near end audio data as reverbed.
6. The computer-implemented method of claim 1 , further comprising:
iteratively determining filter parameters until a threshold is satisfied.
7. One or more non-transitory computer-readable storage media storing computer executable instructions that when executed by a computing system perform operations comprising:
receiving, for a first position associated with a near end room, room data comprising (i) input audio data and (ii) target audio data;
generating measurement vector data from the input audio data;
generating initial input vector data for a second position associated with the near end room;
generating input data from (i) the measurement vector data, (ii) the first position, (iii) the initial input vector data, and (iv) the second position;
applying initial filter parameters to the input data that results in filtered data;
generating target data from the target audio data;
determining an estimated loss from the filtered data and the target data;
determining a matrix from a decoder of a trained generative model;
combining the matrix, the estimated loss, and a step value that results in a tangent vector;
applying the decoder to a point in a tangent space indicated by the tangent vector, wherein the decoder outputs updated filter parameters;
receiving far end audio data;
in response to receiving the far end audio data, substantially in real-time:
generating near end audio data from (i) the far end audio data, (ii) the updated filter parameters, and (iii) the second position; and
transmitting the near end audio data.
8. The one or more non-transitory computer-readable storage media of claim 7 storing further computer-executable instructions that when executed by the computing system perform further operations comprising:
training a machine learning model with training data comprising a plurality of impulse responses for a second room as input training data and a training label,
wherein the training data further comprises, for each impulse response in the plurality of impulse responses, a position relative to a source in the second room, and
wherein training the machine learning model further comprises:
determining a loss and a gradient of a neural network; and
updating, based on the loss and the gradient, a weight of the neural network that results in the trained generative model.
9. The one or more non-transitory computer-readable storage media of claim 8 , wherein determining the loss of the neural network further comprises:
applying a loss function with a regularization term, wherein the regularization term causes a representation of a Hessian matrix of an adaptive filter cost function in a latent space to be approximately diagonal.
10. The one or more non-transitory computer-readable storage media of claim 7 , wherein combining the matrix, the estimated loss, and the step value further comprises:
calculating an inverse Hessian matrix from the matrix; and
calculating the tangent vector from the matrix, the inverse Hessian matrix, the estimated loss, and the step value.
11. The one or more non-transitory computer-readable storage media of claim 7 , wherein generating the near end audio data further comprises:
applying a machine learning model to the far end audio data, wherein the machine learning model outputs de-reverbed audio data.
12. The one or more non-transitory computer-readable storage media of claim 11 , wherein generating the near end audio data further comprises:
determining a near end impulse response from the updated filter parameters at the second position; and
applying the near end impulse response at the second position to the de-reverbed audio data that results in the near end audio data as reverbed.
13. The one or more non-transitory computer-readable storage media of claim 7 , wherein the trained generative model comprises a variational autoencoder.
14. A system comprising:
a non-transitory data storage medium; and
a computer hardware processor in communication with the non-transitory data storage medium, wherein the computer hardware processor is configured to execute computer-executable instructions to at least:
receive, for a first position associated with a near end room, room data comprising (i) input audio data and (ii) target audio data;
generate measurement vector data from the input audio data;
generate initial input vector data for a second position associated with the near end room;
generate input data from (i) the measurement vector data, (ii) the first position, (iii) the initial input vector data, and (iv) the second position;
apply initial filter parameters to the input data that results in filtered data;
generate target data from the target audio data;
determine an estimated loss from the filtered data and the target data;
determine a matrix from a decoder of a trained generative model;
combine the matrix, the estimated loss, and a step value that results in a tangent vector;
apply the decoder to a point in a tangent space indicated by the tangent vector, wherein the decoder outputs updated filter parameters;
receive far end audio data;
generate near end audio data from (i) the far end audio data, (ii) the updated filter parameters, and (iii) the second position; and
transmit the near end audio data.
15. The system of claim 14 , wherein the computer hardware processor executes additional computer-executable instructions to at least:
train a machine learning model with training data comprising a plurality of impulse responses for a second room as input training data and a training label,
wherein the training data further comprises, for each impulse response in the plurality of impulse responses, a position relative to a source in the second room, and
wherein to train the machine learning model, the computer hardware processor executes the additional computer-executable instructions to at least:
determine a loss and a gradient of a neural network; and
update, based on the loss and the gradient, a weight of the neural network that results in the trained generative model.
16. The system of claim 15 , wherein to train the machine learning model with the training data, the computer hardware processor executes further computer-executable instructions to at least:
apply a Kirchhoff-Helmholtz integral to the plurality of impulse responses at a respective position relative to the source in the second room.
17. The system of claim 15 , wherein the training data further comprises a room type for the second room, and the input data further comprises a near end room type.
18. The system of claim 17 , wherein the room type comprises at least one of a small room type, a medium room type, or a large room type.
19. The system of claim 14 , wherein to generate the near end audio data, the computer hardware processor executes additional computer-executable instructions to at least:
apply a machine learning model to the far end audio data, wherein the machine learning model outputs de-reverbed audio data.
20. The system of claim 19 , wherein to generate the near end audio data, the computer hardware processor executes further computer-executable instructions to at least:
determine a near end impulse response from the updated filter parameters at the second position; and
apply the near end impulse response at the second position to the de-reverbed audio data that results in the near end audio data as reverbed.