Systems and methods for classifying beamformed signals for binaural audio playback

1. A computer-implemented method comprising:

receiving, by a computing device, a signal for each channel of an audio transducer array on a wearable device, wherein the received signal is captured by the wearable device when worn by a user;

calculating, by the computing device, a beamformed signal for each beam direction of a set of beamforming filters for the wearable device;

classifying, by the computing device, a first beamformed signal from the calculated beamformed signals into a first class of sound and a second beamformed signal from the calculated beamformed signals into a second class of sound;

adjusting, based on the classifying, a gain of the first beamformed signal relative to the second beamformed signal to enable sound mixing;

converting, by the computing device, the calculated and adjusted beamformed signals into spatialized binaural audio based on a position of the user wearing the wearable device and a position of a virtual user independent of the position of the user wearing the wearable device, wherein the spatialized binaural audio is updated in real time based on a new position of the user wearing the wearable device and a new position of the virtual user;

rotating a corresponding video and recalculating a timing of the corresponding video based on the new position of the user wearing the wearable device and the new position of the virtual user; and

transmitting the spatialized binaural audio and the corresponding video to a playback device of the virtual user, wherein the spatialized binaural audio and the corresponding video replicates a real-world user experience shared from the wearable device to the playback device as the virtual user's point-of-view.

2. The method of claim 1 , wherein the set of beamforming filters comprises preprocessed filters created by testing the audio transducer array of the wearable device in an anechoic chamber.

3. The method of claim 2 , wherein testing the audio transducer array of the wearable device comprises:

capturing test audio with the audio transducer array of the wearable device worn on a model head;

measuring a set of array transfer functions (ATFs) for the audio transducer array based on the captured test audio; and

calculating the set of beamforming filters using the set of ATFs.

4. The method of claim 3 , wherein measuring the set of ATFs further comprises measuring a set of head-related transfer functions (HRTFs).

5. The method of claim 1 , wherein calculating the beamformed signal comprises:

convolving each received signal with a corresponding beamforming filter for the beam direction; and

taking a sum of the convolved signals for the beam direction.

6. The method of claim 1 , wherein:

the first class of sound comprises at least one of:

a class of sound selected by the virtual user; or

a predetermined class of sound; and

the second class of sound comprises at least one of:

an alternate class of sound selected by the virtual user; or

a predetermined alternate class of sound.

7. The method of claim 1 , wherein classifying the first beamformed signal and the second beamformed signal comprises applying a deep learning model of sound classification to the first beamformed signal and the second beamformed signal.

8. The method of claim 1 , wherein converting the calculated and adjusted beamformed signals into the spatialized binaural audio comprises:

identifying a set of left-ear beam directions and a set of right-ear beam directions based on the position of the user wearing the wearable device and the position of the virtual user;

calculating a left-ear signal for the set of left-ear beam directions; and

calculating a right-ear signal for the set of right-ear beam directions.

9. The method of claim 8 , wherein calculating the left-ear signal comprises:

convolving the calculated and adjusted beamformed signals with a set of left-ear HRTF filters for the set of left-ear beam directions; and

taking a sum of the convolved beamformed signals for the set of left-ear beam directions.

10. The method of claim 8 , wherein calculating the right-ear signal comprises:

convolving the calculated and adjusted beamformed signals with a set of right-ear HRTF filters for the set of right-ear beam directions; and

taking a sum of the convolved beamformed signals for the set of right-ear beam directions.

11. The method of claim 1 , further comprising:

detecting the new position of the user wearing the wearable device or the virtual user; and

recalculating the spatialized binaural audio for the new position of the user wearing the wearable device or the virtual user.

12. The method of claim 1 , further comprising:

adjusting the timing of the corresponding video based on a timing of the spatialized binaural audio; and

transmitting the adjusted corresponding video to the playback device of the virtual user.

13. A system comprising:

a reception module, stored in memory, that receives a signal for each channel of an audio transducer array on a wearable device, wherein the received signal is captured by the wearable device when worn by a user;

a calculation module, stored in memory, that calculates a beamformed signal for each beam direction of a set of beamforming filters for the wearable device;

a classification module, stored in memory, that classifies a first beamformed signal from the calculated beamformed signals into a first class of sound and a second beamformed signal from the calculated beamformed signals into a second class of sound;

an adjustment module, stored in memory, that adjusts, based on the classifying, a gain of the first beamformed signal relative to the second beamformed signal to enable sound mixing;

a conversion module, stored in memory, that:

converts the calculated and adjusted beamformed signals into spatialized binaural audio based on a position of the user wearing the wearable device and a position of a virtual user independent of the position of the user wearing the wearable device, wherein the spatialized binaural audio is updated in real time based on a new position of the user wearing the wearable device and a new position of the virtual user; and

rotates a corresponding video and recalculating a timing of the corresponding video based on the new position of the user wearing the wearable device and the new position of the virtual user;

a transmitting module, stored in memory, that transmits the spatialized binaural audio and the corresponding video to a playback device of the virtual user, wherein the spatialized binaural audio and the corresponding video replicates a real-world user experience shared from the wearable device to the playback device as the virtual user's point-of-view; and

at least one processor that executes the reception module, the calculation module, the classification module, the adjustment module, the conversion module, and the transmitting module.

14. The system of claim 13 , wherein the calculation module calculates the beamformed signal by:

convolving each received signal with a corresponding beamforming filter for the beam direction; and

taking a sum of the convolved signals for the beam direction.

15. The system of claim 13 , wherein the classification module classifies the first beamformed signal and the second beamformed signal by applying a deep learning model of sound classification to the first beamformed signal and the second beamformed signal.

16. The system of claim 13 , wherein the conversion module converts the calculated and adjusted beamformed signals into the spatialized binaural audio by:

identifying a set of left-ear beam directions and a set of right-ear beam directions based on the position of the user wearing the wearable device and the position of the virtual user;

calculating a left-ear signal for the set of left-ear beam directions; and

calculating a right-ear signal for the set of right-ear beam directions.

17. The system of claim 16 , wherein calculating the left-ear signal comprises:

convolving the calculated and adjusted beamformed signals with a set of left-ear HRTF filters for the set of left-ear beam directions; and

taking a sum of the convolved beamformed signals for the set of left-ear beam directions.

18. The system of claim 16 , wherein calculating the right-ear signal comprises:

convolving the calculated and adjusted beamformed signals with a set of right-ear HRTF filters for the set of right-ear beam directions; and

taking a sum of the convolved beamformed signals for the set of right-ear beam directions.

19. The system of claim 13 , wherein the conversion module further:

detects the new position of the user wearing the wearable device or the virtual user; and

recalculates the spatialized binaural audio for the new position of the user wearing the wearable device or the virtual user.

20. A non-transitory computer-readable medium comprising one or more computer-executable instructions that, when executed by at least one processor of a computing device, cause the computing device to:

receive a signal for each channel of an audio transducer array on a wearable device, wherein the received signal is captured by the wearable device when worn by a user;

calculate a beamformed signal for each beam direction of a set of beamforming filters for the wearable device;

classify a first beamformed signal from the calculated beamformed signals into a first class of sound and a second beamformed signal from the calculated beamformed signals into a second class of sound;

adjust, based on the classifying, a gain of the first beamformed signal relative to the second beamformed signal to enable sound mixing;

convert the calculated and adjusted beamformed signals into spatialized binaural audio based on a position of the user wearing the wearable device and a position of a virtual user independent of the position of the user wearing the wearable device, wherein the spatialized binaural audio is updated in real time based on a new position of the user wearing the wearable device and a new position of the virtual user;

rotate a corresponding video and recalculating a timing of the corresponding video based on the new position of the user wearing the wearable device and the new position of the virtual user; and

transmit the spatialized binaural audio and the corresponding video to a playback device of the virtual user, wherein the spatialized binaural audio and the corresponding video replicates a real-world user experience shared from the wearable device to the playback device as the virtual user's point-of-view.