IP Library Granted Patent US 10,887,709
Granted Patent B1
US 10,887,709 · App. 16/582,820 · Granted Jan 5, 2021

Aligned beam merger

Inventors: Mohamed Mansour (Cupertino, CA); Carlos Renato Nakagawa (San Jose, CA)
Assignee: Amazon Technologies, Inc.
H04R29/001G10K11/34H04R3/00G10L2021/02166H04R1/32H04R2430/20
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,887,709
App. No.
16/582,820
Granted
Jan 5, 2021
Kind
B1
Abstract

A system configured to perform aligned beam merger (ABM) processing to combine multiple beamformed signals. The system may capture audio data and perform beamforming to generate beamformed audio signals corresponding to a plurality of directions. The system may apply an ABM algorithm to select a number of the beamformed audio signals, align the selected audio signals, and merge the selected audio signals to generate a distortionless output audio signal. The system may scale the selected audio signals based on relative magnitude and apply a complex correction factor to compensate for a phase error for each of the selected audio signals.

Claims (132)

1. A computer-implemented method, the method comprising, by a device:

receiving a plurality of audio signals that include a first audio signal corresponding to a first direction and a second audio signal corresponding to a second direction;

determining a plurality of signal-to-noise ratio (SNR) values including a first SNR value corresponding to a portion of the first audio signal that is within a first frequency band;

selecting, using the plurality of SNR values, a first number of audio signals of the plurality of audio signals, the first number of audio signals including the first audio signal, wherein the selecting further comprises:

determining a highest SNR value of the plurality of SNR values within the first frequency band;

determining a threshold SNR value corresponding to the first frequency band by multiplying a fixed value by the highest SNR value;

determining that the first SNR value exceeds the threshold SNR value;

increasing a first counter value for the first audio signal, the first counter value being one of a plurality of counter values;

determining a first number of highest counter values from the plurality of counter values; and

selecting the first number of audio signals having the first number of highest counter values;

determining that a third audio signal of the first number of audio signals has a highest counter value of the plurality of counter values;

determining a target angle value indicating a third direction corresponding to the third audio signal;

determining a phase value for the portion of the first audio signal, the phase value indicating a phase shift between a first angle value of the first audio signal and the target angle value;

determining a group SNR value by summing a first number of SNR values of the plurality of SNR values, wherein the first number of SNR values correspond to both the first number of audio signals and the first frequency band;

determining a first scaling value for the portion of the first audio signal, the first scaling value indicating a ratio of the first SNR value to the group SNR value; and

generating an output audio signal using the first scaling value, the phase value, and the portion of the first audio signal.

2. The computer-implemented method of claim 1 , wherein generating the output audio signal further comprises:

generating a first coefficient value using the phase value;

generating a first portion of the output audio signal by multiplying the first scaling value, the first coefficient value, and the portion of the first audio signal;

generating a second coefficient value using a second phase value for a portion of the third audio signal that is within the first frequency band;

determining a second scaling value for the portion of the third audio signal, the second scaling value indicating a ratio of a third SNR value of the portion of the third audio signal to the group SNR value;

generating a second portion of the output audio signal by multiplying the second scaling value, the second coefficient value, and the portion of the third audio signal; and

generating the output audio signal by combining the first portion of the output audio signal and the second portion of the output audio signal.

3. The computer-implemented method of claim 1 , wherein determining the plurality of SNR values further comprises:

determining, using a first time constant, a first energy value of the portion of the first audio signal that is within the first frequency band;

determining, using the first time constant, a second energy value of a portion of the second audio signal that is within the first frequency band;

determining, during a first time period, that playback audio is being generated by one or more loudspeakers of the device;

determining that the second energy value is lowest of a plurality of energy values associated with the first frequency band; and

determining the first SNR value by dividing the first energy value by the second energy value.

4. The computer-implemented method of claim 3 , further comprising:

determining, during a second time period, that the playback audio is not being generated by the one or more loudspeakers;

determining, using the first time constant, a third energy value corresponding to a second portion of the first audio signal that is within the first frequency band and associated with the second time period;

determining, using a second time constant that is different than the first time constant, a fourth energy value corresponding to the second portion of the first audio signal; and

determining a second SNR value by dividing the third energy value by the fourth energy value.

5. A computer-implemented method, the method comprising:

receiving a plurality of audio signals that includes a first audio signal corresponding to a first direction and a second audio signal corresponding to a second direction;

determining a first signal quality metric value corresponding to a portion of the first audio signal that is within a first frequency band;

determining a second signal quality metric value corresponding to a portion of the second audio signal that is within the first frequency band;

determining, using the first signal quality metric value and the second signal quality metric value, a first number of audio signals of the plurality of audio signals, the first number of audio signals including the first audio signal;

determining a first value corresponding to the portion of the first audio signal, the first value representing a ratio of the first signal quality metric value to a sum of signal quality metric values that are associated with the first number of audio signals and the first frequency band;

determining a second value representing a first phase shift of the portion of the first audio signal, the second value determined using a first angle associated with the first direction and a target angle associated with the first number of audio signals; and

generating an output audio signal using the first value, the second value, and the first number of audio signals.

6. The computer-implemented method of claim 5 , wherein generating the output audio signal further comprises:

generating a first coefficient value using the second value;

generating a first portion of the output audio signal by multiplying the first value, the first coefficient value, and the portion of the first audio signal;

determining a third value corresponding to the portion of the second audio signal, the third value representing a ratio of the second signal quality metric value to the sum of signal quality metric values;

determining a fourth value representing a second phase shift of the portion of the second audio signal, the fourth value determined using a second angle associated with the second direction and the target angle;

generating a second coefficient value using the fourth value;

generating a second portion of the output audio signal by multiplying the third value, the second coefficient value, and the portion of the second audio signal; and

generating the output audio signal by combining the first portion of the output audio signal and the second portion of the output audio signal.

7. The computer-implemented method of claim 5 , wherein determining the first number of audio signals further comprises:

determining that the first signal quality metric value exceeds a first threshold value;

incrementing a first counter value for the first audio signal;

determining that the second signal quality metric value does not exceed the first threshold value;

determining a third signal quality metric value corresponding to a portion of a third audio signal that is within a second frequency band;

determining that the third signal quality metric value exceeds a second threshold value; and

incrementing a second counter value for the third audio signal.

8. The computer-implemented method of claim 7 , wherein determining the first number of audio signals further comprises:

determining a first number of highest counter values from a plurality of counter values, the plurality of counter values including the first counter value and the second counter value; and

using the first number of highest counter values to determine the first number of audio signals.

9. The computer-implemented method of claim 5 , wherein determining the second value further comprises:

selecting a third audio signal of the first number of audio signals;

identifying the target angle corresponding to the third audio signal;

determining a steering vector using the target angle;

determining a beamformer filter associated with a first portion of the first audio signal; and

determining the second value using the beamformer filter and the steering vector.

10. The computer-implemented method of claim 5 , wherein determining the first signal quality metric value further comprises:

determining, during a first time period, that audio data is being sent to one or more loudspeakers;

determining, using a first time constant, a first energy value corresponding to the portion of the first audio signal;

determining, using the first time constant, a second energy value corresponding to the portion of the second audio signal;

determining that the second energy value is lowest of a plurality of energy values associated with the plurality of audio signals and the first frequency band; and

determining the first signal quality metric value using the first energy value and the second energy value.

11. The computer-implemented method of claim 10 , further comprising:

determining, during a second time period, that the audio data is not being sent to the one or more loudspeakers;

determining, using the first time constant, a third energy value corresponding to a second portion of the first audio signal that is within the first frequency band and associated with the second time period;

determining, using a second time constant that is different than the first time constant, a fourth energy value corresponding to the second portion of the first audio signal; and

determining a third signal quality metric value using the third energy value and the fourth energy value.

12. The computer-implemented method of claim 5 , further comprising:

determining a third signal quality metric value corresponding to a portion of a third audio signal of the plurality of audio signals, the portion of the third audio signal being within the first frequency band;

determining, using the third signal quality metric value, a threshold value;

determining that the first signal quality metric value is below the threshold value; and

setting the first value equal to a value of zero.

13. A system comprising:

at least one processor; and

memory including instructions operable to be executed by the at least one processor to cause the system to:

receive a plurality of audio signals that includes a first audio signal corresponding to a first direction and a second audio signal corresponding to a second direction;

determine a first signal quality metric value corresponding to a portion of the first audio signal that is within a first frequency band;

determine a second signal quality metric value corresponding to a portion of the second audio signal that is within the first frequency band;

determine, using the first signal quality metric value and the second signal quality metric value, a first number of audio signals of the plurality of audio signals, the first number of audio signals including the first audio signal;

determine a first value corresponding to the portion of the first audio signal, the first value representing a ratio of the first signal quality metric value to a sum of signal quality metric values that are associated with the first number of audio signals and the first frequency band;

determine a second value representing a first phase shift of the portion of the first audio signal, the second value determined using a first angle associated with the first direction and a target angle associated with the first number of audio signals; and

generate an output audio signal using the first value, the second value, and the first number of audio signals.

14. The system of claim 13 , wherein the memory further comprises instructions that, when executed by the at least one processor, further cause the system to:

generate a first coefficient value using the second value;

generate a first portion of the output audio signal by multiplying the first value, the first coefficient value, and the portion of the first audio signal;

determine a third value corresponding to the portion of the second audio signal, the third value representing a ratio of the second signal quality metric value to the sum of signal quality metric values;

determine a fourth value representing a second phase shift of the portion of the second audio signal, the fourth value determined using a second angle associated with the second direction and the target angle;

generate a second coefficient value using the fourth value;

generate a second portion of the output audio signal by multiplying the third value by the second coefficient value and the portion of the second audio signal; and

generate the output audio signal by combining the first portion of the output audio signal and the second portion of the output audio signal.

15. The system of claim 13 , wherein the memory further comprises instructions that, when executed by the at least one processor, further cause the system to:

determine that the first signal quality metric value exceeds a first threshold value;

increment a first counter value for the first audio signal;

determine that the second signal quality metric value does not exceed the first threshold value;

determine a third signal quality metric value corresponding to a portion of a third audio signal that is within a second frequency band;

determine that the third signal quality metric value exceeds a second threshold value; and

increment a second counter value for the third audio signal.

16. The system of claim 15 , wherein the memory further comprises instructions that, when executed by the at least one processor, further cause the system to:

determine a first number of highest counter values from a plurality of counter values, the plurality of counter values including the first counter value and the second counter value; and

use the first number of highest counter values to determine the first number of audio signals.

17. The system of claim 13 , wherein the memory further comprises instructions that, when executed by the at least one processor, further cause the system to:

select a third audio signal of the first number of audio signals;

identify the target angle corresponding to the third audio signal;

determine a steering vector using the target angle;

determine a beamformer filter associated with a first portion of the first audio signal; and

determine the second value using the beamformer filter and the steering vector.

18. The system of claim 13 , wherein the memory further comprises instructions that, when executed by the at least one processor, further cause the system to:

determine, during a first time period, that audio data is being sent to one or more loudspeakers;

determine, using a first time constant, a first energy value corresponding to the portion of the first audio signal;

determine, using the first time constant, a second energy value corresponding to the portion of the second audio signal;

determine that the second energy value is lowest of a plurality of energy values associated with the plurality of audio signals and the first frequency band; and

determine the first signal quality metric value using the first energy value and the second energy value.

19. The system of claim 18 , wherein the memory further comprises instructions that, when executed by the at least one processor, further cause the system to:

determine, during a second time period, that the audio data is not being sent to the one or more loudspeakers;

determine, using the first time constant, a third energy value corresponding to a second portion of the first audio signal that is within the first frequency band and associated with the second time period;

determine, using a second time constant that is different than the first time constant, a fourth energy value corresponding to the second portion of the first audio signal; and

determine a third signal quality metric value using the third energy value and the fourth energy value.

20. The system of claim 13 , wherein the memory further comprises instructions that, when executed by the at least one processor, further cause the system to:

determine a third signal quality metric value corresponding to a portion of a third audio signal of the plurality of audio signals, the portion of the third audio signal being within the first frequency band;

determine, using the third signal quality metric value, a threshold value;

determine that the first signal quality metric value is below the threshold value; and

set the first value equal to a value of zero.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 25, 2019
From: MANSOUR, MOHAMED; NAKAGAWA, CARLOS RENATO
To: AMAZON TECHNOLOGIES, INC.
Reel/Frame 050490/0494 →
Cited By (5)
US 12,401,942 US 12,531,048 US 12,548,582 US 12,677,329 US 12,739,564