Binaural data sharing in ear-worn devices using neural networks
Described herein is binaural data sharing technology for ear-worn devices to improve audio processing performance. Different embodiments may include sharing of various data types, such as processed microphone signals, beamformed signals, neural network products (e.g., masks), and environmental metrics. For beamforming, devices may combine signals from both ears for improved directional selectivity or process separate beamformed signals independently. Devices may be configured to generate identical masks or average mask magnitude portions while preserving device-specific phase components. Neural networks may be trained to handle mixed-latency data, processing current local data with “stale” data from the other device. Environmental metrics like signal-to-noise ratios may be shared for coordinated responses to acoustic conditions. The technology may also apply to integrated devices like eyeglasses.
1 . A system, comprising:
a first ear-worn device comprising:
first neural network circuitry, and
first communication circuitry; and
a second ear-worn device comprising:
second neural network circuitry, and
second communication circuitry;
wherein:
the first communication circuitry and the second communication circuitry are
configured to communicate over a wireless communication link;
the first neural network circuitry is configured to:
receive one or more first audio signals generated by the first ear-worn device, and
implement one or more first neural network layers, wherein the first neural network circuitry is configured to use the one or more first neural network layers to generate a first neural network product based on the one or more first audio signals;
the second neural network circuitry is configured to:
receive one or more second audio signals generated by the second ear-worn device, and
implement one or more second neural network layers, wherein the second neural network circuitry is configured to use the one or more second neural network layers to generate a second neural network product based on the one or more second audio signals;
the first communication circuitry is configured to:
transmit, to the second communication circuitry over the wireless communication link, first data comprising or originating from the first neural network product, and
receive, from the second communication circuitry over the wireless communication link, second data comprising or originating from the second neural network product;
the first data comprises a first mask and the second data comprises a second mask, or the first data comprises a processed version of the first mask and the second data comprises a processed version of the second mask;
the first ear-worn device is configured to combine the first mask with the second mask, thereby generating a first combined mask;
the first ear-worn device is configured, when combining the first mask with the second mask, to combine a magnitude portion of the first mask with a magnitude portion of the second mask; and
the first combined mask comprises:
a magnitude portion based on combining the magnitude portion of the first mask with the magnitude portion of the second mask, and
a phase portion based on a phase portion of the first mask.
2 . The system of claim 1 , wherein the first ear-worn device is configured, when combining the magnitude portion of the first mask with the magnitude portion of the second mask, to average the magnitude portion of the first mask with the magnitude portion of the second mask.
3 . The system of claim 1 , wherein the second ear-worn device is configured to combine the first mask with the second mask, thereby generating a second combined mask.
4 . The system of claim 3 , wherein the first combined mask and the second combined mask are the same.
5 . The system of claim 3 , wherein magnitude portions of the first combined mask and the second combined mask are the same.
6 . The system of claim 3 , wherein:
the first ear-worn device is configured to apply the first combined mask to one of the one or more first audio signals;
the second ear-worn device is configured to apply the second combined mask to one of the one or more second audio signals;
the one of the one or more first audio signals comprises a beamformed audio signal; and
the one of the one or more second audio signals comprises a beamformed audio signal.
7 . The system of claim 3 , wherein:
the first ear-worn device is configured to apply the first combined mask to one of the one or more first audio signals;
the second ear-worn device is configured to apply the second combined mask to one of the one or more second audio signals; and
the one of the one or more first audio signals and the one of the one or more second audio signals are different.
8 . The system of claim 1 , wherein the first ear-worn device is configured to apply the first combined mask to an audio signal received by the first ear-worn device subsequently to when the one or more first audio signals are received.
9 . The system of claim 1 , wherein:
the second data comprises the processed version of the second mask; and
the first ear-worn device is configured to generate the second mask from the second data using decoding or interpolation.
10 . The system of claim 1 , wherein the second data comprises an encoded version of the second neural network product.
11 . A system, comprising:
a first ear-worn device comprising:
first neural network circuitry, and
first communication circuitry; and
a second ear-worn device comprising:
second neural network circuitry, and
second communication circuitry;
wherein:
the first communication circuitry and the second communication circuitry are configured to communicate over a wireless communication link;
the first neural network circuitry is configured to:
receive one or more first audio signals generated by the first ear-worn device, and
implement one or more first neural network layers, wherein the first neural network circuitry is configured to use the one or more first neural network layers to generate a first neural network product based on the one or more first audio signals;
the second neural network circuitry is configured to:
receive one or more second audio signals generated by the second ear-worn device, and
implement one or more second neural network layers, wherein the second neural network circuitry is configured to use the one or more second neural network layers to generate a second neural network product based on the one or more second audio signals;
the first communication circuitry is configured to:
transmit, to the second communication circuitry over the wireless communication link, first data comprising or originating from the first neural network product, and
receive, from the second communication circuitry over the wireless communication link, second data comprising or originating from the second neural network product;
the first data comprises a first mask and the second data comprises a second mask, or the first data comprises a processed version of the first mask and the second data comprises a processed version of the second mask;
the first ear-worn device is configured to compare the first mask with the second mask;
the first ear-worn device further comprises mixing circuitry configured to perform mixing of at least two audio signals, thereby generating an output audio signal; and
based on the comparison, the mixing circuitry is further configured to modulate weighting of the at least two audio signals in the mixing.
12 . The system of claim 11 , wherein:
the second data comprises the processed version of the second mask; and
the first ear-worn device is configured to generate the second mask from the second data using decoding or interpolation.
13 . The system of claim 11 , wherein the second data comprises an encoded version of the second neural network product.
14 . A system, comprising:
a first ear-worn device comprising:
one or more first microphones configured to generate one or more first microphone signals,
first processing circuitry comprising first neural network circuitry, the first processing circuitry configured to process the one or more first microphone signals, thereby generating first data, and
first communication circuitry; and
a second ear-worn device comprising:
one or more second microphones configured to generate one or more second microphone signals,
second processing circuitry comprising second neural network circuitry, the second processing circuitry configured to process the one or more second microphone signals, thereby generating second data, and
second communication circuitry;
wherein:
the first communication circuitry and the second communication circuitry are configured to communicate over a wireless communication link;
the first communication circuitry is configured to:
transmit the first data to the second communication circuitry over the wireless communication link, and
receive the second data from the second communication circuitry over the wireless communication link;
the second communication circuitry is configured to:
transmit the second data to the first communication circuitry over the wireless communication link, and
receive the first data from the first communication circuitry over the wireless communication link;
the first neural network circuitry is configured to:
implement one or more first neural network layers configured to receive inputs comprising or originating from the first data and the second data;
the second neural network circuitry is configured to:
implement one or more second neural network layers configured to receive the inputs comprising or originating from the first data and the second data; and
based on the first ear-worn device receiving the second data and the second ear-worn device receiving the first data, the first neural network circuitry and the second neural network circuitry are configured to generate same neural network products having same values.
15 . The system of claim 14 , wherein the first neural network circuitry is configured to input the second data or the processed version thereof to the at least one of the one or more first neural network layers when processing audio signals received subsequent to the one or more first audio signals.
16 . The system of claim 14 , wherein the first neural network circuitry is configured to use the one or more first neural network layers to decode the second data.
17 . The system of claim 14 , wherein the second data comprises an encoded version of the second neural network product.
18 . The system of claim 14 , wherein the first neural network circuitry and the second neural network circuitry are configured, when generating the same neural network products having the same values, to generate masks having same magnitude portions.
19 . The system of claim 14 , wherein the first neural network circuitry and the second neural network circuitry are configured, when generating the same neural network products having the same values, to generate masks having same magnitude portions and different phase portions.
20 . A system, comprising:
a first ear-worn device comprising:
one or more first microphones configured to generate one or more first microphone signals, first processing circuitry comprising first neural network circuitry, the first processing circuitry configured to process the one or more first microphone signals, thereby generating first data, and
first communication circuitry; and
a second ear-worn device comprising:
one or more second microphones configured to generate one or more second microphone signals,
second processing circuitry comprising second neural network circuitry, the second processing circuitry configured to process the one or more second microphone signals, thereby generating second data, and
second communication circuitry;
wherein:
the first communication circuitry and the second communication circuitry are configured to communicate over a wireless communication link;
the first communication circuitry is configured to:
transmit the first data to the second communication circuitry over the wireless communication link, and
receive the second data from the second communication circuitry over the wireless communication link; and
the first neural network circuitry is configured to:
implement one or more first neural network layers configured to receive inputs comprising or originating from the first data and the second data, and
generate a neural network product based on the first data and the second data,
wherein the second data originates from an earlier frame of audio data than the first data.
21 . The system of claim 20 , wherein the second data comprises an encoded version of the second neural network product.
22 . The system of claim 20 , wherein the first data and the second data are generated at least 2-20 milliseconds apart.