Speaker beam-steering based on microphone array and depth camera assembly input

1. A device comprising:

a plurality of speakers;

a plurality of microphones configured to capture audio data from a local area surrounding the plurality of microphones;

a depth camera assembly configured to capture video data of the local area and to determine depth information identifying depths from the depth camera assembly to locations within the local area;

a controller coupled to the plurality of microphones, to the depth camera assembly, and to the plurality of speakers, the controller configured to:

determine, from the audio data captured by the plurality of microphones, a distance relative to an axis perpendicular to a center of the device of a location in the local area from which the plurality of microphones captured audio data;

identifying an object within the location in the local area from which the plurality of microphones captured audio data and from video data captured by the depth camera assembly;

determine a depth from the depth camera assembly to the identified object from depth information captured by the depth camera assembly; and

direct audio output by each of the speakers to the identified object at the distance relative to the axis perpendicular to the center of the device of the location in the local area from which the plurality of microphones captured audio data and at the depth from the depth camera assembly by:

determining a shading function based on a product of a probability determined by the controller and an angle from an axis normal to a center of a speaker located in a center of the plurality of speakers to the location in the local area from which the plurality of microphones captured audio data; and

applying the shading function to the audio output by the plurality of speakers.

2. The device of claim 1 , wherein the depth camera assembly comprises a pair of cameras separated by a specific distance and positioned side by side.

3. The device of claim 1 , wherein the depth camera assembly comprises a camera and a depth sensor.

4. The device of claim 1 , wherein the depth camera assembly comprises a camera, and the controller is configured to determine depth from the depth camera assembly to the identified object from video data captured by the camera.

5. The device of claim 1 , wherein the plurality of microphones comprises a first order differential microphone array.

6. The device of claim 5 , wherein the first order differential microphone array comprises a center microphone and other microphones having equal distances from the center microphone.

7. The device of claim 1 , wherein the plurality of microphones comprises a second order differential microphone array.

8. The device of claim 7 , wherein the second order differential microphone array comprises at least six microphones arranged in a hexagonal pattern.

9. The device of claim 1 , wherein the controller is further configured to:

determine a change in location of the identified object within the local area from data captured by the depth camera assembly;

determine an updated distance relative to the axis perpendicular to the center of the device and the changed location of the identified object;

determine an updated depth from the depth camera assembly to the identified object in the changed location; and

redirect audio output by each of the speakers to the identified object at the changed location within the local area from the updated distance and the updated depth.

10. The device of claim 1 , wherein the controller is further configured to:

determine a change in location of the identified object within the local area from data captured by the depth camera assembly;

determine an updated distance relative to the axis perpendicular to the center of the device and the changed location of the identified object;

redirect audio output by each of the speakers to the identified object at the changed location within the local area from the updated distance and at the depth.

11. The device of claim 1 , wherein the controller is further configured to:

determine an updated depth from the depth camera assembly to the identified object; and

redirect audio output by each of the speakers to the identified object at the distance relative to the axis perpendicular to the center of the device of the location in the local area from which the plurality of microphones captured audio data and at the updated depth.

12. The device of claim 1 , further comprising a subwoofer coupled to the controller.

13. A computer program product comprising a non-transitory computer readable storage medium having instructions encoded thereon that, when executed by a processor, cause the processor to:

identify an object within a local area of a device from data captured by one or more sensors included in the device, the captured data including audio data and video data;

determine a depth from the device to the identified object from video data captured by a sensor included in the device;

determine a distance relative to an axis perpendicular to a center of the device of a location in the local area from which one or more microphones included in the device captured the audio data;

and direct audio output by each of a plurality of speakers of the device to the identified object at the determined depth and at the distance relative to an axis perpendicular to a center of the device of a location in the local area from which one or more microphones included in the device captured the audio data by

determining a shading function based on a product of a probability determined by the processor and an angle from an axis normal to a center of a speaker located in a center of the plurality of speakers to the determined depth and a determined location of arrival or the audio data captured by the one or more microphones; and

applying the shading function to the audio output by the plurality of speakers.

14. The non-transitory computer readable storage medium of claim 13 , wherein the sensor comprises a camera.

15. The non-transitory computer readable storage medium of claim 13 , wherein the sensor comprises a depth camera assembly.

16. A computer program product comprising a non-transitory computer readable storage medium having instructions encoded thereon that, when executed by a processor, cause the processor to:

determine, from audio data captured by a plurality of microphones included in a device, a distance relative to an axis perpendicular to a center of the device of a location in a local area from which the plurality of microphones included in the device captured the audio data;

obtain video data of the local area from a depth camera assembly and depth information identifying depths from the depth camera assembly to locations within the local area from the depth camera assembly identify an object within the location in the local area from which the plurality of microphones captured audio data from data captured by the depth camera assembly;

determine a depth from the depth camera assembly to the identified object from data captured by the depth camera assembly; and

direct audio output by each of a plurality of speakers of the device to the identified object at the distance relative to the axis perpendicular to the center of the device of the location in the local area from which the plurality of microphones captured audio data and to the depth from the depth camera assembly to the identified object by

determining a shading function based on a product of a probability determined by the processor and an angle from an axis normal to a center of a speaker located in a center of the plurality of speakers to the location in the local area from which the plurality of microphones captured audio data; and

applying the shading function to the audio output by the plurality of speakers.

17. The computer program product of claim 16 , wherein the non-transitory computer readable storage medium further has instructions encoded thereon that, when executed by the processor, cause the processor to:

determine a change in location of the identified object within the local area from data captured by the depth camera assembly;

determine an updated distance relative to the axis perpendicular to the center of the device and the changed location of the identified object;

determine an updated depth from the depth camera assembly to the identified object in the changed location; and

redirect audio output by each of the plurality of speakers to the identified object at the changed location within the local area from the updated distance and the updated depth.

18. The computer program product of claim 16 , wherein the non-transitory computer readable storage medium further has instructions encoded thereon that, when executed by the processor, cause the processor to:

determine a change in location of the identified object within the local area from data captured by the depth camera assembly;

determine an updated distance relative to the axis perpendicular to the center of the device and the changed location of the identified object;

redirect audio output by each of the plurality of speakers to the identified object at the changed location within the local area from the updated distance and at the depth.

19. The computer program product of claim 16 , wherein the non-transitory computer readable storage medium further has instructions encoded thereon that, when executed by the processor, cause the processor to:

determine an updated depth from the depth camera assembly to the identified object; and

redirect audio output by each of the plurality of speakers to the identified object at the distance relative to the axis perpendicular to the center of the device of the location in the local area from which the plurality of microphones captured audio data and at the updated depth.

20. A computer program product comprising a non-transitory computer readable storage medium having instructions encoded thereon that, when executed by a processor, cause the processor to:

determine, from audio data captured by a plurality of microphones included in a device, a distance relative to an axis perpendicular to a center of the device of a location in a local area from which the plurality of microphones included in the device captured the audio data;

obtain video data of the local area from a depth camera assembly and depth information identifying depths from the depth camera assembly to locations within the local area from the depth camera assembly

identify an object within the location in the local area from which the plurality of microphones captured audio data from data captured by the depth camera assembly;

determine a depth from the depth camera assembly to the identified object from data captured by the depth camera assembly; and

direct audio output by each of a plurality of speakers of the device to the identified object at the distance relative to the axis perpendicular to the center of the device of the location in the local area from which the plurality of microphones captured audio data and to the depth from the depth camera assembly to the identified object by

for each of the plurality of speakers, determine an angle from an axis perpendicular to a center of a speaker to the location in the local area from which the plurality of microphones captured audio data;

determine a weighting function for each of the plurality of speakers, the weighting function for a speaker based on the angle determined for the speaker and having a maximum at the determined distance and at the determined depth; and

for each of the plurality of speakers, apply the weighting function corresponding the speaker to audio output by the speaker.

21. The computer program product of claim 20 , wherein the non-transitory computer readable storage medium further has instructions encoded thereon that, when executed by the processor, cause the processor to:

determine a change in location of the identified object within the local area from data captured by the depth camera assembly;

determine an updated distance relative to the axis perpendicular to the center of the device and the changed location of the identified object;

determine an updated depth from the depth camera assembly to the identified object in the changed location; and

redirect audio output by each of the plurality of speakers to the identified object at the changed location within the local area from the updated distance and the updated depth.

22. The computer program product of claim 20 , wherein the non-transitory computer readable storage medium further has instructions encoded thereon that, when executed by the processor, cause the processor to:

determine a change in location of the identified object within the local area from data captured by the depth camera assembly;

determine an updated distance relative to the axis perpendicular to the center of the device and the changed location of the identified object;

redirect audio output by each of the plurality of speakers to the identified object at the changed location within the local area from the updated distance and at the depth.

23. The computer program product of claim 20 , wherein the non-transitory computer readable storage medium further has instructions encoded thereon that, when executed by the processor, cause the processor to:

determine an updated depth from the depth camera assembly to the identified object; and

redirect audio output by each of the plurality of speakers to the identified object at the distance relative to the axis perpendicular to the center of the device of the location in the local area from which the plurality of microphones captured audio data and at the updated depth.