Face detection guided sound source localization pan angle post processing for smart camera talker tracking and framing

1. A method comprising:

acquiring an image using a camera ( 1116 ), wherein the image comprises pixels arranged in rows and columns;

determining one or more bounding boxes ( 1003 ) corresponding to one or more faces depicted in the image;

determining a plurality of bins ( 602 ) associated with the image, wherein each of the plurality of bins is associated with one or more of the columns;

acquiring a plurality of audio samples ( 1002 ) using a microphone array ( 1214 );

determining sound source localization (SSL) pan angle values ( 1002 ) associated with one or more of the plurality of audio samples;

determining, for respective SSL pan angle values ( 1002 ), one or more corresponding bins of the plurality of bins;

incrementing a count of bin entries ( 606 ) for each of the one or more corresponding bins that is associated with each respective SSL pan angle value;

determining, relative to each bounding box of the one or more bounding boxes, a first range ( 616 , 640 , 650 ), the first range indicative of pixels extending from a first column to a second column;

determining, for the each bounding box, a sum of the count of bin entries ( 618 ) associated with the first range ( 616 , 640 , 650 ) for the each bounding box;

determining, based on the sums of the count of bin entries ( 618 ) of the one or more bounding boxes, a first bounding box having a greatest sum ( 620 ); and

determining output data ( 624 ) indicative of the first bounding box.

2. The method of claim 1 , wherein the determining the plurality of bins ( 602 ) associated with the image comprises:

determining a first pixel width of the image;

determining a second range, wherein the second range includes a range of possible SSL pan angle values associated with output from the microphone array during operation; and

dividing the first pixel width by the second range.

3. The method of claim 1 , further comprising:

processing the one or more of the plurality of audio samples ( 1002 ) using an SSL algorithm ( 1002 ) to determine the SSL pan angle values that are indicative of a direction of a sound.

4. The method of claim 1 , further comprising:

processing the image using a trained machine learning system ( 1003 ) to determine the one or more bounding boxes depicted in the image.

5. The method of claim 1 , further comprising:

determining a first width ( 610 ) of the first bounding box;

determining the first width ( 610 ) of the first bounding box is less than a first value ( 612 , 614 ), wherein the first value ( 612 ) is indicative of a minimum pixel value ( 612 ); and

determining the first range ( 616 ) for the first bounding box by multiplying the first value ( 614 ) by a second value ( 616 ), wherein the second value is a predetermined value ( 616 ).

6. The method of claim 1 , further comprising:

determining a first width ( 610 ) of the first bounding box;

determining the first width ( 610 ) of the first bounding box is greater than a first value ( 612 , 616 ), the first value ( 612 ) is indicative of a minimum pixel value ( 612 ); and

determining the first range ( 616 ) for the first bounding box by multiplying the first width ( 610 ) by a second value ( 616 ), wherein the second value is a predetermined value ( 616 ).

7. The method of claim 1 , further comprising:

determining a reference pixel ( 610 ) for the first bounding box; and

designating a predetermined range of pixels ( 640 ), relative to the reference pixel, as the first range ( 640 ) for the first bounding box.

8. The method of claim 1 , further comprising:

determining a distance ( 650 ) between two or more bounding boxes; and

determining the first range ( 650 ) for at least one of the two or more bounding boxes based on the distance ( 650 ) between the two or more bounding boxes.

9. The method of claim 1 , further comprising:

determining that the sum of the count of bin entries ( 618 ) for the first bounding box is greater than a threshold value ( 622 ); and

sending the output data ( 624 ), responsive to the determining that the sum of the count of bin entries for the first bounding box is greater than the threshold value ( 622 , 624 ).

10. The method of claim 1 , further comprising:

determining a reference pixel ( 610 ) associated with the first bounding box; and

sending the output data ( 624 ), wherein the output data ( 624 ) is indicative of a direction associated with the reference pixel ( 624 ).

11. A system comprising:

a camera ( 1116 );

a microphone array ( 1214 );

a memory ( 1104 , 1204 ) storing computer-executable instructions; and

a hardware processor ( 1102 , 1202 ) coupled to the camera ( 1116 ) and the microphone array ( 1214 ), the hardware processor ( 1102 , 1202 ) configured to execute the computer-executable instructions to:

acquire an image ( 300 ) using the camera ( 1116 ), wherein the image ( 300 ) comprises pixels arranged in rows and columns ( 300 , 400 );

determine one or more bounding boxes ( 404 - 410 ) corresponding to one or more users ( 1 - 4 ) depicted in the image ( 300 );

determine a plurality of bins ( 450 ) associated with the image ( 400 ), wherein each of the plurality of bins is associated with one or more of the columns ( 400 );

acquire a plurality of audio samples ( 1002 ) using the microphone array ( 1214 );

determine sound source localization (SSL) pan angle values ( 1002 ) associated with one or more of the plurality of audio samples ( 1002 );

determine, for respective SSL pan angle values ( 1002 ), one or more corresponding bins ( 450 ) of the plurality of bins ( 450 );

increment a count of bin entries ( 606 ) for each of the one or more corresponding bins ( 450 ) that is associated with each respective SSL pan angle value ( 606 );

determine a first range ( 616 , 640 , 650 ) for each bounding box of the one or more bounding boxes ( 404 - 410 ), the first range ( 616 , 640 , 650 ) indicative of pixels extending from a first column to a second column ( 400 );

determine, for the each bounding box, the count of bin entries ( 618 ) associated with the first range ( 616 , 640 , 650 ) for the each bounding box ( 404 - 410 );

determine a first bounding box ( 404 - 410 ) having a greatest count of entries ( 620 ); and

determine output data ( 620 ) indicative of the first bounding box ( 404 - 410 ).

12. The system of claim 11 , the computer-executable instructions to determine the plurality of bins associated with the image further comprising instructions to:

determine a first pixel width ( 400 ) of the image ( 300 );

determine a second range ( 400 ), wherein the second range ( 400 ) includes a range of possible SSL pan angle values ( 400 ) associated with output ( 1002 ) from the microphone array ( 1214 ) during operation; and

divide ( 602 ) the first pixel width ( 400 ) by the second range ( 400 ).

13. The system of claim 11 , the hardware processor further configured to execute the computer-executable instructions to:

process the image ( 300 ) using a trained machine learning system ( 1003 ) to determine the one or more bounding boxes ( 404 - 410 ) depicted in the image ( 300 ); and

process the one or more of the plurality of audio samples ( 1002 ) using an SSL algorithm ( 1002 ) to determine the SSL pan angle values ( 400 ) that are indicative of a direction of a sound.

14. The system of claim 11 , the hardware processor further configured to execute the computer-executable instructions to:

determine the first range ( 616 , 640 , 650 ) for the first bounding box ( 404 - 410 ) by one of:

multiplying ( 616 ) a first width ( 616 ) of the first bounding box ( 404 - 410 ) by a first value ( 616 ),

using a predetermined range of pixels ( 640 ), or

using a distance ( 650 ) between two or more of the bounding boxes ( 404 - 410 ).

15. The system of claim 11 , the hardware processor further configured to execute the computer-executable instructions to:

send the output data ( 624 ), wherein the output data ( 624 ) is indicative of a direction associated with a user that is speaking ( 624 , 1006 , 1012 ).

16. A method comprising:

acquiring an image using a camera ( 1116 ), wherein the image comprises pixels arranged with respect to a first axis and a second axis;

determining one or more bounding boxes ( 1003 ) corresponding to one or more faces depicted in the image;

determining a plurality of bins ( 602 ) associated with the image, wherein each of the plurality of bins is associated with one or more pixels along the first axis of the image;

acquiring a plurality of audio samples ( 1002 ) using a microphone array ( 1214 );

determining sound source localization (SSL) pan angle values associated with one or more of the plurality of audio samples ( 1002 );

determining, for respective SSL pan angle values ( 1002 ), one or more corresponding bins of the plurality of bins;

incrementing a count of bin entries ( 606 ) for each of the one or more corresponding bins that is associated with each respective SSL pan angle value;

determining, relative to each bounding box of the one or more bounding boxes, a first range ( 616 , 640 , 650 ), the first range ( 616 , 640 , 650 ) indicative of pixels extending from a first pixel along the first axis to a second pixel along the first axis;

determining, for the each bounding box, the count of bin entries ( 618 ) associated with the first range ( 618 ) for the each bounding box;

determining a first bounding box having a greatest count of bin entries ( 620 ); and

sending output data ( 624 ) associated with the first bounding box.

17. The method of claim 16 , wherein:

the first axis specifies columns ( 1002 ), and

the second axis specifies rows ( 1002 ).

18. The method of claim 16 , wherein:

the first axis specifies rows ( 1002 ), and

the second axis specifies columns ( 1002 ).

19. The method of claim 16 , further comprising:

determining, for the each bounding box, a first value ( 618 ) associated with the count of bin entries that are associated with the first range ( 618 ) for the each bounding box;

determining, based on the first values ( 618 ), that the first bounding box has a greatest first value ( 620 ); and

wherein the determining the first bounding box having the greatest count of bin entries ( 620 ) is based on the first bounding box having the greatest first value ( 620 ).

20. The method of claim 16 , wherein the output data ( 624 ) is indicative of a final direction ( 1006 ) associated with a user that is speaking at a time the image is acquired and the plurality of audio samples are acquired.