Face detection for video calls
Exemplary embodiments relate to uses of face detection in video, and especially in video calls. In some embodiments, face detection may be used to center a camera shot by maintaining a face in the center of a screen. The centering may be applied selectively, such as by overriding centering if the user is looking off-screen. The video may also be cropped to better fit a face in a screen, or to allow multiple faces to appear on screen. In some embodiments, emphasizing the face over the background (or parts of the face over the whole face) allows for improvement in video call performance. Moreover, these techniques can be used to bring certain areas of a camera shot into focus while de-emphasizing the background (or vice versa).
1. A method, comprising:
accessing one or more frames from a video call, the video call comprising image data and audio data, the one or more frames comprising a face;
performing face detection to detect a portion of the one or more frames that substantially comprises the face;
detecting expressive and non-expressive features in the detected face, the expressive features including one or more eyes and a mouth and the non-expressive features including all other areas of the face;
based on the face detection, assigning an area of the one or more frames not belonging to the face as a background of the one or more frames;
blurring, filtering or reducing in resolution the background and the non-expressive features of the face in the one or more frames to emphasize the expressive features of the face.
2. The method of claim 1 further comprising:
transmitting the one or more frames to a receiving participant in the video call.
3. The method of claim 1 further comprising:
rendering the expressive features of the face at a first resolution;
rendering the non-expressive features of the face at a second resolution, the second resolution being lower than the first resolution; and
rendering the background at third resolution, third resolution being lower than the second resolution.
4. The method of claim 2 further comprising removing the background in the one or more frames prior to transmitting the one or more frames to the receiving participant in lieu of blurring, filtering or reducing in resolution the background.
5. The method of claim 4 further comprising:
transmitting, with the one or more frames, a control signal comprising an identifier of a background to be applied by the receiving participant in the video call.
6. The method of claim 1 wherein the area identified as the background excludes any portion of a person appearing in the frame.
7. An apparatus comprising:
a processor;
memory containing instructions for execution by the processor, the instructions causing the processor to perform functions for facilitating a video call comprising:
accessing one or more frames from a video call, the video call comprising image data and audio data, the one or more frames comprising a face;
performing face detection to detect a portion of the one or more frames that substantially comprises the face;
detecting expressive and non-expressive features in the detected face the expressive features including one or more eyes and a mouth and the non-expressive features including all other areas of the face;
based on the face detection, assigning an area of the one or more frames not belonging to the face as a background of the one or more frames;
blurring, filtering or reducing in resolution the background and the non-expressive features of the face in the one or more frames to emphasize the expressive features of the face.
8. The system of claim 7 , comprising further instructions causing the processor to perform the function of:
transmitting the one or more frames to a receiving participant in the video call.
9. The system of claim 7 comprising further instructions causing the processor to perform the functions of:
rendering the expressive features of the face at a first resolution;
rendering the non-expressive features of the face at a second resolution, the second resolution being lower than the first resolution; and
rendering the background at third resolution, third resolution being lower than the second resolution.
10. The system of claim 8 further comprising removing the background in the one or more frames prior to transmitting the one or more frames to the receiving participant in lieu of blurring, filtering or reducing in resolution the background.
11. The system of claim 10 comprising further instructions causing the processor to perform the functions of:
transmitting, with the one or more frames, a control signal comprising an identifier of a background to be applied by the receiving participant in the video call.
12. A non-transitory, computer-readable storage medium storing instructions configured to cause a processor to:
access one or more frames from a video call, the video call comprising image data and audio data, the one or more frames comprising a face;
perform face detection to detect a portion of the one or more frames that substantially comprises the face;
detect expressive and non-expressive features in the detected face, the expressive features including one or more eyes and a mouth and the non-expressive features including all other areas of the face;
based on the face detection, assign an area of the one or more frames not belonging to the face as a background of the one or more frames;
blurring, filtering or reducing in resolution the background and the non-expressive areas of the face in the one or more frames to emphasize the expressive features of the face.
13. The medium of claim 12 , storing further instructions to cause the processor to:
transmit the one or more frames to a receiving participant in the video call.
14. The medium of claim 12 , storing further instructions to cause the processor to:
render the expressive features of the face at a first resolution;
render the non-expressive features of the face at a second resolution, the second resolution being lower than the first resolution; and
render the background at third resolution, third resolution being lower than the second resolution.
15. The medium of claim 13 , storing further instructions to cause the processor to:
remove the background in the one or more frames prior to transmitting the one or more frames to the receiving participant in lieu of blurring, filtering or reducing in resolution the background; and
transmit, with the one or more frames, a control signal comprising an identifier of a background to be applied by the receiving participant in the video call.