Method and system for image processing in video conferencing

1. A method for image processing in video conferencing, for correcting the gaze of an interlocutor in an image or a sequence of images captured by at least one real camera, comprising the steps of:

the at least one real camera acquiring an original image of the interlocutor and localizing the perimeter of the face of the interlocutor;

synthesizing a corrected view of only the interlocutor's face, as seen by a virtual camera, the virtual camera being located on the interlocutor's line of sight and oriented towards the interlocutor;

transferring the corrected view of only the interlocutor's face from the synthesized view into the original image by determining a seam line between the corrected view and the original image, for cutting the interlocutor's face from the corrected image along the seam line, and blending the corrected view and the original image along a seam line, thereby generating a final image, while leaving the image outside of the identified perimeter of the interlocutor's face unchanged for purposes of interlocutor-gaze correction; and

at least one of displaying the final image and transmitting the final image.

2. The method of claim 1 , wherein the step of synthesizing the corrected view of the interlocutor's face comprises mapping the original image onto a 3D model of the interlocutor's face, and rendering the 3D model from a virtual camera placed along an estimate of the interlocutor's line of sight.

3. The method of claim 1 or claim 2 , wherein the step of transferring the corrected view of the interlocutor's face from the synthesized view into the original image comprises determining the seam line as an optimal seam.

4. The method of claim 1 , comprising the steps of, in a calibration phase, determining transformation parameters for a geometrical transform relating the position and orientation of the real camera and the virtual camera by:

displaying the final image to the interlocutor; and

accepting user input from the interlocutor to adapt the transformation parameters until the final image is satisfactory.

5. The method of claim 1 , comprising the steps of, in a calibration phase, determining transformation parameters for a geometrical transform relating the position and orientation of the real camera and the virtual camera by:

acquiring a first depth map of at least the interlocutor's face when the interlocutor is looking at the real camera;

acquiring a second depth map of at least the interlocutor's face when the interlocutor is looking at a display screen that is placed to display an image of an interlocutor's conference partner; and

computing the transformation parameters from a transform that relates the first and second depth maps.

6. The method of claim 1 , comprising the steps of, in a calibration phase, adapting a 2D translation vector for positioning the corrected view of the interlocutor's face in the original image by:

displaying the final image to the interlocutor; and

accepting user input from the interlocutor to adapt the (2D) translation vector until the final image is satisfactory.

7. The method of claim 2 , comprising the step of estimating the 3D model of the interlocutor's face by adapting a generic 3D model to facial features recognized in one or more original images.

8. The method of claim 2 , comprising the step of using a generic 3D model of the interlocutor's face without adapting it to one or more original images.

9. The method of claim 2 , comprising the steps of:

acquiring, for each original image, an associated depth map comprising the face of the interlocutor; and

determining the 3D model of the interlocutor's face based on the depth map.

10. The method of claim 9 , comprising the step of temporal smoothing of the 3D position of face tracking vertices over a sequence of depth maps by:

estimating the 3D position of the interlocutor's head in each depth map; and

combining the 3D position of the vertices as observed in a current depth map with a prediction of their position computed from their position in at least one preceding depth map and from the change in the head's 3D position and orientation.

11. The method of claim 9 , comprising the steps of:

identifying the 3D location of the interlocutor's eyeballs using a face tracker;

approximating the shape of the eyeballs by a sphere; and

using, in the depth map at the location of the eyes, this approximation in place of the acquired depth map information.

12. The method of claim 9 , comprising the step of smoothing the depth map comprising the face of the interlocutor.

13. The method of claim 12 , wherein said smoothing of the depth map uses Laplacian smoothing.

14. The method of claim 9 , comprising one or both of the steps of:

artificially extending the depth map comprising the face of the interlocutor at its boundaries; and

filling holes within the depth map.

15. A data processing system for image processing in video conferencing, for correcting the gaze of an interlocutor in an image or a sequence of images captured by at least one real camera, comprising at least one real camera for acquiring an image of the interlocutor, the system being programmed to execute the method according to claim 1 .

16. A non-transitory, computer-readable medium comprising computer-readable program code encoding a computer program that, when loaded and executed on a computer, causes the computer to execute the method according to claim 1 .

17. The method of claim 3 , wherein the optimal seam is a seam line between the corrected view and the original image that minimizes a sum of differences between the corrected view and the original image along the seam line.

18. The method of claim 17 , wherein determining the optimal seam line comprises the steps of:

starting with either:

an ellipsoidal polygon fitted to chin points determined by a face tracker, or

with a polygon determined for a previous image of a sequence of images; and

adapting vertex points of the polygon in order to minimize the sum of differences.

19. The method of claim 18 , wherein said step of adapting vertex points is limited to only vertices of an upper part of the polygon, corresponding to the upper portion of the interlocutor's face above said chin points.