Gaze correction of multi-view images

1. A telecommunications device comprising:

a display;

a camera system configured to capture multi-view images of a head;

an image processor arranged to process the multi-view images of a head by:

in each image, identifying image patches containing the left and right eyes of the head, respectively;

in respect of the image patches containing the left eyes of the head in each image of the multi-view images, and also in respect of the image patches containing the right eyes of the head in each image of the multi-view images, performing the steps of:

deriving a feature vector from plural local image descriptors of the image patch in at least one image of the multi-view images; and

deriving a displacement vector field representing a transformation of an image patch, using the derived feature vector to look up reference data comprising reference displacement vector fields associated with possible values of the feature vector; and

adjusting each image of the multi-view images by transforming the image patches containing the left and right eyes of the head in accordance with the derived displacement vector field; and

a telecommunications interface arranged to communicate the adjusted images over a telecommunications network to a destination device for display thereon.

2. A telecommunications device according to claim 1 wherein the camera system is located in a position that is offset from an edge of a display of the electronic device.

3. A telecommunications device according to claim 1 wherein the camera system is located in a position that is adjacent to an edge of a display of the electronic device.

4. A telecommunications device according to claim 1 wherein the camera system comprises at least two cameras.

5. A telecommunications device according to claim 4 wherein the at least two cameras comprise different sensing modalities.

6. A telecommunications device according to claim 4 wherein the at least two cameras comprise a visible light camera and an infrared camera.

7. A telecommunications device according to claim 4 wherein the at least two cameras have different fields of view.

8. A telecommunications device according to claim 4 wherein the at least two cameras are spaced apart by a distance less than average human intrapupilar distance.

9. A telecommunications device according to claim 1 wherein a machine learning algorithm is used to obtain a map from the displacement vector field of each individual to respective feature vectors derived from plural local image descriptors of the image patch.

10. A telecommunications device according to claim 1 , wherein the multi-view images are stereoscopic images.

11. A telecommunications device according to claim 1 , wherein

said plural local image descriptors are plural local image descriptors in each images of the multi-view images, and

said reference data comprising pairs of reference displacement vector fields for each image of the multi-view images, which pairs of reference displacement vector fields are associated with possible values of the feature vector.

12. A telecommunications device according to claim 1 , wherein

said plural local image descriptors are plural local image descriptors in one image of the multi-view images,

said step of deriving displacement vector fields comprises:

deriving a displacement vector field representing a transformation of the image patch in said one image of the multi-view images, using the derived feature vector to look up reference data comprising reference displacement vector fields associated with possible values of the feature vector; and

deriving a displacement vector field representing a transformation of the image patch in the other multi-view image or images by transforming the derived displacement vector field representing a transformation of the image patch in said one image of the multi-view images in accordance with an estimate of the optical flow between the image patches in said one image of the multi-view images and the other multi-view image or images.

13. A telecommunications device according to claim 1 , wherein

said plural local image descriptors are plural local image descriptors in both images of the stereoscopic pair,

said step of deriving displacement vector fields comprises:

deriving an initial displacement vector field representing a notional transformation of a notional image patch in a notional image having a notional camera location relative to the camera locations of the multi-view images, using the derived feature vector to look up reference data comprising reference displacement vector fields associated with possible values of the feature vector; and

deriving displacement vector fields representing a transformation of the image patches in each image of the multi-view images by transforming the initial displacement vector field in accordance with an estimate of the optical flows between the notional image patches in the notional images and the image patches in the images of the multi-view images.

14. A telecommunications device according to claim 13 , wherein the multi-view images are a stereoscopic pair of images and the notional camera location is between the camera locations of the images of the stereoscopic pair.

15. A telecommunications device according to claim 1 , wherein

the step of deriving a displacement vector field comprises deriving displacement vector fields in respect of the image patches in each image of the multi-view images, and

the step of transforming the image patches containing the left and right eyes of the head is performed in accordance with the displacement vector fields derived in respect of the image patches in each image of the multi-view images.

16. A telecommunications device according to claim 1 , wherein

the step of deriving a displacement vector field comprises deriving displacement vector fields in respect of the image patches in each image of the multi-view images, and further deriving a merged displacement vector field from the displacement vector fields derived in respect of the image patches in each image of the multi-view images, and

the step of transforming the image patches containing the left and right eyes of the head is performed in accordance with the merged displacement vector field.

17. A telecommunications device according to claim 1 , wherein

the reference displacement vector fields are further associated with confidence values,

the step of deriving displacement vector fields in respect of the image patches in each image of the multi-view images further comprises deriving a confidence value associated with each derived displacement vector field, and

the merged displacement vector field is an average of the displacement vector fields derived in respect of the image patches in each image of the multi-view images weighted by the derived confidence values.

18. A telecommunications device according to claim 1 , wherein

the device uses location data representing an observer location relative to a display location of the multi-view images,

said reference data comprises plural sets of reference displacement vector fields associated with possible values of the feature vector, which sets are associated with different observer locations, and

said step of deriving displacement vector fields representing a transformation of an image patch is performed using the derived feature vector to look up the set of reference displacement vector fields associated with the observer location represented by the location data.

19. A telecommunications device according to claim 1 , wherein the local image descriptors are local image descriptors derived in a normalized coordinate system, and the reference and derived displacement vector fields are displacement vector fields in the same normalized coordinate system.

20. A telecommunications device according to claim 1 , wherein

the feature vector comprises features that are values derived from the local image descriptors in a discrete space,

the reference displacement vector fields comprise a reference displacement vector field associated with each possible value of the feature vector in the discrete space, and

the step of deriving a displacement vector field for the image patch comprises selecting the reference displacement field associated with the actual value of the derived feature vector.

21. A telecommunications device according to claim 1 , wherein

the feature vector comprises features that are discrete values of the local image descriptors in a continuous space, and

the step of deriving a displacement vector field for the image patch comprises interpolating a displacement vector field from the reference displacement vector fields based on the relationship between the actual value of the derived feature vector and the values of the feature vector associated with the reference displacement vector fields.