Updating 3D models of persons
A method for updating a current three dimensional (3D) model of a person, that method may include calculating current locations, within a two-dimensional (2D) space, of current face landmark points of a face of a person within a first image; the calculating is based on the current 3D model, and one or more current acquisition parameters of a 2D camera; wherein the current 3D model of the person is located within a 3D space; calculating second locations, within the 2D space, of second face landmark points of the face of the person within a second image that follows the first image; calculating correspondences between the current locations and the second locations; calculating, based on the correspondences, locations of the second face landmark points within the 3D space; and modifying the current 3D model based on the locations of the second face landmark points within the 3D space.
1. A method for updating a current three dimensional (3D) model of a person, comprising:
(a) participating, using a computerized system of the person, in a virtual 3D video conference;
(b) calculating current locations, within a two-dimensional (2D) space, of current face landmark points of a face of a person within a first image; the calculating is based on the current 3D model, and one or more current acquisition parameters of a 2D camera; wherein the current 3D model of the person is located within a 3D space;
(c) calculating second locations, within the 2D space, of second face landmark points of the face of the person within a second image that follows the first image;
(d) calculating correspondences between the current locations and the second locations;
(e) calculating, based on the correspondences, locations of the second face landmark points within the 3D space; and
(f) modifying the current 3D model based on the locations of the second face landmark points within the 3D space;
wherein steps (b)-(f) are also executed by the computerized device of the person; wherein step (a) also comprises capturing the first image and the second image by a camera of the computerized device of the person;
wherein the method further comprises:
determining parameters of the current 3D model of the person; wherein the parameters differ from image pixels and there are less than three hundred parameters of the current 3D model;
transmitting the parameters of the current 3D model, without transmitting the current 3D model, from the computerized device of the person to computerized devices of other participants of the virtual 3D video conference; wherein the transmitting comprises reducing latency by directly transmitting the parameters to the computerized devices of other participants without sending the parameters via a central server; and
reconstructing an avatar of the person by the computerized devices of the other participants based on the parameters of the current 3D model;
wherein the transmitting the parameters of the current 3D model is preceded by selecting the parameters of the current 3D model, out of a larger group of parameters of the current 3D model, based on (i) an available bandwidth allocated for transmission and (ii) a contribution of the parameters of the current 3D model to a visual perception of the face of the person; where the available bandwidth is several hundred bits per second.
2. The method according to claim 1 wherein the current face landmark points are edge points of the current face landmarks.
3. The method according to claim 1 wherein the current face landmark points are edge points of the current face landmarks and non-edge points of the current face landmarks.
4. The method according to claim 1 wherein the current 3D model comprises a reference 3D model and a current 3D deformation model, wherein the modifying of the current 3D model comprises modifying the current 3D deformation model without substantially modifying the reference 3D model.
5. The method according to claim 4 , comprising determining an update frequency of the 3D model based on the bandwidth.
6. The method according to claim 1 further comprising allocating bitrate, based on the available bandwidth, for a transmission of audio and for the transmission of the parameters of the current 3D model.
7. The method according to claim 1 wherein step (c) comprises segmentation.
8. The method according to claim 1 wherein the parameters define a shape, an expression and a pose of the person.
9. The method according to claim 1 wherein the user computerized device is a laptop without a graphics processing unit (GPU).
10. The method according to claim 1 , wherein the user computerized device is mobile phone.
11. The method according to claim 1 , wherein the parameters are ordered according to importance of the parameters.
12. The method according to claim 1 , wherein a number of bits per parameter is determined according to the available bandwidth allocated to the transmission.
13. A non-transitory computer readable medium for updating a current three dimensional (3D) model of a person, the non-transitory computer readable medium stores instructions for:
(a) participating, using a computerized system of the person, in a virtual 3D video conference;
(b) calculating current locations, within a two-dimensional (2D) space, of current face landmark points of a face of a person within a first image; the calculating is based on the current 3D model, and one or more current acquisition parameters of a 2D camera; wherein the current 3D model of the person is located within a 3D space;
(c) calculating second locations, within the 2D space, of second face landmark points of the face of the person within a second image that follows the first image;
(d) calculating correspondences between the current locations and the second locations;
(e) calculating, based on the correspondences, locations of the second face landmark points within the 3D space; and
(f) modifying the current 3D model based on the locations of the second face landmark points within the 3D space;
wherein steps (b)-(f) are also executed by the computerized device of the person; wherein step (a) also comprises capturing the first image and the second image by a camera of the computerized device of the person;
wherein the non-transitory computer readable medium further stores instructions for:
determining parameters of the current 3D model of the person; wherein the parameters differ from image pixels and there are less than three hundred parameters of the current 3D model;
transmitting the parameters of the current 3D model, without transmitting the current 3D model, from the computerized device of the person to computerized devices of other participants of the virtual 3D video conference; wherein the transmitting comprises reducing latency by directly transmitting the parameters to the computerized devices of other participants without sending the parameters via a central server; and
reconstructing an avatar of the person by the computerized devices of the other participants based on the parameters of the current 3D model;
wherein the transmitting the parameters of the current 3D model is preceded by selecting the parameters of the current 3D model, out of a larger group of parameters of the current 3D model, based on (i) an available bandwidth allocated for transmission and (ii) a contribution of parameters of the current 3D model to a visual perception of the face of the person; and wherein the available bandwidth is several hundred bandwidths per second.
14. The non-transitory computer readable medium according to claim 13 wherein the current face landmark points are edge points of the current face landmarks.
15. The non-transitory computer readable medium according to claim 13 wherein the current face landmark points are edge points of the current face landmarks and non-edge points of the current face landmarks.
16. The non-transitory computer readable medium according to claim 13 wherein the current 3D model comprises a reference 3D model and a current 3D deformation model, wherein the modifying of the current 3D model comprises modifying the current 3D deformation model without substantially modifying the reference 3D model.
17. The non-transitory computer readable medium according to claim 16 , comprising determining an update frequency of the 3D model based on the bandwidth.
18. The non-transitory computer readable medium according to claim 13 that stores instructions for allocating bitrate, based on the available bandwidth, for a transmission of audio and for the transmission of the parameters of the current 3D model.
19. The non-transitory computer readable medium according to claim 13 wherein step (b) comprises segmentation.
20. The non-transitory computer readable medium according to claim 13 wherein the parameters define a shape, an expression and a pose of the person.
21. The non-transitory computer readable medium according to claim 13 , wherein the user computerized device is a laptop without a graphics processing unit (GPU).