Method for real-time video processing involving changing features of an object in the video

1. A method, comprising:

detecting at least a portion of an object in frames of a video;

obtaining a mean face based on a picture with a fixed size, the mean face being related to the object;

obtaining a current observation of the object using the video;

computing an observation error based on a square of a difference between a logarithm of a function of the current observation of the object and the logarithm of the function of the mean face with the fixed size, the portion of the object being detected based on the observation error;

transforming a feature of the portion of the object within the frames of the video to generate modified frames with a modified feature, the feature associated with an element of the portion of the object and the feature being transformed in the modified frames within the video in which the portion of the object is detected while the video is provided at a computing device and the portion of the object is detected in the video; and

providing the modified frames including the modified feature.

2. The method of claim 1 , wherein transforming the feature comprises:

generating a mesh based on one or more characteristic points of the portion of the object; and

transforming the feature based on the mesh and the one or more characteristic points of the portion of the object.

3. The method of claim 2 , wherein generating the mesh comprises:

generating a first set of points on the mesh for characteristic points associated with the element of the portion of the object;

generating a second set of points on the mesh based on the set of first points and a modification to be applied in generating the modified feature; and

transforming the frames of the video based on the second set of points on the mesh.

4. The method of claim 3 , further comprising:

receiving a request for modification representing a modification to be applied to the feature; and

generating the second set of points on the mesh based on the request for modification and the set of first points.

5. The method of claim 1 , further comprising:

identifying an area on the portion of the object in the video, the area corresponding to the feature of the portion of the object; and

transforming the area on the portion of the object the object within the frames of the video to generate modified frames with at least one modified area.

6. The method of claim 5 , further comprising:

generating a mesh based on one or more characteristic points of the portion of the object;

generating a first set of points on the mesh for characteristic points associated with the element of the portion of the object and identifying the area based on the first set of points generated on the mesh; and

transforming the area on the portion of the object within the frames of the video based on the first set of points on the mesh.

7. The method of claim 1 , further comprising:

generating a mesh based on one or more characteristics points of the portion of the object;

generating a grid associated with a background of the video; and

transforming the feature based on the mesh and the one or more characteristic points of the portion of the object while maintaining the background of the video based on the grid.

8. A system, comprising:

one or more processors; and

a non-transitory processor-readable storage medium storing processor executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:

detecting at least a portion of an object in frames of a video;

obtaining a mean face based on a picture with a fixed size, the mean face being related to the object;

obtaining a current observation of the object using the video;

computing an observation error based on a square of a difference between a logarithm of a function of the current observation of the object and the logarithm of the function of the mean face with the fixed size, the portion of the object being detected based on the observation error;

transforming a feature of the portion of the object within the frames of the video to generate modified frames with a modified feature, the feature associated with an element of the portion of the object and the feature being transformed in the modified frames within the video in which the portion of the object is detected while the video is provided at a computing device and the portion of the object is detected in the video; and

providing the modified frames including the modified feature.

9. The system of claim 8 , wherein transforming the feature comprises:

generating a mesh based on one or more characteristic points of the portion of the object; and

transforming the feature based on the mesh and the one or more characteristic points of the portion of the object.

10. The system of claim 9 , wherein generating the mesh comprises:

generating a first set of points on the mesh for characteristic points associated with the element of the portion of the object;

generating a second set of points on the mesh based on the set of first points and a modification to be applied in generating the modified feature; and

transforming the frames of the video based on the second set of points on the mesh.

11. The system of claim 10 , wherein the operations further comprise:

receiving a request for modification representing a modification to be applied to the feature; and

generating the second set of points on the mesh based on the request for modification and the set of first points.

12. The system of claim 8 , wherein the operations comprise:

identifying an area on the portion of the object in the video, the area corresponding to the feature of the portion of the object; and

transforming the area on the portion of the object within the frames of the video to generate modified frames with at least one modified area.

13. The method of claim 1 , further comprising:

initializing a tracking process to detect the portion of the object in the video;

from time-to-time, re-initializing the tracking process to re-detect the portion of the object in the video to continue modifying the feature.

14. The method of claim 1 , wherein the portion of the object is detected based on shape units intensity vector, action units intensity vector and a position vector, the position vector indicating vertical positions of eyebrows, eyes, nose and mouth, the position vector further indicating widths of eyes, mouth and chin, the action units intensity vector indicating facial movement comprising upper lip raising, lip stretching, brow rising, and brow lowering.

15. A non-transitory processor-readable storage medium storing processor executable instructions that, when executed by a processor of a machine, cause the machine to perform operations comprising:

detecting at least a portion of an object in frames of a video;

obtaining a mean face based on a picture with a fixed size, the mean face being related to the object;

obtaining a current observation of the object using the video;

computing an observation error based on a square of a difference between a logarithm of a function of the current observation of the object and the logarithm of the function of the mean face with the fixed size, the portion of the object being detected based on the observation error;

transforming a feature of the portion of the object within the frames of the video to generate modified frames with a modified feature, the feature associated with an element of the portion of the object and the feature being transformed in the modified frames within the video in which the portion of the object is detected while the video is provided at a computing device and the portion of the object is detected in the video; and

providing the modified frames including the modified feature.

16. The non-transitory processor-readable storage medium of claim 15 , wherein transforming the feature comprises:

generating a mesh based on one or more characteristic points of the portion of the object; and

transforming the feature based on the mesh and the one or more characteristic points of the portion of the object.

17. The method of claim 1 , further comprising:

obtaining grayscale values of each pixel of the portion of the object in the video;

applying a logarithm to the grayscale values of each pixel of the portion of the object in the video before tracking the portion of the object; and

tracking the portion of the object in the video based on the logarithm of the grayscale values.

18. The method of claim 1 , wherein the portion of the object is detected based on an eye separation distance, a mouth width, a chin width and action unit information indicating a jaw drop and lip corner depression.

19. The method of claim 1 , further comprising:

generating a plurality of square grids associated with one or more regions; and

modifying the one or more regions based on the plurality of square grids to distort the one or more regions.

20. The method of claim 1 , further comprising:

generating a square grid associated with a background of the video; and

applying distortion to the background of the video based on the square grid.