Region-based stabilized face tracking

1. A method comprising:

accessing a set of images depicting at least a portion of a real-world object;

identifying a set of regions of the real-world object;

tracking the real-world object across the set of images, the tracking comprising:

solving for rigid pose parameters and expression parameters in alternating optimizations, such that during rigid pose optimization, expression parameters are fixed while the rigid pose parameters are determined and, during expression optimization, rigid pose parameters are fixed while the expression parameters are determined; and

based on the rigid pose parameters and expression parameters, tracking a first region of the set of regions based on a first model and a second region of the set of regions based on a second model; and

overlaying a virtual object over a portion of the real-world object depicted in the image in accordance with tracking the real-world object across the set of images using the first and second models.

2. The method of claim 1 , wherein the object includes a face depicted in the set of images, further comprising:

identifying a set of facial regions of a face, each facial region of the set of facial regions intersecting another facial region with at least one common vertex which is a member of a set of facial vertices; and

adaptively modifying weights generated for each facial region to prioritize tracking of the face based on the weights.

3. The method of claim 2 , wherein identifying the set of facial regions comprises segmenting each facial region in the set of facial regions separately, and for each facial region of the set of facial regions, generating a weight formed from a set of region coefficients, wherein the object is tracked based on the set of facial regions and the weight of each facial region of the set of facial regions.

4. The method of claim 1 , wherein tracking the object comprises applying rigid and non-rigid optimizations to jointly estimate model and the rigid pose parameters.

5. The method of claim 1 , wherein identifying the set of regions comprises:

determining that a first object region in the set of regions corresponds to a first portion of the object that is more flexible than a second portion of the object corresponding to a second object region in the set of regions.

6. The method of claim 5 further comprising:

assigning a first weight to the first object region and a second weight to the second object region; and

adjusting the first weight to be greater than the second weight for optimizing an expression of the object.

7. The method of claim 5 further comprising:

assigning a first weight to the first object region and a second weight to the second object region; and

adjusting the second weight to be greater than the first weight for optimizing a head pose corresponding to the object.

8. The method of claim 5 , wherein the first portion of the object comprises at least one of a cheek, mouth, or eye.

9. The method of claim 5 further comprising performing the tracking until convergence.

10. The method of claim 1 further comprising determining values for weights based on training data comprising a plurality of synthetic facial animation sequences and first and second rigid transformations of the plurality of synthetic facial animation sequences.

11. The method of claim 10 , wherein the first transformation comprises a transformation captured from a video and the second transformation comprises a transformation captured from a static image.

12. The method of claim 1 , wherein a set of region coefficients comprises a rigid coefficient and a non-rigid coefficient, further comprising:

computing the rigid coefficient for each object region while maintaining the non-rigid coefficient at a first fixed value; and

computing the non-rigid coefficient for each object region while maintaining the rigid coefficient at a second fixed value.

13. The method of claim 1 , wherein the virtual object comprises virtual makeup.

14. The method of claim 1 further comprising animating an avatar in accordance with tracking the object across the set of images.

15. A system comprising:

one or more processors; and

a non-transitory processor-readable storage medium storing processor executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:

accessing a set of images depicting at least a portion of a real-world object;

identifying a set of regions of the real-world object;

tracking the real-world object across the set of images, the tracking comprising:

solving for rigid pose parameters and expression parameters in alternating optimizations, such that during rigid pose optimization, expression parameters are fixed while the rigid pose parameters are determined and, during expression optimization, rigid pose parameters are fixed while the expression parameters are determined; and

based on the rigid pose parameters and expression parameters, tracking a first region of the set of regions based on a first model and a second region of the set of regions based on a second model; and

overlaying a virtual object over a portion of the real-world object depicted in the image in accordance with tracking the real-world object across the set of images using the first and second models.

16. The system of claim 15 , wherein the operations further comprise overlaying a graphical object over a portion of the object in accordance with tracking the object across the set of images.

17. The system of claim 16 , wherein the graphical object comprises virtual makeup.

18. The system of claim 15 further comprising animating an avatar in accordance with tracking the object across the set of images.

19. A non-transitory computer-readable medium comprising instructions that, when executed by a processor, configure the processor to perform operations comprising:

accessing a set of images depicting at least a portion of a real-world object;

identifying a set of regions of the real-world object;

tracking the real-world object across the set of images, the tracking comprising:

solving for rigid pose parameters and expression parameters in alternating optimizations, such that during rigid pose optimization, expression parameters are fixed while the rigid pose parameters are determined and, during expression optimization, rigid pose parameters are fixed while the expression parameters are determined; and

based on the rigid pose parameters and expression parameters, tracking a first region of the set of regions based on a first model and a second region of the set of regions based on a second model; and

overlaying a virtual object over a portion of the real-world object depicted in the image in accordance with tracking the real-world object across the set of images using the first and second models.

20. The non-transitory computer-readable medium of claim 19 , wherein the virtual object comprises virtual makeup.