Automatic mesh tracking for 3D face modeling
View Patent ↗The mesh tracking described herein involves mesh tracking on 3D face models. In contrast to existing mesh tracking algorithms which generally require user intervention and manipulation, the mesh tracking algorithm is fully automatic once a template mesh is provided. In addition, an eye and mouth boundary detection algorithm is able to better reconstruct the shape of eyes and mouths.
1. A method programmed in a non-transitory memory of a device comprising:
inputting unaligned 3D scans;
implementing pose correction via rigid alignment on the unaligned 3D scans to generate aligned meshes;
detecting eye and mouth boundaries on the aligned meshes;
implementing mesh tracking on the aligned meshes; and
outputting a tracked mesh based on the mesh tracking,
wherein detecting the eye and mouth boundaries includes 3D contour detection, wherein the 3D contour detection includes:
applying a mask Regional Convolutional Neural Network (RCNN) which results in a segmentation probability, left and right corners, and a region of interest;
generating an edge map from the region of interest; and
using an improved active contour fitting snake algorithm.
2. The method of claim 1 wherein implementing the mesh tracking utilizes Wrap3.
3. The method of claim 1
wherein the rigid alignment is performed to remove head movement using coded targets.
4. The method of claim 1
wherein detecting the eye and mouth boundaries includes analyzing a view of a mesh to perform Region Of Interest (ROI) detection.
5. The method of claim 1 wherein the snake algorithm involves minimizing E snake where E snake =E corner +E edge +E region +E int .
6. The method of claim 1 wherein implementing the mesh tracking includes
dividing a video into short clips to reduce drifting,
performing forward and backward tracking within each clip, and
merging the forward and backward clips together.
7. The method of claim 1 further comprising eyelid correction through mesh deformation.
8. An apparatus comprising:
a non-transitory memory for storing an application, the application for:
inputting unaligned 3D scans;
implementing pose correction via rigid alignment on the unaligned 3D scans to generate aligned meshes;
detecting eye and mouth boundaries on the aligned meshes;
implementing mesh tracking on the aligned meshes; and
outputting a tracked mesh based on the mesh tracking;
wherein detecting the eye and mouth boundaries includes 3D contour detection, wherein the 3D contour detection includes:
applying a mask Regional Convolutional Neural Network (RCNN) which results in a segmentation probability, left and right corners, and a region of interest;
generating an edge map from the region of interest; and
using an improved active contour fitting snake algorithm a processor coupled to the memory, the processor configured for processing the application.
9. The apparatus of claim 8
wherein implementing the mesh tracking utilizes Wrap3.
10. The apparatus of claim 8
wherein the rigid alignment is performed to remove head movement using coded targets.
11. The apparatus of claim 8
wherein detecting the eye and mouth boundaries includes analyzing a view of a mesh to perform Region Of Interest (ROI) detection.
12. The apparatus of claim 8 wherein the snake algorithm involves minimizing E snake where E snake =E corner +E edge +E region +E int .
13. The apparatus of claim 8
wherein implementing the mesh tracking includes dividing a video into short clips to reduce drifting, performing forward and backward tracking within each clip, and
merging the forward and backward clips together.
14. The apparatus of claim 8 further comprising eyelid correction through mesh deformation.
15. A system comprising:
one or more cameras for acquiring three dimensional content; and
a device for processing the three dimensional content by:
receiving three dimensional content including unaligned 3D scans;
implementing pose correction via rigid alignment on the unaligned 3D scans to generate aligned meshes;
detecting eye and mouth boundaries on the aligned meshes;
implementing mesh tracking on the aligned meshes; and
outputting a tracked mesh based on the mesh tracking,
wherein detecting the eye and mouth boundaries includes 3D contour detection, wherein the 3D contour detection includes:
applying a mask Regional Convolutional Neural Network (RCNN) which results in a segmentation probability, left and right corners, and a region of interest;
generating an edge map from the region of interest; and
using an improved active contour fitting snake algorithm.
16. The system of claim 15
wherein implementing the mesh tracking utilizes Wrap3.
17. The system of claim 15
wherein the rigid alignment is performed to remove head movement using coded targets.
18. The system of claim 15
wherein detecting the eye and mouth boundaries includes analyzing a view of a mesh to perform Region Of Interest (ROI) detection.
19. The system of claim 15 wherein the snake algorithm involves minimizing E snake where E snake =E corner +E edge +E region +E int .
20. The system of claim 15
wherein implementing the mesh tracking includes dividing a video into short clips to reduce drifting, performing forward and backward tracking within each clip, and merging the forward and backward clips together.
21. The system of claim 15 further comprising eyelid correction through mesh deformation.
22. A method programmed in a non-transitory memory of a device comprising:
inputting unaligned 3D scans;
implementing pose correction via rigid alignment on the unaligned 3D scans to generate aligned meshes;
detecting eye and mouth boundaries on the aligned meshes;
implementing mesh tracking on the aligned meshes, wherein implementing the mesh tracking includes dividing a video into short clips to reduce drifting, performing forward and backward tracking within each clip, and merging the forward and backward clips together; and
outputting a tracked mesh based on the mesh tracking.
23. The method of claim 22 wherein implementing the mesh tracking utilizes Wrap3.
24. The method of claim 22 wherein the rigid alignment is performed to remove head movement using coded targets.
25. The method of claim 22 wherein detecting the eye and mouth boundaries includes 3D contour detection.
26. The method of claim 25 wherein detecting the eye and mouth boundaries includes analyzing a view of a mesh to perform Region Of Interest (ROI) detection.
27. The method of claim 22 further comprising eyelid correction through mesh deformation.
28. An apparatus comprising: a non-transitory memory for storing an application, the application for:
inputting unaligned 3D scans;
implementing pose correction via rigid alignment on the unaligned 3D scans to generate aligned meshes;
detecting eye and mouth boundaries on the aligned meshes;
implementing mesh tracking on the aligned meshes, wherein implementing the mesh tracking includes dividing a video into short clips to reduce drifting, performing forward and backward tracking within each clip, and merging the forward and backward clips together; and
outputting a tracked mesh based on the mesh tracking; and
a processor coupled to the memory, the processor configured for processing the application.
29. The apparatus of claim 28 wherein implementing the mesh tracking utilizes Wrap3.
30. The apparatus of claim 28 wherein the rigid alignment is performed to remove head movement using coded targets.
31. The apparatus of claim 28 wherein detecting the eye and mouth boundaries includes 3D contour detection.
32. The apparatus of claim 31 wherein detecting the eye and mouth boundaries includes analyzing a view of a mesh to perform Region Of Interest (ROI) detection.
33. The apparatus of claim 28 further comprising eyelid correction through mesh deformation.
34. A system comprising:
one or more cameras for acquiring three dimensional content; and
a device for processing the three dimensional content by:
receiving three dimensional content including unaligned 3D scans;
implementing pose correction via rigid alignment on the unaligned 3D scans to generate aligned meshes;
detecting eye and mouth boundaries on the aligned meshes;
implementing mesh tracking on the aligned meshes,
wherein implementing the mesh tracking includes dividing a video into short clips to reduce drifting, performing forward and backward tracking within each clip, and merging the forward and backward clips together; and
outputting a tracked mesh based on the mesh tracking.
35. The system of claim 34 wherein implementing the mesh tracking utilizes Wrap3.
36. The system of claim 34 wherein the rigid alignment is performed to remove head movement using coded targets.
37. The system of claim 34 wherein detecting the eye and mouth boundaries includes 3D contour detection.
38. The system of claim 37 wherein detecting the eye and mouth boundaries includes analyzing a view of a mesh to perform Region Of Interest (ROI) detection.
39. The system of claim 34 further comprising eyelid correction through mesh deformation.