Edge detection and smoothing for virtual object presentation
Image processing techniques for rendering holographic images are described. Image data captured from cameras can be converted into a triangle mesh. The triangle mesh may be processed by determining the location of edges, normals, and edge distances in the mesh. Mesh and edge smoothing can be performed on the mesh to reduce noise and jagged edges. Additional mesh cleanup and simplification can be performed to reduce the number of vertices and triangles and enhance the qualify of the mesh. Data from the resulting mesh can be transmitted to a receiving device where the mesh data is used to reproduce a holographic image of the image data captured by the depth sensors.
1. A computer-implemented method comprising:
obtaining image data from a pair of cameras;
determining a mesh of triangles from the image data;
performing mesh simplification of the mesh of triangles by:
identifying a pair of vertices in the mesh of triangles that belong to the same triangle;
determining an effect on a shape of the mesh of triangles if the pair of vertices are merged;
determining that the effect on the shape of the mesh of triangles is less than a contribution threshold; and
removing a vertex from the mesh of triangles in response to determining that the effect on the shape of the mesh of triangles is less than the contribution threshold, wherein performing the mesh simplification of the mesh of triangles further comprises:
determining vertices that are the same distance to an edge of the mesh of triangles; and
merging two or more vertices of the vertices that are the same distance to the edge of the mesh of triangles and that are located within a threshold distance of the edge of the mesh of triangles;
storing, in a memory, mesh data that characterizes the mesh of triangles after performing the mesh simplification; and
transmitting the mesh data to a rendering device configured to render a holographic image based on the mesh data.
2. The computer-implemented method of claim 1 , wherein determining the effect on the shape of the mesh of triangles if the pair of vertices are merged comprises:
determining a change in angle between all neighboring triangles before and after the pair of vertices are merged.
3. The computer-implemented method of claim 1 , wherein the pair of vertices are associated with a flat surface and are determined to have an effect on the shape of the mesh of triangles that is less than the contribution threshold.
4. The computer-implemented method of claim 1 , wherein the pair of vertices are associated with a flat surface and are determined to have no effect of the shape of the mesh of triangles.
5. The computer-implemented method of claim 1 , comprising:
identifying one or more triangles in the mesh of triangles with an area of zero in the mesh of triangles; and
removing the one or more triangles.
6. The computer-implemented method of claim 1 , further comprising:
in response to performing the mesh simplification, applying texture projection on the mesh of triangles.
7. The computer-implemented method of claim 6 , wherein applying the texture projection on the mesh of triangles comprises:
determining one or more colors and cameras to use for each vertex in the mesh of triangles;
determining a camera weight for each vertex in the mesh of triangles; and
applying the one or more colors, the cameras, and the camera weights to the mesh of triangles.
8. A virtual or augmented reality system, comprising:
a frame structure comprising a pair of depth sensors configured to obtain image data;
a computing device coupled to the frame structure, the computing device configured to:
determining a mesh of triangles from the image data;
performing mesh simplification of the mesh of triangles by:
identifying a pair of vertices in the mesh of triangles that belong to the same triangle;
determining an effect on a shape of the mesh of triangles if the pair of vertices are merged;
determining that the effect on the shape of the mesh of triangles is less than a contribution threshold; and
removing a vertex from the mesh of triangles in response to determining that the effect on the shape of the mesh of triangles is less than the contribution threshold, wherein performing the mesh simplification of the mesh of triangles further comprises:
determining vertices that are the same distance to an edge of the mesh of triangles; and
merging two or more vertices of the vertices that are the same distance to the edge of the mesh of triangles and that are located within a threshold distance of the edge of the mesh of triangles;
a memory configured to store mesh data that characterizes the mesh of triangles after performing the mesh simplification; and
a headset configured to render a holographic image based on the mesh data.
9. The virtual or augmented reality system of claim 8 , wherein determining the effect on the shape of the mesh of triangles if the pair of vertices are merged comprises:
determining a change in angle between all neighboring triangles before and after the pair of vertices are merged.
10. The virtual or augmented reality system of claim 8 , wherein the pair of vertices are associated with a flat surface and are determined to have no effect of the shape of the mesh of triangles.
11. The virtual or augmented reality system of claim 8 , wherein the computing device is configured to:
identify one or more triangles in the mesh of triangles with an area of zero in the mesh of triangles; and
remove the one or more triangles.
12. The virtual or augmented reality system of claim 8 , wherein:
the computing device is configured to apply texture projection on the mesh of triangles in response to performing the mesh simplification; and
applying the texture projection on the mesh of triangles comprises:
determining one or more colors and cameras to use for each vertex in the mesh of triangles;
determining a camera weight for each vertex in the mesh of triangles; and
applying the one or more colors, the cameras, and the camera weights to the mesh of triangles.
13. A non-transitory computer-readable storage medium comprising instructions, which, when executed by one or more computing devices, cause the one or more computing devices to perform operations comprising:
obtaining image data from a pair of cameras;
determining a mesh of triangles from the image data;
performing mesh simplification of the mesh of triangles by:
identifying a pair of vertices in the mesh of triangles that belong to the same triangle;
determining an effect on a shape of the mesh of triangles if the pair of vertices are merged;
determining that the effect on the shape of the mesh of triangles is less than a contribution threshold; and
removing a vertex from the mesh of triangles in response to determining that the effect on the shape of the mesh of triangles is less than the contribution threshold, wherein performing the mesh simplification of the mesh of triangles further comprises:
determining vertices that are the same distance to an edge of the mesh of triangles; and
merging two or more vertices of the vertices that are the same distance to the edge of the mesh of triangles and that are located within a threshold distance of the edge of the mesh of triangles;
storing, in a memory, mesh data that characterizes the mesh of triangles after performing the mesh simplification; and
transmitting the mesh data to a rendering device configured to render a holographic image based on the mesh data.
14. The non-transitory computer-readable storage medium of claim 13 , wherein the pair of vertices are associated with a flat surface and are determined to have an effect on the shape of the mesh of triangles that is less than the contribution threshold.
15. The non-transitory computer-readable storage medium of claim 13 , wherein the pair of vertices are associated with a flat surface and are determined to have no effect of the shape of the mesh of triangles.
16. The non-transitory computer-readable storage medium of claim 13 , wherein the operations further comprise:
identifying one or more triangles in the mesh of triangles with an area of zero in the mesh of triangles; and
removing the one or more triangles.
17. The non-transitory computer-readable storage medium of claim 13 , wherein the operations further comprise:
in response to performing the mesh simplification, applying texture projection on the mesh of triangles, wherein applying the texture projection on the mesh of triangles comprises:
determining one or more colors and cameras to use for each vertex in the mesh of triangles;
determining a camera weight for each vertex in the mesh of triangles; and
applying the one or more colors, the cameras, and the camera weights to the mesh of triangles.