Reconstruction of essential visual cues in mixed reality applications
A mixed reality (MR) simulation system includes a console and a head mounted device (HMD). The MR system captures stereoscopic images from a real-world environment using outward-facing stereoscopic cameras mounted to the HMD. The MR system preprocesses the stereoscopic images to maximize contrast and then extracts a set of features from those images, including edges or corners, among others. For each feature, the MR system generates one or more two-dimensional (2D) polylines. Then, the MR system triangulates between 2D polylines found in right side images and corresponding 2D polylines found in left side images to generate a set of 3D polylines. The MR system interpolates between 3D vertices included in the 3D polylines or extrapolates additional 3D vertices, thereby generating a geometric reconstruction of the real-world environment. The MR system may map textures derived from the real-world environment onto the geometric representation faster than the geometric reconstruction is updated.
1. A computer-implemented method, comprising:
identifying a set of features associated with one or more objects in a real-world scene;
generating a plurality of vertices associated with a set of two-dimensional polylines, the set of two-dimensional (2D) polylines representing the set of features;
generating, based on the plurality of vertices, 2D geometry that represents a two-dimensional view of the real-world scene;
generating three-dimensional (3D) geometry based on the 2D geometry, the 3D geometry comprising a coarse geometric reconstruction of the real-world scene; and
rendering, based on the 3D geometry, a first graphical representation of the real-world scene for display.
2. The computer-implemented method of claim 1 , wherein the coarse geometric reconstruction includes one or more visual cues for navigating the real-world scene.
3. The computer-implemented method of claim 1 , wherein the coarse geometric reconstruction comprises a representation of the real-world scene at a first point in time, and further comprising rendering, based on the 3D geometry and at a second point in time later than the first point in time, a second graphical representation of the real-world scene.
4. The computer-implemented method of claim 1 , wherein the first graphical representation of the real-world scene is displayed via a mixed reality system.
5. The computer-implemented method of claim 1 , wherein identifying the set of features comprises identifying at least one of an edge, a corner, and a surface within a set of images depicting the real-world scene.
6. The computer-implemented method of claim 5 , wherein the set of images comprises a stereoscopic pair of images that includes a left camera image and a right camera image.
7. The computer-implemented method of claim 1 , wherein generating the 2D geometry comprises generating a different 2D polyline for each feature included in the set of features.
8. The computer-implemented method of claim 1 , wherein generating the 3D geometry comprises:
identifying a first polyline included in the 2D geometry;
identifying a second polyline included in the 2D geometry;
triangulating between the first polyline and the second polyline to generate a plurality of depth values; and
generating a 3D polyline based on the first polyline, the second polyline, and the plurality of depth values.
9. The computer-implemented method of claim 1 , wherein generating the 3D geometry comprises:
identifying a first polyline included in the 2D geometry;
identifying a second polyline included in the 2D geometry; and
generating a triangulated mesh based on the first polyline and the second polyline to produce a plurality of 3D vertices for the 3D geometry.
10. The computer-implemented method of claim 9 , wherein the first polyline is generated based on a first image depicting the real-world scene from a first perspective, and wherein the second polyline is generated based on a second image depicting the real-world scene from a second perspective.
11. The computer-implemented method of claim 1 , wherein the graphics are rendered for display based further on at least one attribute of a user.
12. The computer-implemented method of claim 11 , wherein the at least one attribute of the user comprises an interpupillary distance associated with the user.
13. The computer-implemented method of claim 1 , further comprising increasing a locally-adaptive contrast value associated with a set of images depicting the real-world scene to generate a set of preprocessed images.
14. The computer-implemented method of claim 13 , wherein the set of features are identified within the set of preprocessed images.
15. The computer-implemented method of claim 1 , wherein rendering the first graphical representation of the real-world scene comprises:
extracting a texture from a region within an image depicting the real-world scene; and
mapping the texture onto a portion of the coarse geometric reconstruction corresponding to the region.
16. A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform the steps of:
identifying a set of features associated with one or more objects in a real-world scene;
generating a plurality of vertices associated with a set of two-dimensional polylines, the set of two-dimensional (2D) polylines representing the set of features;
generating, based on the plurality of vertices 2D geometry that represents a two-dimensional view of the real-world scene;
generating three-dimensional (3D) geometry based on the 2D geometry, the 3D geometry comprising a coarse geometric reconstruction of the real-world scene; and
rendering, based on the 3D geometry, a first graphical representation of the real-world scene for display.
17. The non-transitory computer-readable medium of claim 16 , further comprising the steps of:
updating the 3D geometry with a first latency; and
mapping a first texture onto a portion of the 3D geometry with a second latency, the first latency exceeding the second latency.
18. The non-transitory computer-readable medium of claim 17 , wherein the first graphical representation is rendered for display with the second latency.
19. A system, comprising:
a memory storing program instructions; and
a processor that executes the instructions to perform the steps of:
identifying a set of features associated with one or more objects in a real-world scene,
generating a plurality of vertices associated with a set of two-dimensional polylines, the set of two-dimensional (2D) polylines representing the set of features,
generating, based on the plurality of vertices 2D geometry that represents a two-dimensional view of the real-world scene,
generating three-dimensional (3D) geometry based on the 2D geometry, the 3D geometry comprising a coarse geometric reconstruction of the real-world scene, and
rendering, based on the 3D geometry, a first graphical representation of the real-world scene for display.
20. The system of claim 19 , wherein the processor performs the steps of:
updating the 3D geometry with a first latency; and
mapping a first texture onto a portion of the 3D geometry with a second latency, the first latency exceeding the second latency.