Identifying planes in artificial reality systems
View Patent ↗In one embodiment, a method includes accessing an image of a physical world scene, detecting a number of straight-line segments in the accessed image, identifying a first vanishing point and a second vanishing point in the image, where each vanishing point corresponds to a sub-set of the number of straight-line segments, and where the first vanishing point and the second vanishing point are orthogonal, identifying a planar region in the accessed image represented by one or more straight-line segments associated with the first vanishing point and one or more straight-line segments associated with the second vanishing point, transforming a virtual object associated with the planar region based on one or more properties associated with the planar region, and displaying the transformed virtual object over the image.
1. A method comprising:
by a computing device, accessing an image of a physical world scene;
by the computing device, detecting a plurality of straight-line segments in the accessed image;
by the computing device, identifying a first vanishing point and a second vanishing point in the image, wherein each vanishing point corresponds to a sub-set of the plurality of straight-line segments, wherein the first vanishing point and the second vanishing point are orthogonal;
by the computing device, identifying a planar region in the accessed image based on one or more straight-line segments associated with the first vanishing point and one or more straight-line segments associated with the second vanishing point;
by the computing device, transforming a virtual object associated with the planar region based on one or more properties associated with the planar region;
by the computing device, displaying the transformed virtual object over the image;
by the computing device, tracking one or more straight-line segments corresponding to the first vanishing point in a following frame; and
by the computing device, maintaining a relative size of the transformed virtual object to sizes of the tracked one or more straight-line segments in the following frame.
2. The method of claim 1 , wherein detecting the plurality of straight-line segments in the accessed image comprises:
detecting edges in the accessed image, wherein an edge is a set of points at which image brightness has discontinuities; and
filtering non-straight edges from the detected edges in the accessed image.
3. The method of claim 2 , wherein edges in the accessed image represent boundaries of objects, boundaries of material properties, or boundaries of lighting.
4. The method of claim 2 , wherein edges in the accessed image comprise step edges and bar edges.
5. The method of claim 1 , wherein the first vanishing point or the second vanishing point is a finite vanishing point, at which lines extended from the corresponding sub-set of the plurality of straight-line segments intersect in a 2-dimensional plane comprising the image.
6. The method of claim 1 , wherein the first vanishing point or the second vanishing point is an infinite vanishing point, for which lines extended from the corresponding sub-set of the plurality of straight-line segments remain parallel in the image.
7. The method of claim 1 , wherein identifying the first vanishing point or the second vanishing point comprises clustering the plurality of straight-line segments.
8. The method of claim 1 , wherein identifying the first vanishing point or the second vanishing point comprises utilizing random sample consensus (RANSAC) to generate candidate vanishing points.
9. The method of claim 1 , wherein identifying the first vanishing point or the second vanishing point is based on Inertial Measurement Unit (IMU) information available on the computing device.
10. The method of claim 1 , wherein point clouds from Simultaneous Localization and Mapping (SLAM) are used to detect a depth of the planar region.
11. The method of claim 1 , wherein the virtual object is a 2-dimensional object.
12. The method of claim 1 , wherein the virtual object is a 3-dimensional object.
13. The method of claim 1 , further comprising:
moving the virtual object to a new position within the planar region based on a command from a user, wherein the virtual object is transformed in a way to keep a relative orientation of the virtual object to an orientation of the planar region and a relative size of the virtual object to a size of the planar region same after moving.
14. The method of claim 1 , further comprising:
re-sizing the virtual object within the planar region based on a command from a user, wherein the virtual object is transformed in a way to keep a relative orientation of the virtual object to an orientation of the planar region and a placement position of the virtual object same after re-sizing.
15. The method of claim 1 , wherein the planar region is tracked in subsequent frames by utilizing a planar tracking algorithm.
16. The method of claim 15 , further comprising:
detecting an orientation of the planar region changes in a following frame; and
transforming the virtual object to maintain a relative orientation of the virtual object to the orientation of the planar region consistent.
17. The method of claim 15 , further comprising:
detecting scale of the planar region changes in a following frame; and
transforming the virtual object to maintain a relative size of the virtual object to a size of the planar region consistent.
18. One or more computer-readable non-transitory storage media embodying software that is operable when executed to:
access an image of a physical world scene;
detect a plurality of straight-line segments in the accessed image;
identify a first vanishing point and a second vanishing point in the image, wherein each vanishing point corresponds to a sub-set of the plurality of straight-line segments, and wherein the first vanishing point and the second vanishing point are orthogonal;
identify a planar region in the accessed image based on one or more straight-line segments associated with the first vanishing point and one or more straight-line segments associated with the second vanishing point;
transform a virtual object associated with the planar region based on one or more properties associated with the planar region;
display the transformed virtual object over the image;
track one or more straight-line segments corresponding to the first vanishing point in a following frame; and
maintain a relative size of the transformed virtual object to sizes of the tracked one or more straight-line segments in the following frame.
19. The media of claim 18 , wherein detecting the plurality of straight-line segments in the accessed image comprises:
detecting edges in the accessed image, wherein an edge is a set of points at which image brightness has discontinuities; and
filtering non-straight edges from the detected edges in the accessed image.
20. A system comprising: one or more processors; and a non-transitory memory coupled to the processors comprising instructions executable by the processors, the processors operable when executing the instructions to:
access an image of a physical world scene;
detect a plurality of straight-line segments in the accessed image;
identify a first vanishing point and a second vanishing point in the image, wherein each vanishing point corresponds to a sub-set of the plurality of straight-line segments, and wherein the first vanishing point and the second vanishing point are orthogonal;
identify a planar region in the accessed image based on one or more straight-line segments associated with the first vanishing point and one or more straight-line segments associated with the second vanishing point;
transform a virtual object associated with the planar region based on one or more properties associated with the planar region;
display the transformed virtual object over the image;
track one or more straight-line segments corresponding to the first vanishing point in a following frame; and
maintain a relative size of the transformed virtual object to sizes of the tracked one or more straight-line segments in the following frame.