Granted Patent B2

US 12,307,575 · App. 17/969,622 · Granted May 20, 2025

Inventors: Gioacchino Noris (Zurich, CH); Sony Nguyen (San Jose, CA); Andrea Alejandra Cohen (Redwood City, CA); Anush Mohan (San Jose, CA); Matthew Banks (Kirkland, WA)

Assignee: Meta Platforms Technologies, LLC

G06T15/06G06T15/08G06T17/00G06T19/006G06T19/20G06T2200/24G06T2210/12G06T2219/004G06T2219/2004

Abstract

Claims (70)

1. A method comprising, by a computing system:

initiating a scene capture process to capture a scene of a physical environment surrounding a user wearing an artificial-reality system, the scene comprising one or more of planes or objects;

sending a first set of instructions to the user to outline one or more planes of the scene;

casting a first set of rays to outline the one or more planes according to the first set of instructions;

creating one or more plane anchors corresponding to the one or more planes based on outlining by the first set of rays;

sending a second set of instructions to the user to outline one or more objects of the scene;

casting a second set of rays to outline the one or more objects according to the second set of instructions;

creating one or more object anchors corresponding to the one or more objects based on outlining by the second set of rays;

generating a scene model of the scene based on the one or more plane anchors and the one or more object anchors; and

providing a subset of the one or more plane anchors and/or the one or more object anchors, of the scene model, to a client application, responsive to a query by the client application for the subset of the one or more plane anchors and/or the one or more object anchors,

wherein the client application generates an artificial reality environment using the subset of the one or more plane anchors and/or the one or more object anchors of the scene model.

2. The method of claim 1 , wherein the one or more planes comprise walls, and wherein casting the first set of rays to outline the one or more planes according to the first set of instructions comprises:

casting a first ray to put a first point on a bottom corner of a first wall according to a first instruction of the first set of instructions;

casting a second ray to put a second point on a top corner on the same side of the first wall according to a second instruction of the first set of instructions; and

casting subsequent rays to put subsequent points on top corners of each subsequent wall present in the scene according to a third instruction of the first set of instructions.

3. The method of claim 2 , wherein creating the one or more planes comprises:

creating one or more two dimensional (2D) bounded boxes for the one or more planes based on the first point, the second point, and the subsequent points defined by the first ray, the second ray, and the subsequent rays, respectively.

4. The method of claim 1 , wherein the one or more objects comprise a desk, and wherein casting the second set of rays to outline the one or more one or more objects according to the second set of instructions comprises:

casting a first ray to put a first point on a floor directly below a top left corner of the desk;

casting a second ray to put a second point on the top left corner of the desk;

casting a third ray to put a third point on a top right corner of the desk; and

casting a fourth ray to put a fourth point on a corner directly behind the third point.

5. The method of claim 4 , wherein creating the one or more planes comprises:

creating one or more three dimensional (3D) volumes for the one or more objects based on the first point, the second point, third point, and the fourth point defined by the first ray, the second ray, third ray, and the fourth ray, respectively.

6. The method of claim 1 , wherein generating the scene model comprises:

saving the one or more plane anchors and the one or more object anchors;

grouping a first set of the one or more plane anchors into a first component;

grouping a second set of the one or more plane anchors into a second component;

grouping the one or more plane anchors and the one or more object anchors into a third component; and

associating, with each anchor, a component type and a semantic type.

7. The method of claim 1 , wherein the scene model is used by the client application or a user to add one or more augmented reality elements to the scene.

8. The method of claim 1 , wherein each casted ray of the first set of rays and/or the second set of rays places a point at a particular location based on an instruction of the first set of instructions and/or the second set of instructions.

9. The method of claim 8 , wherein the one or more plane anchors or the one or more object anchors are created based on points placed by casted rays at particular locations in the physical environment of the scene.

10. The method of claim 1 , wherein each ray of the first set of rays and/or the second set of rays is cast via a controller of the artificial-reality system.

11. The method of claim 1 , wherein the artificial-reality system is a virtual reality headset.

12. The method of claim 1 , wherein the query further requests one or more additional components of the scene model, and wherein the method further comprises:

determining that the one or more additional components of the scene model are not found;

reinitiating the scene capture process in response to determining that the one or more additional components of the scene model are not found; and

providing the one or more additional components to the client application,

wherein the client application generates the artificial reality environment further using the one or more additional components.

13. The method of claim 1 , wherein the scene is a living room of the user.

14. The method of claim 13 , wherein the one or more planes comprise one or more walls, a ceiling, a floor, one or more windows, one or more doors, or any combination thereof.

15. The method of claim 13 , wherein the one or more objects comprise a couch, a desk, a television, a bed, a plant, a chair, or any combination thereof.

16. The method of claim 1 , wherein the scene capture process is initiated by the client application running on the artificial-reality system.

17. The method of claim 16 , wherein the client application is a first-party application or a third-party application on the artificial-reality system.

18. The method of claim 1 , wherein the client application is an existing application on the artificial-reality system.

19. One or more computer-readable non-transitory storage media embodying software that is operable when executed to:

initiate a scene capture process to capture a scene of a physical environment surrounding a user wearing an artificial-reality system, the scene comprising one or more of planes or objects;

send a first set of instructions to the user to outline one or more planes of the scene;

cast a first set of rays to outline the one or more planes according to the first set of instructions;

create one or more plane anchors corresponding to the one or more planes based on outlining by the first set of rays;

send a second set of instructions to the user to outline one or more objects of the scene;

cast a second set of rays to outline the one or more objects according to the second set of instructions;

create one or more object anchors corresponding to the one or more objects based on outlining by the second set of rays;

generate a scene model of the scene based on the one or more plane anchors and the one or more object anchors, and

provide a subset of the one or more plane anchors and/or the one or more object anchors, of the scene model, to a client application, responsively to a query by the client application for the subset of the one or more plane anchors and/or the one or more object anchors,

wherein the client application generates an artificial reality environment using the subset of the one or more plane anchors and/or the one or more object anchors of the scene model.

20. A system comprising:

one or more processors; and

one or more computer-readable non-transitory storage media coupled to one or more of the processors and comprising instructions operable when executed by one or more of the processors to cause the system to:

initiate a scene capture process to capture a scene of a physical environment surrounding a user wearing an artificial-reality system, the scene comprising one or more of planes or objects;

send a first set of instructions to the user to outline one or more planes of the scene;

cast a first set of rays to outline the one or more planes according to the first set of instructions;

create one or more plane anchors corresponding to the one or more planes based on outlining by the first set of rays;

send a second set of instructions to the user to outline one or more objects of the scene;

cast a second set of rays to outline the one or more objects according to the second set of instructions;

create one or more object anchors corresponding to the one or more objects based on outlining by the second set of rays;

generate a scene model of the scene based on the one or more planes and the one or more objects; and

providing a subset of the one or more plane anchors and/or the one or more object anchors, of the scene model, to a client application, responsive to a query by the client application for the subset of the one or more plane anchors and/or the one or more object anchors,

wherein the client application generates an artificial reality environment using the subset of the one or more plane anchors and/or the one or more object anchors of the scene model.

Assignments (1)

Continuity (2)

References Cited (65)