Granted Patent B2

US 12,469,310 · App. 18/733,184 · Granted Nov 11, 2025

Inventor: Chun-Wei Chan (Foster City, CA)

Assignee: Meta Platforms Technologies, LLC

G06V20/647G06T7/55G06T7/74G06T11/001G06T15/04G06T19/20G06V10/74G06T2207/10016G06T2207/10024G06T2207/20081G06T2219/004G06T2219/2008

Abstract

Claims (56)

1 . A computing system for generating an artificial reality environment, the computing system comprising:

one or more processors; and

one or more memories storing instructions that, when executed by the one or more processors, cause the computing system to:

receive one or more source images with depicted objects;

select one or more objects, of the depicted objects, wherein the selected one or more objects exclude one or more depicted objects identified as being transient;

create two or more virtual 3D objects by generating 3D models for each of the selected one or more objects;

identify, based on the one or more source images, relative locations for each of the two or more virtual 3D objects; and

compile the two or more virtual 3D objects, based on the identified relative locations, into the artificial reality environment, wherein the compiling the two or more virtual 3D objects includes reducing space between the two or more virtual 3D objects, while maintaining relative directions between the two or more virtual 3D objects consistent from their identified relative locations.

2 . The computing system of claim 1 ,

wherein the one or more source images are one or more of (a) a photograph, (b) a video frame, (c) a drawing, or (d) any combination thereof.

3 . The computing system of claim 1 ,

wherein the selecting the one or more objects comprises matching object identifiers, specified for the one or more source images, to tags defined for candidate virtual objects in a virtual object library.

4 . The computing system of claim 3 ,

wherein the instructions, when executed, further cause the computing system to generate the object identifiers by applying a machine learning model trained to match image data to object identifiers.

5 . The computing system of claim 1 ,

wherein the one or more source images comprises a plurality of images derived from a plurality of photographs and/or video frames.

6 . The computing system of claim 5 ,

wherein a relative location, of the relative locations, between two of the two or more virtual 3D objects is determined based on an automatic determination of relative positioning of two or more of the depicted objects included in the plurality of photographs and/or video frames.

7 . The computing system of claim 5 ,

wherein a relative location, of the relative locations, between two of the two or more virtual 3D objects is determined based on metadata for one or more of the plurality of photographs and/or video frames.

8 . The computing system of claim 5 ,

wherein a relative location, of the relative locations, between two of the two or more virtual 3D objects is determined based on an identification of at least one of the depicted objects as being related to a real-world object with a known geographical position.

9 . The computing system of claim 1 ,

wherein the generating at least one of the 3D models is performed by applying at least portions of representations of the one or more source images to a machine learning model.

10 . The computing system of claim 9 ,

wherein the generating at least one of the 3D models is further performed by obtaining, in response to the applying the machine learning model, depth data and converting at least a part of depth data into a 3D mesh.

11 . The computing system of claim 10 ,

wherein the machine learning model is trained to generate depth data for portions of images based on training data comprising flat images paired with corresponding depth profiles.

12 . The computing system of claim 1 ,

wherein the generating the 3D models is performed in part by applying texture data to 3D representations, wherein the texture data is generated based on portions of the one or more source images respectively corresponding to each of the virtual objects.

13 . A non-transitory computer-readable storage medium storing instructions, for generating an artificial reality environment, the instructions, when executed by a computing system, cause the computing system to:

receive one or more source images with depicted objects;

select one or more objects, of the depicted objects, wherein the one or more first virtual objects exclude one or more depicted objects identified as being transient;

create two or more virtual 3D objects by generating 3D models for each of the selected one or more objects;

identify relative locations for each of the two or more virtual 3D objects; and

compile two or more virtual 3D objects, based on the identified relative locations, into the artificial reality environment, wherein the compiling the two or more virtual 3D objects includes reducing space between the two or more virtual 3D objects, while maintaining relative directions between the two or more virtual 3D objects consistent from their identified relative locations.

14 . The non-transitory computer-readable storage medium of claim 13 ,

wherein the selecting the one or more objects comprises matching object identifiers, specified for the one or more source images, to tags defined for candidate virtual objects in a virtual object library; and

wherein the instructions, when executed, further cause the computing system to generate the object identifiers by applying a machine learning model trained to match image data to object identifiers.

15 . The non-transitory computer-readable storage medium of claim 13 ,

wherein the one or more source images comprises a plurality of images derived from a plurality of photographs and/or video frames.

16 . The non-transitory computer-readable storage medium of claim 15 ,

wherein a relative location, of the relative locations, between two of the two or more virtual 3D objects is determined based on an automatic determination of relative positioning of two or more of the depicted objects included in the plurality of photographs and/or video frames.

17 . The non-transitory computer-readable storage medium of claim 15 ,

wherein a relative location, of the relative locations, between two of the two or more virtual 3D objects is determined based on an identification of at least one of the depicted objects as being related to a real-world object with a known geographical position.

18 . A method for generating an artificial reality environment, the method comprising:

receiving one or more source images with depicted objects;

selecting one or more objects, of the depicted objects, wherein the one or more first virtual objects exclude one or more depicted objects identified as being transient;

creating two or more virtual 3D objects by generating 3D models for each of the selected one or more objects;

identifying relative locations for each of the two or more virtual 3D objects; and

compiling two or more virtual 3D objects, based on the identified relative locations, into the artificial reality environment, wherein the compiling the two or more virtual 3D objects includes reducing space between the two or more virtual 3D objects, while maintaining relative directions between the two or more virtual 3D objects consistent from their identified relative locations.

19 . The method of claim 18 ,

wherein the generating at least one of the 3D models is performed by applying at least portions of representations of the one or more source images to a machine learning model.

20 . The method of claim 19 ,

wherein the generating at least one of the 3D models is further performed by obtaining, in response to the applying the machine learning model, depth data and converting at least a part of depth data into a 3D mesh; and

wherein the machine learning model is trained to generate depth data for portions of images based on training data comprising flat images paired with corresponding depth profiles.

Continuity (3)

References Cited (229)