Granted Patent B2

US 12,598,286 · App. 18/526,726 · Granted Apr 7, 2026

Inventors: Yingen Xiong (Mountain View, CA); Christopher A. Peri (Mountain View, CA)

Assignee: Samsung Electronics Co., Ltd.

H04N13/344G06T19/006H04N13/128H04N13/239

Abstract

Claims (89)

1 . A method comprising:

obtaining (i) images of a scene captured using a stereo pair of imaging sensors of an extended reality (XR) device and (ii) depth data associated with the images, the scene including one or more first objects and one or more second objects;

obtaining volume-based three-dimensional (3D) models of the first and second objects included in the scene, the volume-based 3D models including: (i) one or more 3D models of the one or more first objects and (ii) one or more 3D models of the one or more second objects;

for the one or more first objects, performing depth-based reprojection of the one or more 3D models of the one or more first objects to a left virtual view and a right virtual view based on one or more depths of the one or more first objects;

for the one or more second objects, performing constant-depth reprojection of the one or more 3D models of the one or more second objects to the left virtual view and the right virtual view based on a specified depth; and

rendering the left virtual view and the right virtual view for presentation by the XR device.

2 . The method of claim 1 , wherein obtaining the volume-based 3D models of the first and second objects in the scene comprises, for each of the first and second objects:

determining whether a corresponding volume-based 3D model for the object exists in a library of previously-generated volume-based 3D models; and

one of:

responsive to determining that the corresponding volume-based 3D model for the object does exist in the library, obtaining the corresponding volume-based 3D model from the library; or

responsive to determining that the corresponding volume-based 3D model for the object does not exist in the library, generating a new volume-based 3D model representing the object.

3 . The method of claim 2 , wherein generating the new volume-based 3D model representing the object comprises:

generating a dense depth map based on the images of the scene, the depth data, and position data associated with the XR device;

performing volume reconstruction based on the dense depth map; and

generating the new volume-based 3D model representing the object based on the volume reconstruction and at least one color texture captured in the images.

4 . The method of claim 2 , wherein:

the library of previously-generated volume-based 3D models is configured to store 3D models of scenes, real-world objects, and virtual objects; and

each of the 3D models in the library comprises a 3D mesh representing a geometry of an associated scene or object, a color texture map representing at least one color of the associated scene or object, and one or more parameters identifying at least one of a pattern, dimensions, a pose, or a transformation of the associated scene or object.

5 . The method of claim 1 , wherein:

the one or more first objects are positioned within a threshold distance from the XR device;

the one or more second objects are positioned beyond the threshold distance from the XR device; and

the method further comprises identifying the threshold distance using a trained machine learning model.

6 . The method of claim 1 , wherein:

the method further comprises identifying a focus region of a user;

the one or more first objects are positioned within the focus region of the user; and

the one or more second objects are positioned outside the focus region of the user.

7 . The method of claim 1 , wherein:

the one or more first objects comprise one or more foreground objects in the scene; and

the one or more second objects comprise one or more background objects in the scene.

8 . An extended reality (XR) device comprising:

at least one display;

imaging sensors configured to capture images of a scene, the scene including one or more first objects and one or more second objects; and

at least one processing device configured to:

obtain (i) the images of the scene and (ii) depth data associated with the images;

obtain volume-based three-dimensional (3D) models of the first and second objects included in the scene, the volume-based 3D models including: (i) one or more 3D models of the one or more first objects and (ii) one or more 3D models of the one or more second objects;

for the one or more first objects, perform depth-based reprojection of the one or more 3D models of the one or more first objects to a left virtual view and a right virtual view based on one or more depths of the one or more first objects;

for the one or more second objects, perform constant-depth reprojection of the one or more 3D models of the one or more second objects to the left virtual view and the right virtual view based on a specified depth; and

render the left virtual view and the right virtual view for presentation by the at least one display.

9 . The XR device of claim 8 , wherein, to obtain the volume-based 3D models of the first and second objects in the scene, the at least one processing device is configured, for each of the first and second objects, to:

determine whether a corresponding volume-based 3D model for the object exists in a library of previously-generated volume-based 3D models; and

one of:

responsive to determining that the corresponding volume-based 3D model for the object does exist in the library, obtain the corresponding volume-based 3D model from the library; or

responsive to determining that the corresponding volume-based 3D model for the object does not exist in the library, generate a new volume-based 3D model representing the object.

10 . The XR device of claim 9 , wherein, to generate the new volume-based 3D model representing the object, the at least one processing device is configured to:

generate a dense depth map based on the images of the scene, the depth data, and position data associated with the XR device;

perform volume reconstruction based on the dense depth map; and

generate the new volume-based 3D model representing the object based on the volume reconstruction and at least one color texture captured in the images.

11 . The XR device of claim 9 , wherein:

the library of previously-generated volume-based 3D models is configured to store 3D models of scenes, real-world objects, and virtual objects; and

each of the 3D models in the library comprises a 3D mesh representing a geometry of an associated scene or object, a color texture map representing at least one color of the associated scene or object, and one or more parameters identifying at least one of a pattern, dimensions, a pose, or a transformation of the associated scene or object.

12 . The XR device of claim 8 , wherein:

the one or more first objects are positioned within a threshold distance from the XR device;

the one or more second objects are positioned beyond the threshold distance from the XR device; and

the at least one processing device is further configured to identify the threshold distance using a trained machine learning model.

13 . The XR device of claim 8 , wherein:

the at least one processing device is further configured to identify a focus region of a user;

the one or more first objects are positioned within the focus region of the user; and

the one or more second objects are positioned outside the focus region of the user.

14 . The XR device of claim 8 , wherein:

the one or more first objects comprise one or more foreground objects in the scene; and

the one or more second objects comprise one or more background objects in the scene.

15 . A non-transitory machine readable medium containing instructions that when executed cause at least one processor to:

obtain (i) images of a scene captured using a stereo pair of imaging sensors of an extended reality (XR) device and (ii) depth data associated with the images, the scene including one or more first objects and one or more second objects;

obtain volume-based three-dimensional (3D) models of the first and second objects included in the scene, the volume-based 3D models including: (i) one or more 3D models of the one or more first objects and (ii) one or more 3D models of the one or more second objects;

for the one or more first objects, perform depth-based reprojection of the one or more 3D models of the one or more first objects to a left virtual view and a right virtual view based on one or more depths of the one or more first objects;

for the one or more second objects, perform constant-depth reprojection of the one or more 3D models of the one or more second objects to the left virtual view and the right virtual view based on a specified depth; and

render the left virtual view and the right virtual view for presentation by the XR device.

16 . The non-transitory machine readable medium of claim 15 , wherein the instructions that when executed cause the at least one processor to obtain the volume-based 3D models of the first and second objects in the scene comprise:

instructions that when executed cause the at least one processor, for each of the first and second objects, to:

determine whether a corresponding volume-based 3D model for the object exists in a library of previously-generated volume-based 3D models; and

one of:

responsive to determining that the corresponding volume-based 3D model for the object does exist in the library, obtain the corresponding volume-based 3D model from the library; or

responsive to determining that the corresponding volume-based 3D model for the object does not exist in the library, generate a new volume-based 3D model representing the object.

17 . The non-transitory machine readable medium of claim 16 , wherein the instructions that when executed cause the at least one processor to generate the new volume-based 3D model representing the object comprise:

instructions that when executed cause the at least one processor to:

generate a dense depth map based on the images of the scene, the depth data, and position data associated with the XR device;

perform volume reconstruction based on the dense depth map; and

generate the new volume-based 3D model representing the object based on the volume reconstruction and at least one color texture captured in the images.

18 . The non-transitory machine readable medium of claim 16 , wherein:

the library of previously-generated volume-based 3D models is configured to store 3D models of scenes, real-world objects, and virtual objects; and

each of the 3D models in the library comprises a 3D mesh representing a geometry of an associated scene or object, a color texture map representing at least one color of the associated scene or object, and one or more parameters identifying at least one of a pattern, dimensions, a pose, or a transformation of the associated scene or object.

19 . The non-transitory machine readable medium of claim 15 , wherein:

the one or more first objects are positioned within a threshold distance from the XR device;

the one or more second objects are positioned beyond the threshold distance from the XR device; and

the instructions when executed further cause the at least one processor to identify the threshold distance using a trained machine learning model.

20 . The non-transitory machine readable medium of claim 15 , wherein:

the instructions when executed further cause the at least one processor to identify a focus region of a user;

the one or more first objects are positioned within the focus region of the user; and

the one or more second objects are positioned outside the focus region of the user.

Assignments (1)

Continuity (3)

References Cited (20)