Granted Patent B2

US 12,518,407 · App. 18/586,111 · Granted Jan 6, 2026

Inventors: Kazim Selim Engin (Weehawken, NJ); Jun-Jee Chao (Minneapolis, MN); Ibrahim Volkan Isler (Saint Paul, MN); Nikhil Narsingh Chavan-Dafle (Jersey City, NJ); Bhoram Lee (Princeton, NJ)

Assignee: SAMSUNG ELECTRONICS CO., LTD.

G06T7/33G06T5/77G06T7/13G06T7/70G06T15/20G06V20/64G06T2207/10016G06T2207/10028

Abstract

Claims (56)

1 . An information processing apparatus for generating a composite map from image data and spatial map data, the information processing apparatus comprising:

at least one memory storing instructions; and

at least one processor operatively connected to the at least one memory, the at least one processor being configured to execute the instructions to:

acquire a spatial map of an environment;

acquire a plurality of images of a portion of the environment;

generate a three-dimensional (3D) image of the portion of the environment using the plurality of images;

identify a floor in the 3D image;

generate a synthetic spatial map of the portion of the environment based on the floor in the 3D image;

determine a location of the portion of the environment within the spatial map by identifying a region of the spatial map that corresponds to the synthetic spatial map; and

generate a composite map by associating the 3D image with the location within the spatial map of the portion of the environment.

2 . The information processing apparatus of claim 1 , wherein the at least one processor is further configured to execute the instructions to:

in the 3D image, inpaint areas of the portion of the environment that are not in the plurality of images.

3 . The information processing apparatus of claim 2 , wherein the at least one processor is further configured to execute the instructions to:

select a first viewpoint for the 3D image based on the location of the first viewpoint relative to the plurality of images;

render a first 3D image from the first viewpoint;

select a second viewpoint for the 3D image different from the first viewpoint;

render a second 3D image from the second viewpoint; and

inpaint missing image data in the second 3D image.

4 . The information processing apparatus of claim 3 , wherein the first viewpoint is selected as a boundary point in a top-down view of the 3D image.

5 . The information processing apparatus of claim 1 , wherein determining the location of the portion of the environment within the spatial map by identifying the region of the spatial map that corresponds to the synthetic spatial map comprises:

estimating edges of the floor using ray casting; and

determining an alignment between the estimated edges of the floor and the spatial map of the environment.

6 . The information processing apparatus of claim 1 , wherein the spatial map of the environment is generated using two-dimensional Light Detection and Ranging, (2D LiDAR).

7 . The information processing apparatus of claim 1 , wherein the plurality of images is obtained from a video of the portion of the environment.

8 . The information processing apparatus of claim 1 , wherein the at least one processor is further configured to execute the instructions to:

identify an object in the 3D image; and

associate an identity of the object with a location of the object in the composite map.

9 . The information processing apparatus of claim 1 , wherein the spatial map is generated by a robot.

10 . The information processing apparatus of claim 1 , wherein the plurality of images are generated by a user terminal.

11 . A method of generating a composite map from image data and spatial map data, the method comprising:

acquiring a spatial map of an environment;

acquiring a plurality of images of a portion of the environment;

generating a three-dimensional (3D) image of the portion of the environment using the plurality of images;

identifying a floor in the 3D image;

generating a synthetic spatial map of the portion of the environment based on the floor in the 3D image;

determining a location of the portion of the environment within the spatial map by identifying a region of the spatial map that corresponds to the synthetic spatial map; and

generating a composite map by associating the 3D image with the location within the spatial map of the portion of the environment.

12 . The method of claim 11 , further comprising:

in the 3D image, inpainting areas of the portion of the environment that are not in the plurality of images.

13 . The method of claim 12 , further comprising:

selecting a first viewpoint for the 3D image based on the location of the first viewpoint relative to the plurality of images;

rendering a first 3D image from the first viewpoint;

selecting a second viewpoint for the 3D image different from the first viewpoint;

rendering a second 3D image from the second viewpoint; and

inpainting missing image data in the second 3D image.

14 . The method of claim 13 , wherein the first viewpoint is selected as a boundary point in a top-down view of the 3D image.

15 . The method of claim 11 , wherein determining the location of the portion of the environment within the spatial map by identifying the region of the spatial map that corresponds to the synthetic spatial map comprises:

estimating edges of the floor using ray casting; and

determining an alignment between the estimated edges of the floor and the spatial map of the environment.

16 . The method of claim 11 , wherein the spatial map of the environment is generated using two-dimensional Light Detection and Ranging, (2D LiDAR).

17 . The method of claim 11 , wherein the plurality of images is obtained from a video of the portion of the environment.

18 . The method of claim 11 , further comprising:

identifying an object in the 3D image; and

associating an identity of the object with a location of the object in the composite map.

19 . The method of claim 11 , wherein the spatial map is generated by a robot.

20 . The method of claim 11 , wherein the plurality of images is generated by a user terminal.

Assignments (1)

Continuity (2)

References Cited (17)