Granted Patent B2

US 12,249,142 · App. 17/748,378 · Granted Mar 11, 2025

Inventors: Juan Liu (Nanjing, CN); Huasin Wang (Nanjing, CN); Sukun Yoon (Suwon-si, KR); Longhai Wu (Nanjing, CN); Jie Chen (Nanjing, CN)

Assignee: SAMSUNG ELECTRONICS CO., LTD.

G06V20/36G06T7/50G06V20/64

Abstract

Claims (81)

1. A method for acquiring a floor map of a room layout, the method comprising:

acquiring a depth map and a two-dimensional (2D) image of an interior of a room;

identifying a boundary line between a ceiling and a wall of the room in the acquired 2D image;

determining valid pixels among pixels on the boundary line according to a first preset rule, wherein the valid pixels indicate a pixel that corresponds to an actual point in the identified line at the junction of the ceiling and the wall;

identifying an electronic device in the room and determining a location of the electronic device based on SLAM positioning technology;

determining three-dimensional (3D) coordinates of the actual points at the junction of the ceiling and the wall of the room corresponding to the valid pixels and 3D coordinates of the determined location of the electronic device based on the depth map and the 2D image;

converting the 3D coordinates of the actual points corresponding to the valid pixels and the 3D coordinates of the determined location of the electronic device coordinate into coordinates in a global coordinate system; and

acquiring the floor map of the room layout by projecting the 3D coordinates of the actual points corresponding to the valid pixels on the boundary line converted into the global coordinate system and the 3D coordinates of the determined location of the electronic device converted into the global coordinate system onto a horizontal plane,

wherein the floor map of the room layout comprises connection lines between projections of the 3D coordinates of the actual points corresponding to the valid pixels and an icon indicating a type and the location of the electronic device.

2. The method of claim 1 , wherein the determining the valid pixels among the pixels on the boundary line according to the first preset rule comprises:

determining 3D coordinate of the actual point corresponding to each pixel on the boundary line and 3D coordinates of actual points corresponding to pixels near the boundary line based on the depth map, wherein the pixels near the boundary line are included in the 2D image and within a preset distance from the boundary line;

determining two adjacent points within a predetermined distance range from the actual point corresponding to the each pixel on the boundary line according to a second preset rule, and determining a direction of a plane formed by the actual point corresponding to the each pixel and the determined two adjacent points, wherein the two adjacent points are included in the actual points corresponding to the pixels near the boundary line;

sequentially determining a difference between directions of planes respectively corresponding to any two adjacent pixels on the boundary line;

based on the difference being less than or equal to a first threshold value, determining that the two adjacent pixels on the boundary line are valid pixels; and

based on the difference being greater than the first threshold value, determining that the two adjacent pixels on the boundary line are invalid pixels, and

wherein, when a previous pixel of a current pixel is determined to be an invalid pixel, a pixel closest to the current pixel among the valid pixels that are previously determined is determined to be a pixel adjacent to the current pixel.

3. The method of claim 1 , wherein the identifying the electronic device in the room and determining the location of the electronic device comprises:

determining whether electronic devices that have been identified include an electronic device of the same type as a currently identified electronic device;

based on the electronic devices that have been identified including the electronic device of the same type as the currently identified electronic device, determining whether a distance between the currently identified electronic device and the electronic device of the same type is greater than or equal to a second threshold;

based on determining that the distance is greater than or equal to the second threshold, determining that the currently identified electronic device is a new electronic device; and

based on determining that the distance is less than the second threshold, determining that the currently identified electronic device and the electronic device of the same type are the same electronic device.

4. The method of claim 3 , wherein the identifying the electronic device in the room and determining the location of the electronic device further comprises:

based on determining that the currently identified electronic device and the electronic device of the same type are the same electronic device, determining newly determined coordinate of the currently identified electronic device to be coordinate of the electronic device of the same type.

5. The method of claim 1 , further comprising:

identifying a furniture in the room and a location of the furniture; and

acquiring the location of the furniture in the floor map of the room layout by projecting 3D coordinates of the location of the furniture onto the horizontal plane,

wherein the floor map of the room layout comprises an icon representing a type and the location of the furniture.

6. The method of claim 1 , further comprising:

displaying the floor map of the room layout on an electronic terminal; and

establishing communication between the electronic terminal and the electronic device,

wherein the icon corresponding to the electronic device is included in the floor map and is used to control the electronic device.

7. An electronic apparatus for acquiring a floor map of a room layout, comprising:

a memory storing instructions; and

a processor configured to execute the instructions to:

acquire a depth map and a two-dimensional (2D) image of an interior of a room;

identify a boundary line between a ceiling and a wall of the room in the acquired 2D image;

determine valid pixels among pixels on the boundary line according to a first preset rule, wherein the valid pixels indicate a pixel that corresponds to the actual point in the identified line at the junction of the ceiling and the wall;

identify an electronic device in the room and determine a location of the electronic device based on SLAM positioning technology;

determine three-dimensional (3D) coordinates of the actual points at the junction of the ceiling and the wall of the room corresponding to the valid pixels and 3D coordinates of the determined location of the electronic device based on the depth map and the 2D image;

convert the 3D coordinates of the actual points corresponding to the valid pixels and the 3D coordinates of the determined location of the electronic device coordinate into coordinates in global coordinate system; and

acquire the floor map of the room layout by projecting the 3D coordinates of the actual points corresponding to the valid pixels on the boundary line converted into the global coordinate system and the 3D coordinate of the determined location of the electronic device converted into the global coordinate system onto a horizontal plane,

wherein the floor map of the room layout comprises connection lines between projections of the 3D coordinates of the actual points corresponding to the valid pixels and an icon indicating a type and the location of the electronic device.

8. The electronic apparatus of claim 7 , wherein the processor is further configured to execute the instructions to:

determine 3D coordinate of actual point corresponding to each pixel on the boundary line and 3D coordinates of actual points corresponding to pixels near the boundary line based on the depth map, wherein the pixels near the boundary line are included in the 2D image and within a preset distance from the boundary line;

determine two adjacent points within a predetermined distance range from the actual point corresponding to the each pixel on the boundary line according to a second preset rule, and determine a direction of a plane formed by the actual point corresponding to the each pixel and the determined two adjacent points, wherein the two adjacent points are included in the actual points corresponding to the pixels near the boundary line;

sequentially determine a difference between directions of planes respectively corresponding to any two adjacent pixels on the boundary line;

based on the difference being less than or equal to a first threshold value, determine that the two adjacent pixels on the boundary line are valid pixels; and

based on the difference being greater than the first threshold value, determine that the two adjacent pixels on the boundary line are invalid pixels, and

wherein, when a previous pixel of a current pixel is determined to be an invalid pixel, a pixel closest to the current pixel among the valid pixels that are previously determined is determined to be a pixel adjacent to the current pixel.

9. The electronic apparatus of claim 7 , wherein the processor is further configured to exe hate the instructions to:

determine whether electronic devices that have been identified include an electronic device of the same type as a currently identified electronic device;

based on the electronic devices that have been identified including the electronic device of the same type as the currently identified electronic device, determine whether a distance between the currently identified electronic device and the electronic device of the same type is greater than or equal to a second threshold;

based on determining that the distance is greater than or equal to the second threshold, determine that the currently identified electronic device is a new electronic device; and

based on determining that the distance is less than the second threshold, determine that the currently identified electronic device and the electronic device of the same type are the same electronic device.

10. The electronic apparatus of claim 9 , wherein the processor is further configured to execute the instructions to:

based on determining that the currently identified electronic device and the electronic device of the same type are the same electronic device, determine newly determined coordinate of the currently identified electronic device as coordinate of the electronic device of the same type.

11. The electronic apparatus of claim 9 , wherein the processor is further configured to execute the instructions to:

identify a furniture in the room and a location of the furniture; and

acquire the location of the furniture in the floor map of the room layout by projecting 3D coordinates of the location of the furniture onto the horizontal plane,

wherein the floor map of the room layout comprises an icon representing a type and the location of the furniture.

12. The electronic apparatus of claim 7 , further comprising:

a display configured to display the floor map of the room layout; and

a communication interface configured to establish a communication between the electronic terminal and the electronic device,

wherein the icon corresponding to the electronic device is included in the floor map and is used to control the electronic device.

13. A non-transitory computer-readable storage medium storing a computer program, when executed by a processor, causing the processor to implement a method including:

acquiring a depth map and a two-dimensional (2D) image of an interior of a room;

identifying a boundary line between a ceiling and a wall of the room in the acquired 2D image;

determining valid pixels among pixels on the boundary line according to a first preset rule, wherein the valid pixels indicate a pixel that corresponds to the actual point in the identified line at the junction of the ceiling and the wall;

identifying an electronic device in the room and determining a location of the electronic device based on SLAM positioning technology;

determining three-dimensional (3D) coordinates of the actual points at the junction of the ceiling and the wall of the room corresponding to the valid pixels and 3D coordinates of the determined location of the electronic device based on the depth map and the 2D image;

converting the 3D coordinates of the actual points corresponding to the valid pixels and the 3D coordinates of the determined location of the electronic device coordinate into coordinates in global coordinate system; and

acquiring the floor map of the room layout by projecting the 3D coordinates of the actual points corresponding to the valid pixels on the boundary line converted into the global coordinate system and the 3D coordinates of the determined location of the electronic device converted into the global coordinate system onto a horizontal plane, and

wherein, the floor map of the room layout comprises connection lines between projections of the 3D coordinates of the actual points corresponding to the valid pixels and an icon indicating a type and the location of the electronic device.

14. The method of claim 1 , wherein the boundary line between the ceiling and the wall of the room is identified using a neural network model that is trained based on sample images including sample boundary lines.

15. The method of claim 1 , wherein the acquiring the depth map and the 2D image of the interior of the room comprises:

acquiring only the depth map and the 2D image of a boundary area between the ceiling and the wall of the room, without acquiring a panoramic 2D image and depth map of the interior of the room.

16. The method of claim 1 , wherein the determining of the valid pixels among the pixels on the boundary line according to the first preset rule comprises:

converting 2D coordinates of actual points in the 2D image to 3D coordinates corresponding to the 2D coordinates.

17. The electronic apparatus of claim 7 , wherein the boundary line between the ceiling and the wall of the room is identified using a neural network model that is trained based on sample images including sample boundary lines.

18. The electronic apparatus of claim 7 , wherein the processor is further configured to execute the instructions to:

acquire only the depth map and the 2D image of a boundary area between the ceiling and the wall of the room, without acquiring a panoramic 2D image and depth map of the interior of the room.

Assignments (1)

Priority Claims (1)

Continuity (2)

References Cited (34)