IP Library › Patent Application 17180415
Patent Application
App. No. 17/180,415

ARTIFICIAL INTELLIGENCE SYSTEMS AND METHODS FOR INTERIOR DESIGN

Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US None
App. No.
17/180,415
Abstract

Systems and methods for visualizing furnishing objects in a property are disclosed. An exemplary system includes a communication interface configured to receive a depth image of an interior space of the property captured by a 3D scanner and the depth image includes one or more existing furnishing objects in the interior space. The system further includes at least one processor configured to remove at least one existing furnishing object from the depth image, leaving at least one hole in the depth image corresponding to where the removed existing furnishing object used to be. The at least one processor is further configured to restore the depth image by filling the at least one hole in the depth image with a scene of the interior space that was blocked by the removed existing furnishing object, using a first neural network model trained with an image inpainting algorithm. The at least one processor is also configured to insert at least one new furnishing object in the restored image and render a 3D view of the interior space with the at least one new furnishing object for display.

Claims (68)

1 . A system for visualizing furnishing objects in a property, comprising:

a communication interface configured to receive a depth image of an interior space of the property captured by a 3D scanner, wherein the depth image includes one or more existing furnishing objects in the interior space; and

at least one processor configured to:

remove at least one existing furnishing object from the depth image, leaving at least one hole in the depth image corresponding to where the removed existing furnishing object used to be;

restore the depth image by filling the at least one hole in the depth image with a scene of the interior space that was blocked by the removed existing furnishing object, using a first neural network model trained with an image inpainting algorithm;

insert at least one new furnishing object in the restored image; and

render a 3D view of the interior space with the at least one new furnishing object for display.

2 . The system of claim 1 , wherein to remove the at least one existing furnishing object from the depth image, the at least one processor is further configured to:

detect the at least one existing furnishing object in the depth image using a second neural network model; and

replace image data associated with each detected existing furnishing object with a predetermined value.

3 . The system of claim 2 , wherein to detect the at least one existing furnishing object in the depth image, the at least one processor is further configured to:

determine 3D point cloud data of the depth image based on depth information captured by the 3D scanner;

identify target point cloud data of each detected existing furnishing object by segmenting the 3D point cloud data of the depth image; and

determine a position of the each detected existing furnishing object in the depth image based on the target point cloud data.

4 . The system of claim 3 , wherein to determine a position of the each detected existing furnishing object, the at least one processor is further configured to determine a contour of the each detected existing furnishing object using the corresponding target point cloud data.

5 . The system of claim 1 , wherein to insert the at least one new furnishing object in the restored image, the at least one processor is further configured to:

insert each new furnishing object into a target position of the restored image, wherein the target position is associated with an area of a hole left by a removed existing furnishing object; and

adjust the new furnishing object to a target dimension to fit the new furnishing object into the area.

6 . The system of claim 1 , wherein the at least one processor is further configured to:

before removing the at least one existing furnishing object from the depth image,

determine attributes of each existing furnishing object in the depth image using a third neural network model;

determine a style of the interior space captured in the depth image; and

identify the at least one existing furnishing object for removal, wherein the attributes of the at least one existing furnishing object do not match the style of the interior space.

7 . The system of claim 6 , wherein the at least one processor is further configured to:

automatically select the at least one new furnishing object for the interior space to be inserted in the restored image based on the style of the interior space.

8 . The system of claim 7 , wherein the at least one processor is further configured to, before inserting the at least one new furnishing object in the restored image:

generate a suggestion indicative of the at least one new furnishing object;

send the suggestion to a user; and

receive a user approval for inserting the at least one new furnishing object.

9 . A computer-implemented method for visualizing furnishing objects in a property, comprising:

receiving a depth image of an interior space of the property captured by a 3D scanner, wherein the depth image includes one or more existing furnishing objects in the interior space;

removing, by at least one processor, at least one existing furnishing object from the depth image, leaving at least one hole in the depth image corresponding to where the removed existing furnishing object used to be;

restoring, by the at least one processor, the depth image by filling the at least one hole in the depth image with a scene of the interior space that was blocked by the removed existing furnishing object, using a first neural network model trained with an image inpainting algorithm;

inserting, by the at least one processor, at least one new furnishing object in the restored image; and

rendering, by the at least one processor, a 3D view of the interior space with the at least one new furnishing object for display.

10 . The computer-implemented method of claim 9 , wherein removing the at least one existing furnishing object from the depth image further comprises:

detecting the at least one existing furnishing object in the depth image using a second neural network model; and

replacing image data associated with each detected existing furnishing object with a predetermined value.

11 . The computer-implemented method of claim 10 , wherein detecting the at least one existing furnishing object in the depth image further comprises:

determining 3D point cloud data of the depth image based on depth information captured by the 3D scanner;

identifying target point cloud data of each detected existing furnishing object by segmenting the 3D point cloud data of the depth image; and

determining a position of the each detected existing furnishing object in the depth image based on the target point cloud data.

12 . The computer-implemented method of claim 9 , wherein the first neural network model is configured to extract features from regions of the depth image outside the at least one hole to learn features to be filled in the at least one hole.

13 . The computer-implemented method of claim 9 , wherein inserting the at least one new furnishing object in the restored image further comprises:

inserting each new furnishing object into a target position of the restored image, wherein the target position is associated with an area of a hole left by a removed existing furnishing object; and

adjusting the new furnishing object to a target dimension to fit the new furnishing object into the area.

14 . The computer-implemented method of claim 13 , wherein the target dimension is determined based on a ratio between a physical dimension of the removed existing furnishing object and a physical dimension of the new furnishing object.

15 . The computer-implemented method of claim 9 , further comprising: before removing the at least one existing furnishing object from the depth image,

determining attributes of each existing furnishing object in the depth image using a third neural network model;

determining a style of the interior space captured in the depth image; and

identifying the at least one existing furnishing object for removal, wherein the attributes of the at least one existing furnishing object do not match the style of the interior space.

16 . The computer-implemented method of claim 15 , further comprising:

automatically selecting the at least one new furnishing object for the interior space to be inserted in the restored image based on the style of the interior space.

17 . A computer-implemented method for suggesting new furnishing objects for a property, comprising:

receiving a depth image of an interior space of the property captured by a 3D scanner, wherein the depth image includes one or more existing furnishing objects in the interior space;

determining, by at least one processor, attributes of each existing furnishing object in the depth image using a neural network model;

determining, by the at least one processor, a style of the interior space captured in the depth image;

identifying, by the at least one processor, at least one existing furnishing object, the attributes of which do not match the style of the interior space; and

automatically, by the at least one processor, selecting at least one new furnishing object for the interior space based on the style of the interior space to replace the at least one existing furnishing object.

18 . The computer-implemented method of claim 17 , further comprising:

generating a suggestion indicative of the at least one new furnishing object;

sending the suggestion to a user; and

receiving a user approval for replacing the at least one existing furnishing object with the at least one new furnishing object.

19 . The computer-implemented method of claim 17 , further comprising:

removing the at least one existing furnishing object from the depth image, leaving at least one hole in the depth image corresponding to where the removed existing furnishing object used to be;

restoring the depth image by filling the at least one hole in the depth image with a scene of the interior space that was blocked by the removed existing furnishing object, using a second neural network model trained with an image inpainting algorithm; and

inserting the at least one new furnishing object in the restored image.

20 . The computer-implemented method of claim 17 , wherein the style of the interior space is determined based on the attributes of the existing furnishing objects in the depth image collectively.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 19, 2022
From: KE.COM (BEIJING)TECHNOLOGY CO., LTD.
To: REALSEE (BEIJING) TECHNOLOGY CO., LTD.
Reel/Frame 059642/0864 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 19, 2021
From: YANG, BIN; HU, YILANG; ZHU, YI; XIN, CHENGCONG; XIANG, CHAORAN; YANG, YUKE; SU, CHONG; DENG, SHILI; BIAN, JIANG; JIANG, XINYUAN
To: KE.COM (BEIJING) TECHNOLOGY CO., LTD.
Reel/Frame 055346/0965 →