IP Library Granted Patent US 11,875,464
Granted Patent B2
US 11,875,464 · App. 17/363,998 · Granted Jan 16, 2024

Systems and methods for scene-independent augmented reality interfaces

Inventors: Shrenik Sadalgi (Cambridge, MA); Kunal Rajesh Khona (Malden, MA)
Assignee: Wayfair LLC
G06T19/006G02B27/017G06F3/011G06F3/017G06F3/04815G06V20/20G02B2027/014G02B2027/0138
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,875,464
App. No.
17/363,998
Filed
Jun 30, 2021
Granted
Jan 16, 2024
Kind
B2
Art Unit
2614
USPC
345/633
Abstract

Some embodiments include a method comprising using a first computing device to perform: obtaining an image of a scene captured using a camera coupled to the first computing device; obtaining camera setting values used to capture the image; determining, using the image, surface attribute values characterizing at least one surface shown in the image of the scene; generating an augmented reality (AR) interface at least in part by using the camera setting values and the surface attribute values to create a first composite image by overlaying a selected virtual furniture object onto the image so that the virtual furniture object is displayed in the AR interface as being on a first surface of the at least one surface shown in the image; and transmitting, to a second computing device and via at least one communication network, the first composite image, the camera setting values and the surface attribute values.

Claims (50)

1. A method, comprising:

using a first computing device to perform:

obtaining an image of a scene captured using a camera coupled to the first computing device;

obtaining camera setting values used by the camera to capture the image;

determining, using the image, surface attribute values characterizing at least one surface shown in the image of the scene, the at least one surface including a first surface on which a first virtual object can be placed, the surface attribute values including first surface attribute values for the first surface;

generating an augmented reality (AR) interface at least in part by using the camera setting values and the surface attribute values to create a first composite image by overlaying the first virtual object onto the image so that the first virtual object is displayed in the AR interface as being on the first surface of the at least one surface shown in the image of the scene;

transmitting, to a second computing device and via at least one communication network, the first composite image, the camera setting values and the surface attribute values;

receiving, from the second computing device, at least one second composite image obtained by changing the first composite image by a second user, wherein changing the first composite image comprises adding a second virtual object to the first composite image, removing the first virtual object from the first composite image, changing a position and/or orientation of the first virtual object in the first composite image, and/or changing a visual characteristic of the first virtual object in the first composite image; and

generating a first graphical user interface (GUI) using the at least one second composite image, the camera setting values, and the surface attribute values.

2. The method of claim 1 , wherein the first computing device is a mobile computing device and the camera is integrated with the mobile computing device.

3. The method of claim 1 , wherein the camera setting values comprise one or more values associated with a field of view of the camera, one or more values indicating a projection from three-dimensional (3D) points in the scene to two-dimensional (2D) points in the image, one or more values indicating a position of the camera, and/or one or more values indicating an orientation of the camera.

4. The method of claim 1 , wherein the surface attribute values comprise one or more values associated with a pose of a plane characterizing the at least one surface.

5. The method of claim 1 , further comprising:

presenting the generated AR interface to a user of the first computing device via a display of the first computing device.

6. The method of claim 1 , wherein the AR interface includes a first portion for displaying the first composite image and a second portion containing one or more GUI elements through which a user may provide input to the AR interface.

7. The method of claim 1 , further comprising:

receiving, through the AR interface, user input selecting the first virtual object.

8. The method of claim 1 , wherein the transmitting further comprises transmitting object attribute values characterizing the first virtual object.

9. The method of claim 8 , wherein the object attribute values specify an identifier of the first virtual object, a position of the first virtual object in the image, an orientation of the first virtual object in the image, and/or a visual characteristic of the first virtual object.

10. The method of claim 1 , wherein the first virtual object is a virtual furniture object.

11. The method of claim 1 , wherein the second computing device is a server computing device or a mobile computing device.

12. The method of claim 1 , wherein the surface attribute values specify three-dimensional (3D) points representing one or more planes of the at least one surface as one or more polygons.

13. The method of claim 12 , wherein the 3D points further represent one or more positions and one or more rotations of the one or more planes with respect to the camera.

14. The method of claim 1 , wherein creating the first composite image comprises determining a position in the image at which to place the first virtual object, based on the first surface attribute values, so that the first virtual object is shown, in the first composite image, as being on the first surface.

15. The method of claim 14 , wherein determining the position in the image at which to place the first virtual object comprises:

identifying a location of the first surface in the image using the first surface attribute values; and

determining the position in the image at which to place the first virtual object based on the location of the first surface in the image.

16. A system, comprising:

at least one computer hardware processor; and

at least one non-transitory computer-readable storage medium storing processor executable instructions that, when executed by the at least one computer hardware processor of a first computing device, cause the at least one computer hardware processor to perform:

obtaining an image of a scene captured using a camera coupled to the first computing device;

obtaining camera setting values used by the camera to capture the image;

determining, using the image, surface attribute values characterizing at least one surface shown in the image of the scene, the at least one surface including a first surface on which a first virtual object can be placed, the surface attribute values including first surface attribute values for the first surface;

generating an augmented reality (AR) interface at least in part by using the camera setting values and the surface attribute values to create a first composite image by overlaying the first virtual object onto the image so that the first virtual object is displayed in the AR interface as being on the first surface of the at least one surface shown in the image of the scene;

transmitting, to a second computing device and via at least one communication network, the first composite image, the camera setting values and the surface attribute values;

receiving, from the second computing device, at least one second composite image obtained by changing the first composite image by a second user, wherein changing the first composite image comprises adding a second virtual object to the first composite image, removing the first virtual object from the first composite image, changing a position and/or orientation of the first virtual object in the first composite image, and/or changing a visual characteristic of the first virtual object in the first composite image; and

generating a first graphical user interface (GUI) using the at least one second composite image, the camera setting values, and the surface attribute values.

17. The system of claim 16 , wherein the first computing device is a mobile computing device and the camera is integrated with the mobile computing device.

18. The system of claim 16 , wherein the camera setting values comprise one or more values associated with a field of view of the camera, one or more values indicating a projection from three-dimensional (3D) points in the scene to two-dimensional (2D) points in the image, one or more values indicating a position of the camera, and/or one or more values indicating an orientation of the camera.

19. The system of claim 16 , wherein the surface attribute values comprise one or more values associated with a pose of a plane characterizing the at least one surface.

20. The system of claim 16 , wherein the transmitting further comprises transmitting object attribute values characterizing the first virtual object, the object attribute values specifying an identifier of the first virtual object, a position of the first virtual object in the image, an orientation of the first virtual object in the image, and/or a visual characteristic of the first virtual object.

21. The system of claim 16 , wherein creating the first composite image comprises determining a position in the image at which to place the first virtual object, based on the first surface attribute values, so that the first virtual object is shown, in the first composite image, as being on the first surface.

22. At least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by at least one processor of a first computing device, cause the at least one processor to perform:

obtaining an image of a scene captured using a camera coupled to the first computing device;

obtaining camera setting values used by the camera to capture the image;

determining, using the image, surface attribute values characterizing at least one surface shown in the image of the scene, the at least one surface including a first surface on which a first virtual object can be placed, the surface attribute values including first surface attribute values for the first surface;

generating an augmented reality (AR) interface at least in part by using the camera setting values and the surface attribute values to create a first composite image by overlaying the first virtual object onto the image so that the first virtual object is displayed in the AR interface as being on the first surface of the at least one surface shown in the image of the scene;

transmitting, to a second computing device and via at least one communication network, the first composite image, the camera setting values and the surface attribute values;

receiving, from the second computing device, at least one second composite image obtained by changing the first composite image by a second user, wherein changing the first composite image comprises adding a second virtual object to the first composite image, removing the first virtual object from the first composite image, changing a position and/or orientation of the first virtual object in the first composite image, and/or changing a visual characteristic of the first virtual object in the first composite image; and

generating a first graphical user interface (GUI) using the at least one second composite image, the camera setting values, and the surface attribute values.

Assignments (6)
SECURITY AGREEMENT Recorded May 20, 2026
From: WAYFAIR LLC
To: U.S. BANK TRUST COMPANY, NATIONAL ASSOCIATION, AS NOTES COLLATERAL AGENT
Reel/Frame 075591/0399 →
SECURITY INTEREST Recorded Nov 10, 2025
From: WAYFAIR LLC
To: U.S. BANK TRUST COMPANY, NATIONAL ASSOCIATION, AS NOTES COLLATERAL AGENT
Reel/Frame 073514/0326 →
SECURITY AGREEMENT Recorded Mar 13, 2025
From: WAYFAIR LLC
To: U.S. BANK TRUST COMPANY, NATIONAL ASSOCIATION, AS NOTES COLLATERAL AGENT
Reel/Frame 070513/0542 →
SECURITY AGREEMENT Recorded Oct 10, 2024
From: WAYFAIR LLC
To: U.S. BANK TRUST COMPANY, NATIONAL ASSOCIATION, AS NOTES COLLATERAL AGENT
Reel/Frame 069143/0399 →
SECURITY AGREEMENT Recorded Sep 22, 2022
From: WAYFAIR, LLC
To: CITIBANK, N.A., AS COLLATERAL AGENT
Reel/Frame 061553/0318 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 17, 2021
From: SADALGI, SHRENIK; KHONA, KUNAL RAJESH
To: WAYFAIR LLC
Reel/Frame 057203/0703 →
Continuity (3)
Continuation 16691952 · Nov 22, 2019
Provisional Application 62828749 · Apr 3, 2019
Related Publication 20210327154A1 · Oct 21, 2021