IP Library Granted Patent US 11,430,191
Granted Patent B2
US 11,430,191 · App. 17/201,693 · Granted Aug 30, 2022

Systems and methods of augmented reality guided image capture

Inventors: Nicole Allison Tan (Brookline, MA); Rachana Sreedhar (Boston, MA); Shrenik Sadalgi (Cambridge, MA); Christian Vázquez (Revere, MA); Jessica Stigile (Cambridge, MA); Ashwin Subramani (Belmont, MA); Maria Paula Gomez Muñoz (New York, NY)
Assignee: Wayfair LLC
G06T19/006G06T7/0002G06T7/70G06T7/80G06T19/003H04N9/07H04N9/646
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,430,191
App. No.
17/201,693
Granted
Aug 30, 2022
Kind
B2
Abstract

Some embodiments provide a mobile device configured to guide a user, via an augmented reality (AR) interface generated by the mobile device, to capture images of the physical object using a camera of the mobile device. The mobile device may be configured to obtain boundary information indicative of a boundary enclosing the physical object (e.g., a box enclosing the physical object). The mobile device may be configured to use the boundary information to determine positions from which the user is to capture images of the physical object. The mobile device may be configured to guide the user to capture the images using the AR interface by guiding the user to each of the positions in the AR interface (e.g., by generating on or more GUI elements in the AR interface that indicate a position from which the user is to capture an image).

Claims (89)

1. A mobile device, comprising:

at least one camera;

at least one computer hardware processor; and

at least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform a method for guiding a user, via an augmented reality (AR) interface generated by the mobile device, to capture a plurality of images of a physical object using the mobile device, the method comprising:

obtaining boundary information indicative of a boundary enclosing the physical object;

determining, using the boundary information, a plurality of positions from which the user is to capture the plurality of images, the plurality of positions including a first position from which the user is to capture a first image of the plurality of images;

guiding, using the AR interface, the user to capture the plurality of images, the guiding comprising:

guiding the user to the first position using the AR interface;

determining whether the user can capture the first image by performing at least one quality control test; and

in response to determining that the user can capture the first image, enabling the user to capture the first image via the AR interface; and

after the plurality of images have been captured by the mobile device, outputting the plurality of images.

2. The mobile device of claim 1 , wherein obtaining the boundary information comprises obtaining information specifying a box enclosing the physical object.

3. The mobile device of claim 2 , wherein obtaining the information specifying the box enclosing the physical object comprises:

obtaining first input indicative of locations of multiple corners of the box, the first input specified by the user tapping multiple respective locations shown in the AR interface; and

obtaining second input indicative of a height for the box, the second input specified by the user using a slider shown in the AR interface.

4. The mobile device of claim 3 , wherein the method further comprises:

displaying, in the AR interface, the box enclosing the physical object.

5. The mobile device of claim 1 , wherein obtaining the boundary information comprises obtaining the boundary information based on input provided by the user via the AR interface.

6. The mobile device of claim 1 , wherein the physical object comprises furniture.

7. The mobile device of claim 1 , wherein determining the plurality of positions comprises determining, for each position of the plurality of positions, a respective angle and a respective distance to the physical object.

8. The mobile device of claim 2 , wherein determining the plurality of positions from which the user is to capture the plurality of images comprises:

determining the first position by determining:

a first angle using the box; and

a first distance to the physical object at which to capture the first image.

9. The mobile device of claim 8 , wherein determining the first distance to the physical object is determined using a field of view for the at least one camera and a height for the physical object.

10. The mobile device of claim 9 , wherein determining the first distance is further performed using a value specifying a desired percentage of the first image to be taken up by the physical object.

11. The mobile device of claim 9 , wherein determining the first distance is further performed based on the first angle.

12. The mobile device of claim 1 , wherein guiding the user to the first position using the AR interface comprises:

displaying, in the AR interface, a visual prompt indicating where the user is to position the mobile device to capture an image from the first position.

13. The mobile device of claim 1 , wherein performing the at least one quality control test comprises determining whether the at least one camera is positioned at the first position.

14. The mobile device of claim 1 ,

wherein guiding the user to the first position using the AR interface further comprises displaying, in the AR interface, a visual prompt indicating a first height, above the first position, at which the at least one camera is to be positioned; and

wherein performing the at least one quality control test comprises determining whether the at least one camera is positioned at the first height.

15. The mobile device of claim 1 , wherein performing the at least one quality control test comprises determining whether the at least one camera is facing the physical object.

16. The mobile device of claim 15 , wherein determining whether the at least one camera is facing the physical object comprises determining whether the physical object is in a frustum view of the at least one camera.

17. The mobile device of claim 1 , wherein performing the at least one quality control test comprises determining whether the at least one camera is held at a desired pitch.

18. The mobile device of claim 1 , wherein performing the at least one quality control test comprises determining whether a measure of light intensity is above a threshold.

19. The mobile device of claim 1 , wherein enabling the user to capture the first image via the AR interface, comprises presenting a selectable element in the AR interface that, when selected by the user, causes the mobile device to capture the first image.

20. The mobile device of claim 1 , wherein the method further comprises:

after the user captures the first image,

determining whether the first image is blurry; and

when it is determined that the first image is blurry, prompting the user via the AR interface to capture a new image to replace the first image.

21. A system, comprising:

the mobile device of claim 1 ; and

at least one computer communicatively coupled to the mobile device using at least one communication network,

wherein outputting the plurality of images comprises transmitting the plurality of images from the mobile device to the at least one computer via the at least one communication network.

22. The system of claim 21 , wherein the at least one computer is configured to:

receive, from the mobile device and via the at least one communication network, the plurality of images; and

generate a three-dimensional (3D) model of the physical object using the plurality of images.

23. The mobile device of claim 1 , wherein the method further comprises:

guiding, using the AR interface, the user to:

position a color calibration target proximate the physical object; and

capture a color calibration image of the physical object and the color calibration target.

24. A system, comprising:

the mobile device of claim 23 ; and

at least one computer communicatively coupled to the mobile device using at least one communication network,

wherein outputting the plurality of images comprises transmitting the plurality of images and the color calibration image from the mobile device to the at least one computer.

25. The system of claim 24 , wherein the at least one computer is configured to perform:

receiving, from the mobile device and via the at least one communication network, the plurality of images and the color calibration image;

obtaining color correction information from the color calibration image; and

correcting color of one or more of the plurality of images using the color correction information.

26. The system of claim 25 , wherein obtaining the color correction information comprises:

detecting the color calibration target in the color calibration image;

identify pixels, in the color calibration image, associated with a gray square in the color calibration target; and

determining RGB values of at least some of the identified pixels.

27. The mobile device of claim 1 , wherein outputting the plurality of images comprises:

storing the plurality of images in the at least one non-transitory computer readable storage medium.

28. The mobile device of claim 1 , wherein outputting the plurality of images comprises:

transmitting the plurality of images to at least one other computing device.

29. The mobile device of claim 1 , wherein guiding, using the AR interface, the user to capture the plurality of images comprises:

guiding the user to a second position of the plurality of positions from which the user is to capture a second image of the plurality of images;

determining whether the user can capture the second image by performing the at least one quality control test; and

in response to determining that the user can capture the second image, enabling the user to capture the second image via the AR interface.

30. A method for guiding a user, via an augmented reality (AR) interface generated by a mobile device, to capture a plurality of images of a physical object using the mobile device, the method comprising:

obtaining boundary information indicative of a boundary enclosing the physical object;

determining, using the boundary information, a plurality of positions from which the user is to capture the plurality of images, the plurality of positions including a first position from which the user is to capture a first image of the plurality of images;

guiding, using the AR interface, the user to capture the plurality of images, the guiding comprising:

guiding the user to the first position using the AR interface;

determining whether the user can capture the first image by performing at least one quality control test; and

in response to determining that the user can capture the first image, enabling the user to capture the first image via the AR interface; and

after the plurality of images have been captured by the mobile device, outputting the plurality of images.

31. At least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by at least one computer hardware processor of a mobile device, cause the at least one computer hardware processor to perform a method for guiding a user, via an augmented reality (AR) interface generated by the mobile device, to capture a plurality of images of a physical object using the mobile device, the method comprising:

obtaining boundary information indicative of a boundary enclosing the physical object;

determining, using the boundary information, a plurality of positions from which the user is to capture the plurality of images, the plurality of positions including a first position from which the user is to capture a first image of the plurality of images;

guiding, using the AR interface, the user to capture the plurality of images, the guiding comprising:

guiding the user to the first position using the AR interface;

determining whether the user can capture the first image by performing at least one quality control test; and

in response to determining that the user can capture the first image, enabling the user to capture the first image via the AR interface; and

after the plurality of images have been captured by the mobile device, outputting the plurality of images.

Assignments (7)
SECURITY AGREEMENT Recorded May 20, 2026
From: WAYFAIR LLC
To: U.S. BANK TRUST COMPANY, NATIONAL ASSOCIATION, AS NOTES COLLATERAL AGENT
Reel/Frame 075591/0399 →
SECURITY INTEREST Recorded Nov 10, 2025
From: WAYFAIR LLC
To: U.S. BANK TRUST COMPANY, NATIONAL ASSOCIATION, AS NOTES COLLATERAL AGENT
Reel/Frame 073514/0326 →
SECURITY AGREEMENT Recorded Mar 13, 2025
From: WAYFAIR LLC
To: U.S. BANK TRUST COMPANY, NATIONAL ASSOCIATION, AS NOTES COLLATERAL AGENT
Reel/Frame 070513/0542 →
SECURITY AGREEMENT Recorded Oct 10, 2024
From: WAYFAIR LLC
To: U.S. BANK TRUST COMPANY, NATIONAL ASSOCIATION, AS NOTES COLLATERAL AGENT
Reel/Frame 069143/0399 →
SECURITY AGREEMENT Recorded Sep 22, 2022
From: WAYFAIR, LLC
To: CITIBANK, N.A., AS COLLATERAL AGENT
Reel/Frame 061553/0318 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 14, 2021
From: STIGILE, JESSICA; TAN, NICOLE ALLISON; SREEDHAR, RACHANA; SADALGI, SHRENIK; SUBRAMANI, ASHWIN; GOMEZ MUÑOZ, MARIA PAULA
To: WAYFAIR LLC
Reel/Frame 057790/0880 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 14, 2021
From: VÁZQUEZ, CHRISTIAN
To: WAYFAIR LLC
Reel/Frame 057478/0947 →
Continuity (2)
Provisional Application 63145462 · Feb 3, 2021
Related Publication 20220245900A1 · Aug 4, 2022
Cited By (2)
US 12,430,856 US 12,657,834