IP Library › Granted Patent US 9,437,045

Granted Patent B2

US 9,437,045 · App. 13/934,617 · Granted Sep 6, 2016

Real-time mobile capture and application of photographic images as textures in three-dimensional models

Inventors: David Joseph Arendash (Fremont, CA); Maribeth Joy Back (Los Gatos, CA)

Assignee: FUJI XEROX CO., LTD.

G06T19/006 G06T15/04

View Patent ↗

Loading inventors, assignments & file history…

Monitor This Case

Get email alerts when status or documents change.

Order Certified Copies

Most orders are placed with the USPTO same day — all within 24 business hours.

Order via The Patent Place →

Pre-filled with this patent's details

Quick Facts

Patent No.: US 9,437,045
App. No.: 13/934,617
Granted: Sep 6, 2016
Kind: B2

External Links

USPTO Patent Center↗ Full Patent PDF↗

Abstract

A computer-implemented method for obtaining texture data for a three-dimensional model is disclosed. The method is performed at a portable electronic device including a camera, one or more processors, and memory storing one or more programs. The method includes obtaining an image of one or more objects with the camera. The method also includes concurrently displaying at least a portion of the image with one or more objects in the three-dimensional model. The one or more objects in the three-dimensional model at least partially overlay the one or more objects in the image. The method furthermore includes storing at least one or more portions of the image as texture data for one or more of the one or more objects in the image.

Claims (79)

1. A method for obtaining texture data for a three-dimensional model, comprising:

at a portable electronic device including a camera, one or more processors, and memory storing one or more programs, the one or more processors executing the one or more programs to perform the operations of:

obtaining with the camera a live-video of real-world objects as the portable electronic device moves;

obtaining a three-dimensional model including at least one virtual object;

concurrently displaying at least a portion of the live-video in real-time and a representation of the three-dimensional model, wherein the representation of the three-dimensional model at least partially overlays the display of the portion of the live-video; and

while concurrently displaying at least the portion of the live-video in real-time and the representation of the three-dimensional model:

aligning the representation of one or more virtual objects in the three-dimensional model with one or more real-world objects in the live-video in real-time;

storing an image of the one or more real-world objects captured by the camera in real-time as texture data for the one or more virtual objects in the three-dimensional model;

concurrently displaying an overhead view of the three-dimensional model and displaying as part of the overhead view a depiction of a field of view of the camera; and

subsequent to obtaining an image of objects with the camera:

determining a recommended location of the camera for obtaining the texture data; and

visually indicating in the overhead view the recommended location of the camera for obtaining the texture data.

2. The method of claim 1 , wherein the one or more virtual objects in the three-dimensional model correspond to the one or more real-world objects in the live-video, and aligning the representation of one or more virtual objects in the three-dimensional model with one or more real-world objects in the live video includes:

associating a first set of markers placed in the three-dimensional model with a second set of markers placed in the image.

3. The method of claim 2 , wherein the device includes a touch-sensitive display and the method includes:

receiving one or more touch gestures on the touch-sensitive display; and

aligning the representation of the one or more virtual objects in the three-dimensional model in accordance with the one or more touch gestures.

4. The method of claim 3 , wherein aligning the one or more virtual objects includes one or more of:

scaling the representation of the one or more virtual objects in the three-dimensional model;

panning the representation of the one or more virtual objects in the three-dimensional model; and

rotating the representation of the one or more virtual objects in the three-dimensional model.

5. The method of claim 2 , wherein the live-video includes one or more augmented reality markers and the method includes:

aligning the representation of the one or more virtual objects in the three-dimensional model in accordance with respective locations of the one or more augmented reality markers.

6. The method of claim 5 , further comprising:

calibrating one or more of a location and an orientation of the portable electronic device in accordance with at least one of the one or more augmented reality markers.

7. The method of claim 5 , wherein:

information for the one or more augmented reality markers is received at the portable electronic device from a remote server,

the one or more augmented reality markers received at the portable electronic device are selected and updated in accordance with a location of the portable electronic device as the portable electronic device moves.

8. The method of claim 1 , further comprising:

obtaining a plurality of images at a respective location; and

generating a respective three-dimensional model of the respective location based on the plurality of images, including identifying one or more interfaces of walls by using edge detection.

9. The method of claim 8 , wherein the device includes an orientation sensor and the method includes:

obtaining the plurality of images at the respective location with orientation information for the plurality of images; and

generating the respective three-dimensional model of the respective location based on the plurality of images and the orientation information.

10. The method of claim 1 , wherein the image is sent to a remote server as texture data for the one or more virtual objects in the three-dimensional model.

11. The method of claim 10 , wherein additional texture data is sent to the remote server from a plurality of portable electronic devices, distinct from the portable electronic device.

12. The method of claim 1 , further comprising:

receiving an image rendered from information for the one or more virtual objects in the three-dimensional model and texture data for the one or more virtual objects in the three-dimensional model; and

displaying at least a portion of the rendered image.

13. The method of claim 12 , further comprising:

receiving a touch gesture on the touch-sensitive display; and

adjusting the texture data in accordance with the touch gesture while maintaining the representation of the one or more virtual objects in the three-dimensional model, wherein adjusting the texture data includes one or more of:

positioning the texture data onto the one or more virtual objects in the three-dimensional model;

resizing the texture data; and

cropping the texture data.

14. The method of claim 1 , further comprising:

receiving information for camera placement, the information for camera placement based at least on a viewing angle and a field of view with respect to the one or more virtual objects.

15. The method of claim 1 , wherein a virtual object of the one or more virtual objects with missing texture data is displayed in a distinct color.

16. The method of claim 1 , including:

concurrently displaying at least the portion of the live-video in real-time and a second representation of the three-dimensional model, wherein the second representation of the three-dimensional model at least partially overlays the display of the portion of the live-video, the image is applied as the texture data for the one or more virtual objects in the three-dimensional model.

17. A non-transitory computer readable storage medium storing one or more programs for execution by one or more processors of an electronic device with a camera, the one or more programs including instructions for: