IP Library Granted Patent US 9,998,684
Granted Patent B2
US 9,998,684 · App. 14/460,914 · Granted Jun 12, 2018

Method and apparatus for virtual 3D model generation and navigation using opportunistically captured images

Inventors: Apu Kapadia (Bloomington, IN); Md Zahidur Rahman (Bloomington, IN); David Crandall (Bloominton, IN); Robert E. Templeman (Bloomington, IN)
Assignees: Indiana University Research and Technology Corporation; The United States of America as Represented by the Secretary of the Navy
H04N5/2621G06T15/205G06T17/00
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 9,998,684
App. No.
14/460,914
Granted
Jun 12, 2018
Kind
B2
Abstract

A method and device for opportunistically collecting images of a location via a mobile device, the device including algorithms that allow the selective winnowing of collected images to facilitate the creation of a 3D model from the images while minimizing the storage needed to hold the pictures.

Claims (36)

1. A method of assembling a data set for the creation of a 3D model of an environment including:

obtaining a plurality of images that have been passively and opportunistically captured by a mobile device;

storing the plurality of images on the mobile device in mobile device storage;

automatically electronically analyzing the stored images by the mobile device to assess at least one of quality, content, and redundancy;

automatically deleting one or more images from the mobile device storage responsive to the automatic electronic analysis to create a retained subset of the obtained plurality of images; and

constructing the 3D model of the environment in which the mobile device was located from the output retained subset of images, wherein the 3D model allows a user to select a portion of the model such that the selection causes presentation of a first set of images of the plurality of images obtained by the mobile device, the first set of images being selected from a set of images that were used to create the selected portion of the 3D model.

2. The method of claim 1 , further including passively capturing the plurality of images at regular time intervals.

3. The method of claim 2 , wherein the time interval is modified based on inputs from a motion detector within the mobile device.

4. The method of claim 1 , further including capturing the plurality of images in a manner that is undetectable by a user of the mobile device.

5. The method of claim 1 , further including outputting the retained subset from the mobile device.

6. The method of claim 1 , wherein the automatic analysis of stored images includes a measurement of anisotropy of each image.

7. The method of claim 6 , wherein images with anisotropy below a defined threshold are automatically deleted from the mobile device.

8. The method of claim 1 , wherein images that have a similarity to other images stored on the mobile device above a threshold similarity value are automatically deleted from the mobile device.

9. The method of claim 1 , further including obtaining orientation information for the mobile device and associating such orientation information with each of the obtained images.

10. A mobile device including:

a camera,

a memory; and

an image analyzer, the image analyzer operable to execute code that causes the image analyzer to:

opportunistically obtain a plurality of images captured by the camera and stored in the memory;

automatically electronically analyze the stored images to assess at least one of quality, content, and redundancy;

automatically delete one or more images from the mobile device storage responsive to the automatic electronic analysis to create a retained subset of the obtained plurality of images; and

constructing a 3D model of an environment in which the mobile device was located from the output retained subset of opportunistically obtained images, wherein the 3D model allows a user to select a portion of the model such that the selection causes presentation of a first set of images of the plurality of images obtained by the mobile device, the first set of images being selected from a set of images that were used to create the selected portion of the 3D model.

11. The mobile device of claim 10 , further including at least one of a motion detector and an orientation detector.

12. The mobile device of claim 10 , further including a transmitter/receiver operable to transmit images of the retained subset.

13. The mobile device of claim 10 , further including an image collector, the collector operable to control the camera to obtain images, where such operation of the camera is done without a mobile device user being contemporaneously conscious that the camera is obtaining images.

14. The mobile device of claim 10 , wherein the automatic analysis includes a measurement of anisotropy of each image and images with anisotropy below a defined threshold are automatically deleted from the mobile device.

15. A non-transitory computer readable medium having instructions thereon that when interpreted by a processor cause the processor to:

opportunistically obtain a plurality of images captured by a camera of a mobile device and stored in a memory of the mobile device;

automatically electronically analyze the stored images to assess at least one of quality, content, and redundancy;

automatically delete one or more images from the mobile device storage responsive to the automatic electronic analysis to create a retained subset of the obtained plurality of images; and

constructing a 3D model of an environment in which the mobile device was located from the output retained subset of opportunistically obtained images, wherein the 3D model allows a user to select a portion of the model such that the selection causes presentation of a first set of images of the plurality of images obtained by the mobile device, the first set of images being selected from a set of images that were used to create the selected portion of the 3D model.

16. The computer readable medium of claim 15 , wherein the instructions further cause the processor to:

passively capture the plurality of images at regular time intervals.

17. The computer readable medium of claim 15 , wherein the instructions further cause the processor to capture the plurality of images in a manner that is undetectable by a user of the mobile device.

18. The computer readable medium of claim 15 , wherein the instructions further cause the processor to determine a measurement of anisotropy of each image such that images with anisotropy below a defined threshold are automatically deleted from the mobile device.

19. The method of claim 1 , further comprising estimating a lag between a hardware capture timestamp and an image system creation timestamp for each image of the plurality of opportunistically captured images.

Assignments (3)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 25, 2018
From: TEMPLEMAN, ROBERT E.
To: UNITED STATES OF AMERICA AS REPRESENTED BY THE SECRETARY OF THE NAVY
Reel/Frame 044734/0081 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 23, 2015
From: CRANDALL, DAVID; KAPADIA, APU; RAHMAN, MD ZAHIDUR
To: INDIANA UNIVERSITY RESEARCH & TECHNOLOGY CORPORATION
Reel/Frame 035477/0797 →
CONFIRMATORY LICENSE Recorded Jan 7, 2015
From: INDIANA UNIVERSITY RESEARCH AND TECHNOLOGY CORPORATION
To: NATIONAL SCIENCE FOUNDATION
Reel/Frame 034741/0888 →
Continuity (2)
Provisional Application 61866727 · Aug 16, 2013
Related Publication 20150049170A1 · Feb 19, 2015