IP Library Granted Patent US 9,519,973
Granted Patent B2
US 9,519,973 · App. 14/020,873 · Granted Dec 13, 2016

Enabling use of three-dimensional locations of features images

Inventors: Eagle Sunrise Jones (San Francisco, CA); Benjamin J. Hirashima (San Francisco, CA); Jordan H. Miller (San Francisco, CA)
Assignee: Intel Corporation
G06T7/0071G06T7/0044G06T2200/08
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 9,519,973
App. No.
14/020,873
Granted
Dec 13, 2016
Kind
B2
Abstract

The disclosed embodiments provide a system that facilitates use of an image. During operation, the system uses a set of images from a camera on a device to obtain a set of features in proximity to the device, wherein the set of images comprises the image. Next, the system uses the set of images and inertial data from one or more inertial sensors on the device to obtain a set of three-dimensional (3D) locations of the features. Finally, the system enables use of the set of 3D locations with the image.

Claims (81)

1. A computer-implemented method for facilitating use of an image, comprising:

using a set of images from a camera on a device to obtain a set of features in proximity to the device;

using the set of images and inertial data from one or more inertial sensors on the device to generate a three-dimensional (3D) model of an environment around the device;

obtaining a selection of the image from the set of images;

selecting, as a subset of the generated 3D model, a set of 3D locations of the features and a set of 2D locations of the features in the image;

storing the selected subset of the 3D model comprising the 3D locations and the 2D locations as metadata for the image for subsequent use of the 3D locations with the features in the image; and

omitting a remainder of the 3D model that is not in the subset from the metadata.

2. The computer-implemented method of claim 1 , further comprising:

enabling use of the 3D locations with one or more additional images from the set of images.

3. The computer-implemented method of claim 2 , wherein the image and the one or more additional images are associated with at least one of:

a panorama; and

two or more images from an image sequence.

4. The computer-implemented method of claim 1 , wherein using the set of images and the inertial data from the one or more inertial sensors to generate the 3D model of the environment around the device comprises:

obtaining a new image from the set of images and a measurement of a 2D location, in the new image, of a feature from the set of features;

providing the inertial data and the measurement as input to generate the 3D model;

using the 3D model to predict a 2D location of the feature in the new image;

and

using a residual between the predicted 2D location and the measurement to estimate a 3D location of the feature.

5. The computer-implemented method of claim 1 , wherein the 3D locations and the 2D locations are embedded in the image.

6. The computer-implemented method of claim 1 , further comprising:

obtaining a selection of one or more points in the image; and

using the 3D locations of one or more features in the metadata to measure one or more attributes associated with the one or more points.

7. The computer-implemented method of claim 6 , wherein the one or more attributes comprise at least one of:

a distance;

an area;

a volume; and

a dimension.

8. The method of claim 6 , wherein the one or more points comprise at least one of:

a feature from the set of features; and

a point in a shape defined using the feature.

9. The computer-implemented method of claim 1 , wherein the set of features comprises at least one of:

a corner;

an edge; and

a specialized feature.

10. The computer-implemented method of claim 1 , wherein the device is a portable electronic device.

11. The method of claim 1 , further comprising:

displaying graphical objects identifying the 2D locations of the features in the image.

12. The method of claim 1 , further comprising:

using the metadata to calculate a 3D location of a point in the image that is not included in the 2D locations of the features.

13. A system for facilitating use of an image, comprising:

a camera configured to obtain a set of images;

one or more inertial sensors configured to obtain inertial data associated with the camera;

an analysis apparatus configured to:

use the set of images to obtain a set of features in proximity to the camera;

use the set of images and the inertial data to generate a three-dimensional (3D) model of an environment around the device;

obtain a selection of the image from the set of images; and

select, as a subset of the generated 3D model, a set of 3D locations of the features and a set of 2D locations of the features in the image; and

a management apparatus configured to:

store the selected subset of the 3D model comprising the 3D locations and the 2D locations as metadata for the image for subsequent use of the 3D locations with the features in the image; and

omit a remainder of the 3D model that is not in the subset from the metadata.

14. The system of claim 13 , wherein using the set of images and the inertial data to generate the 3D model of the environment around the device comprises:

obtaining a new image from the set of images and a measurement of a 2D location, in the new image, of a feature from the set of features;

providing the inertial data and the measurement as input to generate the 3D model;

using the 3D model to predict a 2D location of a feature from the set of features in the new image;

and

using a residual between the predicted 2D location and the measurement to estimate a 3D location of the feature.

15. The system of claim 13 , wherein the management apparatus is further configured to:

obtain a selection of one or more points in the image; and

use the 3D locations of one or more features in the metadata to measure one or more attributes associated with the one or more points.

16. The system of claim 15 , wherein the one or more attributes comprise at least one of:

a distance;

an area;

a volume; and

a dimension.

17. A non-transitory computer-readable storage medium containing instructions embodied therein for causing a computer system to perform a method for facilitating use of an image, the method comprising:

using a set of images from a camera on a device to obtain a set of features in proximity to the device;

using the set of images and inertial data from one or more inertial sensors on the device to generate a three-dimensional (3D) model of an environment around the device;

obtaining a selection of the image from the set of images;

selecting, as a subset of the generated 3D model, a set of 3D locations of the features and a set of 2D locations of the features in the image;

storing the selected subset of the 3D model comprising the 3D locations and the 2D locations as metadata for the image for subsequent use of the 3D locations with the features in the image; and

omitting a remainder of the 3D model that is not in the subset from the metadata.

18. The non-transitory computer-readable storage medium of claim 17 , the method further comprising:

enabling use of the 3D locations with one or more additional images from the set of images.

19. The non-transitory computer-readable storage medium of claim 17 , the method further comprising:

obtaining a selection of one or more points in the image; and

using the 3D locations of one or more features in the metadata to measure one or more attributes associated with the one or more points.

20. The non-transitory computer-readable storage medium of claim 19 , wherein the one or more attributes comprise at least one of:

a distance;

an area;

a volume; and

a dimension.

Assignments (3)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 4, 2026
From: INTEL CORPORATION
To: REALSENSE, INC.
Reel/Frame 074071/0151 →
MERGER Recorded Jul 24, 2016
From: REALITYCAP, INC.
To: INTEL CORPORATION
Reel/Frame 039238/0666 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 8, 2013
From: JONES, EAGLE S.; HIRASHIMA, BENJAMIN J.; MILLER, JORDAN H.
To: REALITYCAP, INC.
Reel/Frame 031160/0253 →
Continuity (1)
Related Publication 20150070352A1 · Mar 12, 2015