IP Library › Granted Patent US 9,501,725
Granted Patent B2
US 9,501,725 · App. 14/302,056 · Granted Nov 22, 2016

Interactive and automatic 3-D object scanning method for the purpose of database creation

Inventors: Bojan Vrcelj (San Diego, CA); Daniel Knoblauch (San Diego, CA); Raghuraman Krishnamoorthi (San Diego, CA); Serafin Diaz Spindola (San Diego, CA)
Assignee: QUALCOMM Incorporated
G06K9/78G06K9/6228G06T7/0046G06T2207/10028G06T2207/20081
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 9,501,725
App. No.
14/302,056
Granted
Nov 22, 2016
Kind
B2
Abstract

Systems, methods, and devices are described for capturing compact representations of three-dimensional objects suitable for offline object detection, and storing the compact representations as object representation in a database. One embodiment may include capturing frames of a scene, identifying points of interest from different key frames of the scene, using the points of interest to create associated three-dimensional key points, and storing key points associated with the object as an object representation in an object detection database.

Claims (61)

1. A method of capturing compact representations of three-dimensional objects suitable for object detection comprising:

capturing, using a camera module of a device, a plurality of images of a scene, wherein each of the plurality of images of the scene captures at least a portion of an object;

identifying a first key frame from the plurality of images and a first position of the device associated with the first key frame;

identifying a second key frame from the plurality of images and a second position of the device associated with the second key frame, and wherein the second position is different from the first position;

identifying a first plurality of points of interest from the first key frame, wherein each of the first plurality of points of interest identify one or more features from the scene;

identifying a second plurality of points of interest from the second key frame, wherein each of the second plurality of points of interest identify one or more of the features from the scene;

matching a subset of the first plurality of points of interest and a subset of the second plurality of points of interest;

identifying a plurality of key points associated with the object based at least in part on the matching of the subset of the first plurality of points of interest and the subset of the second plurality of points of interest, and deleting points of interest with a mean distance to a threshold number of nearest points of interest that is more than a threshold distance; and

storing at least a portion of the plurality of key points associated with the object as an object representation in an object detection database.

2. The method of claim 1 wherein identifying the plurality of key points associated with the object comprises:

filtering the first plurality of points of interest and the second plurality of points of interest to identify the at least a portion of the plurality of key points which are associated with the object.

3. The method of claim 2 wherein filtering the first plurality of points of interest and the second plurality of points of interest comprises one or more of:

deleting the points of interest that are not matched with points of interest from other key frames; and

deleting the key points outside of a defined volume of interest of the scene.

4. The method of claim 1 wherein the scene further comprises a planar target.

5. The method of claim 4 wherein the planar target is used to assist in defining a volume of interest of the scene.

6. The method of claim 4 wherein matching the first plurality of points of interest and the second plurality of points of interest comprises:

identifying the first position of the device from a first location of the planar target in the first image;

identifying the second position of the device from a second location of the planar target in the second image;

determining a relative position between the first position of the device and the second position of the device;

matching the first plurality of points of interest and the second plurality of points of interest based on the relative position between the first position and the second position; and

determining and recording a position of each key point in a coordinate system.

7. The method of claim 1 wherein each key point comprises key point location information and for pixels surrounding the key point.

8. The method of claim 1 wherein identifying the first image as the first key frame comprises a user selection.

9. The method of claim 1 wherein identifying the first image as the first key frame comprises an automatic selection by the device.

10. The method of claim 9 wherein identifying the second image as the second key frame comprises:

identifying a key point density within the second image;

identifying a spatial relationship between the second position and the first position;

determining that a key frame at the second position would provide data with a data value above a threshold value for use in the object representation; and

selecting the second image as the second key frame.

11. A device for capturing compact representations of three-dimensional objects suitable for offline object detection comprising:

a camera module of a device that captures a plurality of images of a scene, wherein each of the plurality of images of the scene captures at least a portion of an object;

one or more processors that (1) identifies a first key frame and a first position of the device associated with the first key frame; (2) identifies a second key frame and a second position of the device associated with the second key frame, wherein the second position is different from the first position; (3) identifies a first plurality of points of interest from the first key frame, wherein the first plurality of points of interest identify features from the scene; (4) identifies a second plurality of points of interest from the second key frame, wherein the second plurality of points of interest identify at least a portion of the features from the scene; (5) matches a portion of the first plurality of points of interest and a portion the second plurality of points of interest; and (6) identifies a plurality of key points associated with the object based at least in part on the matching of the portion of the first plurality of points of interest and the portion of the second plurality of points of interest and deleting points of interest with a mean distance to a threshold number of nearest points of interest that is more than a threshold distance; and

a memory that stores at least a portion of the plurality of key points associated with the object as an object representation in an object detection database.

12. The device of claim 11 wherein the device further comprises:

a display coupled to the camera module, wherein the display outputs an image of at least a portion of the plurality of key points as the camera module of the device captures at least a portion of the plurality of images of the scene.

13. The device of claim 12 wherein the display further outputs a video image of the scene with at least a second portion of the plurality of key points overlaid on the object.

14. The device of claim 11 wherein the device further comprises:

a motion sensor, wherein the second position of the device is identified by the one or more processors using information from the motion sensor.

15. The device of claim 11 wherein the device further comprises:

a user input module, wherein identifying the first key frame comprises a user selection received at the user input module of the device.

16. The device of claim 15 wherein the device further comprises:

an antenna; and

a wireless transceiver;

wherein the one or more processors are coupled to the device via a network, the antenna, and the wireless transceiver.

17. A non-transitory computer-readable medium comprising instructions that, when executed by a processor coupled to the non-transitory computer-readable medium cause a device to:

capture, using a camera module of the device, a plurality of images of a scene, wherein each of the plurality of images of the scene captures at least a portion of an object;

identify a first key frame and a first position of the device associated with the first key frame;

identify a second key frame and a second position of the device associated with the second key frame, wherein the second position is different from the first position;

identify a first plurality of points of interest from the first key frame, wherein the first plurality of points of interest identify features from the scene;

identify a second plurality of points of interest from the second key frame, wherein the second plurality of points of interest identify at least a portion of the features from the scene;

match a portion of the first plurality of points of interest and a portion of the second plurality of points of interest;

identify a plurality of key points associated with the object based at least in part on the match of the portion of the first plurality of points of interest and the portion of the second plurality of points of interest and deleting points of interest with a mean distance to a threshold number of nearest points of interest that is more than a threshold distance; and

store at least a portion of the plurality of key points associated with the object as an object representation in an object detection database.

18. The non-transitory computer-readable medium of claim 17 wherein the instructions, when executed by the processor, further cause the device to:

filter the first plurality of points of interest and the second plurality of points of interest to identify the plurality of points of interest associated with the object as part of identifying key points associated with the object.

19. The non-transitory computer-readable medium of claim 18 wherein the instructions, when executed by the processor, further cause the device to:

delete the points of interest that are not matched with points of interest from other key frames as part of the filtering the first plurality of points of interest and the second plurality of points of interest to identify points of interest associated with the object.

20. The non-transitory computer-readable medium of claim 18 wherein the instructions, when executed by the processor, further cause the device to:

delete a set of key points outside of a defined volume of the object as part of the filtering the first plurality of points of interest and the second plurality of points of interest to identify points of interest associated with the object; and

wherein each key point of the plurality of key points associated with the object as the object representation in the object detection database comprises coordinate information, brightness information, and surrounding pixel pattern information.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 29, 2014
From: VRCELJ, BOJAN; KNOBLAUCH, DANIEL; KRISHNAMOORTHI, RAGHURAMAN; SPINDOLA, SERAFIN DIAZ
To: QUALCOMM INCORPORATED
Reel/Frame 034595/0503 →
Continuity (2)
Provisional Application 61833889 · Jun 11, 2013
Related Publication 20140363048A1 · Dec 11, 2014