IP Library Granted Patent US 10,810,759
Granted Patent B2
US 10,810,759 · App. 16/196,087 · Granted Oct 20, 2020

Creating a three-dimensional model from a sequence of images

Inventors: Thomas G. Zimmerman (Cupertino, CA); Simone Bianco (San Jose, CA); Rebecca M. McGillivary (San Francisco, CA); Wallace F. Marshall (San Francisco, CA)
Assignees: International Business Machines Corporation; The Regents of the University of California
G06T7/73G06T3/0068G06T7/11G06T17/00G06T2200/08G06T2207/20048G06T2207/20132G06T2207/30204G06T2210/56
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,810,759
App. No.
16/196,087
Granted
Oct 20, 2020
Kind
B2
Abstract

A computer-implemented method according to one embodiment includes identifying a plurality of two-dimensional (2D) images illustrating a subject performing a rotation, selecting a representative image of the subject, cropping the plurality of 2D images, utilizing the representative image of the subject, to create a cropped plurality of 2D images, aligning each of the cropped plurality of 2D images, thereby creating an aligned plurality of 2D images, selecting a subset of the aligned plurality of 2D images that illustrate a predetermined amount of rotation of the subject, and creating a three-dimensional (3D) point cloud of the subject, utilizing the subset of the aligned plurality of 2D images.

Claims (63)

1. A computer-implemented method, comprising:

identifying a plurality of two-dimensional (2D) images illustrating a subject performing a rotation;

selecting a representative image of the subject;

cropping the plurality of 2D images, utilizing the representative image of the subject, to create a cropped plurality of 2D images;

aligning each of the cropped plurality of 2D images, thereby creating an aligned plurality of 2D images;

selecting a subset of the aligned plurality of 2D images that illustrate a predetermined amount of rotation of the subject; and

creating a three-dimensional (3D) point cloud of the subject, utilizing the subset of the aligned plurality of 2D images.

2. The computer-implemented method of claim 1 , wherein the plurality of 2D images illustrate the subject rotating at least 180 degrees.

3. The computer-implemented method of claim 1 , wherein the plurality of 2D images are captured using a single 2D camera having a single image sensor.

4. The computer-implemented method of claim 1 , wherein the plurality of 2D images are captured using a monocular microscope.

5. The computer-implemented method of claim 1 , wherein the representative image includes a selected portion of one of the plurality of 2D images.

6. The computer-implemented method of claim 1 , wherein the representative image of the subject is used to track the subject within each of the plurality of 2D images.

7. The computer-implemented method of claim 1 , wherein cropping the plurality of 2D images includes cropping the subject from each of the plurality of 2D images to obtain only an image of the subject.

8. The computer-implemented method of claim 1 , wherein aligning the cropped plurality of 2D images includes removing any vertical or horizontal movement of the subject throughout the cropped plurality of 2D images, thereby creating an aligned plurality of images illustrating only a rotational movement of the subject.

9. The computer-implemented method of claim 1 , wherein the predetermined amount of rotation includes 180 degrees of rotation.

10. The computer-implemented method of claim 1 , wherein selecting the subset of the aligned plurality of 2D images that illustrate the predetermined amount of rotation of the subject includes:

identifying a feature of the subject,

marking the identified feature of the subject within the aligned plurality of 2D images,

tracking the identified feature over each of the aligned plurality of 2D images to create a 2D projection,

identifying a peak marker location and a trough marker location within the 2D projection,

identifying images within the aligned plurality of 2D images that are associated with the peak marker location and the trough marker location, and

selecting the subset of the aligned plurality of 2D images so that it includes all images occurring between the images associated with the peak marker location and the trough marker location.

11. The computer-implemented method of claim 1 , wherein creating the three-dimensional (3D) point cloud of the subject includes, for each of a plurality of X-slices determined for each of the subset of the aligned plurality of 2D images:

calculating a radon transform for the X-slice to create a sinogram,

determining a Fourier transform of the sinogram,

interpolating the Fourier transform in Fourier space to obtain a 2D Fourier transform of the subject, and

inverting the interpolation to recreate an X-slice image.

12. A computer program product for creating a three-dimensional model from a sequence of images, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, wherein the computer readable storage medium is not a transitory signal per se, the program instructions executable by a processor to cause the processor to perform a method comprising:

identifying, by the processor, a plurality of two-dimensional (2D) images illustrating a subject performing a rotation;

selecting, by the processor, a representative image of the subject;

cropping, by the processor, the plurality of 2D images, utilizing the representative image of the subject, to create a cropped plurality of 2D images;

aligning, by the processor, each of the cropped plurality of 2D images, thereby creating an aligned plurality of 2D images;

selecting, by the processor, a subset of the aligned plurality of 2D images that illustrate a predetermined amount of rotation of the subject; and

creating, by the processor, a three-dimensional (3D) point cloud of the subject, utilizing the subset of the aligned plurality of 2D images.

13. The computer program product of claim 12 , wherein the plurality of 2D images illustrate the subject rotating at least 180 degrees.

14. The computer program product of claim 12 , wherein the plurality of 2D images are captured using a single 2D camera having a single image sensor.

15. The computer program product of claim 12 , wherein the plurality of 2D images are captured using a monocular microscope.

16. The computer program product of claim 12 , wherein the representative image includes a selected portion of one of the plurality of 2D images.

17. The computer program product of claim 12 , wherein the representative image of the subject is used to track the subject within each of the plurality of 2D images.

18. The computer program product of claim 12 , wherein cropping the plurality of 2D images includes cropping the subject from each of the plurality of 2D images to obtain only an image of the subject.

19. The computer program product of claim 12 , wherein aligning the cropped plurality of 2D images includes removing any vertical or horizontal movement of the subject throughout the cropped plurality of 2D images, thereby creating an aligned plurality of images illustrating only a rotational movement of the subject.

20. A system, comprising:

a processor; and

logic integrated with the processor, executable by the processor, or integrated with and executable by the processor, the logic being configured to:

identify a plurality of two-dimensional (2D) images illustrating a subject performing a rotation;

select a representative image of the subject;

crop the plurality of 2D images, utilizing the representative image of the subject, to create a cropped plurality of 2D images;

align each of the cropped plurality of 2D images, thereby creating an aligned plurality of 2D images;

select a subset of the aligned plurality of 2D images that illustrate a predetermined amount of rotation of the subject; and

create a three-dimensional (3D) point cloud of the subject, utilizing the subset of the aligned plurality of 2D images.

21. A computer-implemented method, comprising:

identifying a plurality of two-dimensional (2D) images illustrating a subject performing a rotation;

selecting a representative image of the subject;

cropping the plurality of 2D images, utilizing the representative image of the subject, to create a cropped plurality of 2D images;

aligning each of the cropped plurality of 2D images, thereby creating an aligned plurality of 2D images;

selecting a subset of the aligned plurality of 2D images that illustrate a predetermined amount of rotation of the subject, including:

identifying a feature of the subject,

marking the identified feature of the subject within the aligned plurality of 2D images,

tracking the identified feature over each of the aligned plurality of 2D images to create a 2D projection,

identifying a peak marker location and a trough marker location within the 2D projection,

identifying images within the aligned plurality of 2D images that are associated with the peak marker location and the trough marker location, and

selecting the subset of the aligned plurality of 2D images so that it includes all images occurring between the images associated with the peak marker location and the trough marker location; and

creating a three-dimensional (3D) point cloud of the subject, utilizing the subset of the aligned plurality of 2D images.

Assignments (2)
CONFIRMATORY LICENSE Recorded Dec 4, 2019
From: UNIVERSITY OF CALIFORNIA, SAN FRANCISCO
To: NATIONAL SCIENCE FOUNDATION
Reel/Frame 051186/0304 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 11, 2019
From: ZIMMERMAN, THOMAS G.; BIANCO, SIMONE
To: INTERNATIONAL BUSINESS MACHINES CORPORATION
Reel/Frame 048303/0542 →
Continuity (1)
Related Publication 20200160554A1 · May 21, 2020