IP Library › Granted Patent US 11,483,531
Granted Patent B2
US 11,483,531 · App. 17/568,398 · Granted Oct 25, 2022

Generation of three-dimensional images with digital magnification

Inventors: Yang Liu (Westlake, OH); Maziyar Askari Karchegani (Coralville, IA)
Assignee: Unify Medical, Inc.
H04N13/106G06T3/40H04N5/22525H04N13/239G06T2210/22
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,483,531
App. No.
17/568,398
Granted
Oct 25, 2022
Kind
B2
Abstract

A system for generating three-dimensional (3D) images from captured images of a target when executing digital magnification. A controller executes a digital magnification on the first image of the target captured by the first image sensor and on the second image captured by the second image sensor of the target. The controller crops the first image and the second image to overlap a first portion of the target captured by the first image sensor with a second portion of the target captured by the second image sensor. The controller adjusts the cropping of the first image and the second image to provide binocular overlap of the first portion of the target with the second portion of target. The displayed cropped first image and the cropped second image display the 3D image at the digital magnification to the user.

Claims (61)

1. A system for generating three-dimensional (3D) images from captured images of a target when executing digital magnification on the captured images to maintain the 3D images generated of the target after digital magnification, comprising:

a controller configured to:

generate a 3D image based on binocular overlap of a first image of the target captured by a first image sensor and a second image of the target captured by a second image sensor as the digital magnification of the target is adjusted to each corresponding digital magnification level, wherein the binocular overlap of the first image and the second image is maintained for each digital magnification level thereby maintaining alignment of the first image and the second image as depicted by the 3D image, and

instruct a display to display each 3D image at each corresponding magnification level with the binocular overlap in each 3D image maintained thereby preventing a display of a misaligned 3D image to the user as the digital magnification of the target is adjusted.

2. The system of claim 1 , wherein the controller is further configured to:

automatically adjust the binocular overlap of the first image and the second image as the user adjusts the digital magnification level;

generate the 3D image based on the adjusted the binocular overlap of the first image and the second image at the adjusted digital magnification level as adjusted by the user, wherein the binocular overlap is maintained as the user adjusts the digital magnification level thereby maintaining alignment of the first image and the second image as depicted by the 3D image; and

instruct the display to display the 3D image at the adjusted digital magnification level with the binocular overlap in the 3D image maintained thereby preventing a display of a misaligned 3D image to the user as the digital magnification of the target is adjusted by the user.

3. The system of claim 1 , wherein the controller is further configured to:

execute a digital magnification on the first image sensor and on the second image captured by the second image sensor;

crop the first image and the second image to overlap a first portion of the target captured by the first image sensor with a second portion of the target captured by the second image sensor, wherein the first portion of the target overlaps with the second portion of the target;

adjust the cropping of the first image and the second image to provide binocular overlap of the first portion of the target with the second portion of the target, wherein the binocular overlap of the first image and the second image is an overlap threshold that when satisfied results in the 3D image of the target displayed to the user after the digital magnification is executed; and

instruct a display to display the cropped first image and the cropped second image that includes the binocular overlap to the user, wherein the displayed cropped first image and the cropped second image display the 3D image at the digital magnification to the user.

4. The system of claim 3 , wherein the controller is further configured to:

resize the cropped first image to the original size of the first image captured by the first image sensor and the cropped second image to the original size of the second image captured by the second image sensor, wherein the cropped first image as resized and the cropped second image as resized includes the binocular overlap of the first image and the second image; and

instruct the display to display the resized and cropped first image and the resized and cropped second image that includes the binocular overlap to the user, wherein the displayed resized and cropped first image and the resized and cropped second image display the 3D image at the digital magnification to the user.

5. The system of claim 4 , wherein the controller is further configured to:

crop the first image captured by the first image sensor and the second image captured by the second image sensor to vertically align the overlap of the first portion of the target with the second portion of the target, wherein the cropped first image is in vertical alignment of the cropped second image when a first plurality of vertical coordinates of the cropped first image is aligned with each corresponding vertical coordinate from a second plurality of coordinates of the cropped second image;

adjust the cropping of the first image and the second image to provide binocular overlap of the first portion of the target with the second portion of the target, wherein the binocular overlap of the first image and the second image is vertically aligned to satisfy the overlap threshold to generate the 3D image of the target displayed to the user after the digital magnification is executed.

6. The system of claim 5 , wherein the controller is further configured to:

after executing a first digital magnification at a first digital magnification level on the first image captured by the first image sensor and on the second image captured by the second image sensor, maintain the binocular overlap generated by adjusting the cropping of the first image and the second image to satisfy the overlap threshold;

execute a second digital magnification at a second digital magnification level on the first image captured by the first image sensor and on the second image captured by the second image sensor, wherein the second digital magnification level is increased from the first digital magnification level; and

maintain the binocular overlap generated after executing the first digital magnification at the first digital magnification level on the first image and the second image to when executing the second digital magnification at the second digital magnification level.

7. The system of claim 6 , wherein the controller is further configured to:

after executing each previous digital magnification at each previous digital magnification level on the first image and the second image, maintain the binocular overlap and the vertical alignment determined when executing the first digital magnification at the first digital magnification level on the first image and the second image; and

continue to maintain the binocular overlap and the vertical alignment determined from the adjusting of the cropping of the first image and the second image to satisfy the overlap threshold after executing the first digital magnification at the first digital magnification level on the first image and the second image for each subsequent digital magnification at each subsequent digital magnification level, wherein each subsequent digital magnification level is increased from each previous digital magnification level.

8. The system of claim 6 , wherein the controller is further configured to:

execute the first digital magnification at the first digital magnification level on a non-concentric portion of the first image and a non-concentric portion of the second image, wherein the non-concentric portion of the first image and the second image is a portion of the first image and the second image that differs from a center of the first image and the second image;

adjust the cropping of the first image and the second image to provide binocular overlap of the non-concentric portion of the first image and the non-concentric portion of the second image, wherein the binocular overlap of the non-concentric portion of the first image and the non-concentric portion of the second image satisfies the overlap threshold; and

continue to crop a non-concentric portion of the first image and a non-concentric portion of the second image for each subsequent digital magnification at each subsequent digital magnification level, wherein the binocular overlap of the non-concentric portion of the first image and the non-concentric portion of the second image is maintained from the first digital magnification at the first digital magnification level.

9. The system of claim 6 , wherein the controller is further configured to:

determine a distance that the first image sensor and the second image sensor is positioned from the target;

execute the cropping of the first image and the second image to maintain the vertical alignment and the binocular overlap for each digital magnification at each digital magnification level based on the distance of the first image sensor and the second image sensor from the target.

10. The system of claim 6 , further comprising at least one wearable display that displays the resized and cropped first image and the resized and cropped second image to display the 3D image of the target after the digital magnification is executed that includes the binocular overlap of the first image and the second image that are vertically aligned to satisfy the overlap threshold.

11. A method for generating three-dimensional (3D) images from captured images of a target when executing digital magnification on the captured images to maintain the 3D images generated of the target after digital magnification, comprising:

generating a 3D image based on binocular overlap of a first image of the target captured by a first image sensor and a second image of the target captured by a second image sensor as the digital magnification of the target is adjusted to each corresponding digital magnification level, wherein the binocular overlap of the first image and the second image is maintained for each digital magnification level thereby maintaining alignment of the first image and the second image as depicted by the 3D image, and

instructing a display to display each 3D image at each corresponding magnification level with the binocular overlap in each 3D image maintained thereby preventing a display of a misaligned 3D image to the user as the digital magnification of the target is adjusted.

12. The method of claim 11 , further comprising:

automatically adjusting the binocular overlap of the first image and the second image as the user adjusts the digital magnification level;

generating the 3D image based on the adjusted binocular overlap of the first image and the second image at the adjusted digital magnification level as adjusted by the user, wherein the binocular overlap is maintained as the user adjusts the digital magnification level thereby maintaining alignment of the first image and the second image as depicted by the 3D image; and

instructing the display to display the 3D image at the adjusted digital magnification level with the binocular overlap in the 3D image maintained thereby preventing a display of a misaligned 3D image to the user as the digital magnification of the target is adjusted by the user.

13. The method of claim 11 , further comprising:

executing a digital magnification on the first image sensor and on the second image captured by the second image sensor;

cropping the first image and the second image to overlap a first portion of the target captured by the first image sensor with a second portion of the target captured by the second image sensor, wherein the first portion of the target overlaps with the second portion of the target;

adjusting the cropping of the first image and the second image to provide binocular overlap of the first portion of the target with the second portion of the target, wherein the binocular overlap of the first image and the second image is an overlap threshold that when satisfied results in the 3D image of the target displayed to the user after the digital magnification is executed; and

instructing a display to display the cropped first image and the cropped second image that includes the binocular overlap to the user, wherein the displayed cropped first image and the cropped second image display the 3D image at the digital magnification to the user.

14. The method of claim 13 , further comprising:

resizing the cropped first image to the original size of the first image captured by the first image sensor and the cropped second image to the original size of the second image captured by the second image sensor, wherein the cropped first image as resized and the cropped second image as resized includes the binocular overlap of the first image and the second image; and

instructing the display to display the resized and cropped first image and the resized and cropped second image that includes the binocular overlap to the user, wherein the displayed resized and cropped first image and the resized and cropped second image display the 3D image at the digital magnification to the user.

15. The system of claim 14 , further comprising:

cropping the first image captured by the first image sensor and the second image captured by the second image sensor to vertically align the overlap of the first portion of the target with the second portion of the target, wherein the cropped first image is in vertical alignment of the cropped second image when each vertical coordinate of the cropped first image is aligned with each corresponding vertical coordinate of the cropped second image; and

adjusting the cropping of the first image and the second image to provide binocular overlap of the first portion of the target with the second portion of the target, wherein the binocular overlap of the first image and the second image is vertically aligned to satisfy the overlap threshold to generate the 3D image of the target displayed to the user after the digital magnification is executed.

16. The method of claim 15 , further comprising:

after executing a first digital magnification at a first digital magnification level on the first image captured by the first image sensor and on the second image captured by the second image sensor, locking in the binocular overlap generated by adjusting the cropping of the first image and the second image to satisfy the overlap threshold;

executing a second digital magnification at a second digital magnification level on the first image captured by the first image sensor and on the second image captured by the second image sensor, wherein the second digital magnification level is increased from the first digital magnification level; and

maintaining the binocular overlap generated after executing the first digital magnification at the first digital magnification level on the first image and the second image when executing the second digital magnification at the second digital magnification level.

17. The method of claim 14 , further comprising: after executing each previous digital magnification at each previous digital magnification level on the first image and the second image, maintaining the binocular overlap and the vertical alignment determined when executing the first digital magnification at the first digital magnification level on the first image and the second image; and continuing to maintain the binocular overlap and the vertical alignment determined from the adjusting of the cropping of the first image and the second image to satisfy the overlap threshold after executing the first digital magnification at the first digital magnification level on the first image and the second image for each subsequent digital magnification at each subsequent digital magnification level,

wherein each subsequent digital magnification level is increased from each previous digital magnification level.

18. The method of claim 14 , further comprising: executing the first digital magnification at the first digital magnification level on a non-concentric portion of the first image and on a non-concentric portion of the second image, wherein the non-concentric portion of the first image and the second image is a portion of the first image and the second image that differs from a center of the first image and the second image; adjusting the cropping of the first image and the second image to provide binocular overlap of the non-concentric portion of the first image and the non-concentric portion of the second image, wherein the binocular overlap of the non-concentric portion of the first image and the non-concentric portion of the second image satisfies the overlap threshold; and continuing to capture a non-concentric portion of the first image and a non-concentric portion of the second image for each subsequent digital magnification at each subsequent digital magnification level, wherein the binocular overlap of the non-concentric portion of the first image and the non-concentric portion of the second image is maintained from the first digital magnification at the first digital magnification level.

19. The method of claim 14 , further comprising: determining a distance that the first image sensor and the second image sensor is positioned from the target; and executing the cropping of the first image and the second image to maintain the vertical alignment and the binocular overlap for digital magnification at a digital magnification level based on the distance of the first image sensor and the second image sensor from the target.

20. The method of claim 14 , further comprising: displaying by a wearable display the resized and cropped first image and the resized and cropped second image to display the 3D image of the target after the digital magnification is executed that includes the binocular overlap of the first image and the second image that are vertically aligned to satisfy the overlap threshold.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 5, 2022
From: LIU, YANG; KARCHEGANI, MAZIYAR ASKARI
To: HAPPYCAM L.L.C.
Reel/Frame 058557/0831 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 5, 2022
From: HAPPYCAM L.L.C.
To: UNIFY MEDICAL, INC.
Reel/Frame 058557/0859 →
Continuity (3)
Continuation 17331579 · May 26, 2021
Provisional Application 63029831 · May 26, 2020
Related Publication 20220132091A1 · Apr 28, 2022