Granted Patent B2

US 12,262,866 · App. 17/508,865 · Granted Apr 1, 2025

Inventors: Christian Voigt (Abtsgmuend, DE); Stefan Saur (Aalen, DE); Christoph Hauger (Aalen, DE); Martin Fanenbruck (Oberkochen, DE); Helge Jess (Oberkochen, DE); Roland Guckler (Ulm, DE)

Assignee: Carl Zeiss Meditec AG

A61B1/0005A61B1/00006A61B1/00183G06T7/0012G06T7/30G06T2207/10068G06T2207/20081G06T2207/30004

Abstract

Claims (104)

1. A visualization system for operating an optical inspection tool, the visualization system comprising:

an observation apparatus having a first image recording device configured to observe an operation region at a first observation plane having a first observation plane axis and a second observation plane axis and defining a first viewing axis which is perpendicular to the first observation plane axis and the second observation plane axis;

the optical inspection tool having a second image recording device configured to observe the operation region at a second observation plane having a third observation plane axis and a fourth observation plane axis and defining a second viewing axis which is perpendicular to the third observation plane axis and the fourth observation plane axis;

a display device configured to represent at least one of a first image recorded by the first image recording device and a second image recorded by the second image recording device;

a tracking system including a target detection device and at least one target and being configured to determine an orientation of the optical inspection tool relative to the observation apparatus;

a controller including a memory and a processor in communication with the display device, the first image recording device, the second image recording device, the tracking system, and the memory; and

the processor being configured to transform the second image based on the orientation of the optical inspection tool relative to the observation apparatus,

wherein to transform the second image, the processor is further configured to:

generate a projected observation plane by projecting the second observation plane onto the first observation plane, wherein the projected observation plane has a projected third observation plane axis and a projected fourth observation plane axis and defines a projected second viewing axis which is aligned perpendicular to the projected third observation plane axis and the projected fourth observation plane axis, and wherein the projected third observation plane axis, the projected fourth observation plane axis, and projected second viewing axis define a projected coordinate system;

determine a rotation angle which indicates a rotation of the projected coordinate system about the projected second viewing axis such that the projected third observation plane axis is aligned parallel to and equally oriented with the first observation plane axis, and the projected fourth observation plane axis is aligned parallel to and equally oriented with the second observation plane axis; and

rotate the second image about the rotation angle about the projected second viewing axis.

2. The visualization system of claim 1 , wherein:

the observation apparatus is a microscope,

the optical inspection tool is an endoscope,

the target detection device is a camera, and

the at least one target is a marker.

3. The visualization system of claim 1 , wherein to transform the second image, the processor is further configured to:

define a vertical axis of the second image; and

reflect the second image on the vertical axis.

4. The visualization system of claim 1 , wherein:

the second image is transformed by training the visualization system, and

to transform the second image relative to the first image by training:

the second image is repeatedly manually rotated about the second viewing axis corresponding to a rotation angle depending on the orientation of the optical inspection tool relative to the observation apparatus; and

the processor is further configured to:

store values of the rotation angle in a training database each time the second image is rotated about the rotation angle;

compare the values previously stored in the training database with the values subsequently stored in the training database; and

automatically rotate the second image about the rotation angle based on the training of the visualization system.

5. A visualization system for operating an optical inspection tool, the visualization system comprising:

an observation apparatus having a first image recording device configured to observe an operation region at a first observation plane having a first observation plane axis and a second observation plane axis and defining a first viewing axis which is perpendicular to the first observation plane axis and the second observation plane axis;

the optical inspection tool having a second image recording device configured to observe the operation region at a second observation plane having a third observation plane axis and a fourth observation plane axis and defining a second viewing axis which is perpendicular to the third observation plane axis and the fourth observation plane axis;

a display device configured to represent at least one of a first image recorded by the first image recording device and a second image recorded by the second image recording device;

a tracking system including a target detection device and at least one target and being configured to determine an orientation of the optical inspection tool relative to the observation apparatus;

a controller including a memory and a processor in communication with the display device, the first image recording device, the second image recording device, the tracking system, and the memory; and

the processor being configured to transform the second image based on the orientation of the optical inspection tool relative to the observation apparatus,

wherein:

to transform the second image, the processor is further configured to;

define a reference plane,

generate a projected first observation plane by projecting the first observation plane onto the reference plane, wherein the projected first observation plane has a projected first observation plane axis and a projected second observation plane axis and defines a projected first viewing axis; and

when the second viewing axis is aligned perpendicular to the reference plane, determine a first rotation angle α 1 and rotate the second image about the first rotation angle α 1 about the projected second viewing axis such that a rotated third observation plane axis of the rotated second observation plane is aligned parallel to and equally oriented with the projected first observation plane axis, and the projected fourth observation plane axis is aligned parallel to and equally oriented with the second observation plane axis,

wherein the reference plane is a plane having a first reference plane axis and a second reference plane axis, and

wherein the first and second reference plane axes are aligned perpendicular to a gravity.

6. The visualization system of claim 5 , wherein to transform the second image, the processor is further configured to:

define a horizontal plane and a vertical plane, wherein the horizontal plane is aligned parallel to the reference plane and the vertical plane is aligned perpendicular to the reference plane;

generate a projected horizontal observation plane by projecting the second observation plane onto the horizontal plane and a projected vertical observation plane by projecting the second observation plane onto the vertical observation plane;

determine a first rotation angle α 1 such that a rotated projected third observation plane axis of the projected horizontal observation plane is aligned parallel to and equally oriented with the projected first observation plane axis;

determine a second rotation angle α 2 such that a rotated projected fourth observation plane axis of the projected vertical observation plane is directed away from and perpendicular to the reference plane in a direction opposite to the gravity;

determine a tilt angle β relative to the reference plane;

determine a third rotation angle α 3 based on the first rotation angle α 1 , the second rotation angle α 2 , and the tilt angle β; and

rotate the second image about the third rotation angle α 3 about the projected second viewing axis.

7. The visualization system of claim 6 , wherein the third rotation angle α 3 is determined in accordance with

α 3 =g (β)·α 1 +(1− g (β))·α 2 , and

wherein α 1 is the first rotation angle, α 2 is the second rotation angle, and g(β) is a function of the tilt angle β.

8. The visualization system of claim 7 , wherein:

a value of the function g(β) of the tilt angle β is 0 when the tilt angle β is 0°,

the value of the function g(β) of the tilt angle β is 1 when the tilt angle β is 90°,

the function g(β) of the tilt angle β is monotonically increasing, and

the function g(β) of the tilt angle β is adjustable.

9. A method for operating an optical inspection tool, the method comprising:

observing, with an observation apparatus, an operation region at the first observation plane, the first observation plane having a first observation plane axis and a second observation plane axis and defining a first viewing axis which is aligned perpendicular to the first observation plane axis and the second observation plane axis;

observing, with the optical inspection tool, the operation region at the second observation plane, the second observation plane having a third observation plane axis and a fourth observation plane axis and defining a second viewing axis which is aligned perpendicular to the third observation plane axis and the fourth observation plane axis;

representing at least one of a first image recorded by a first image recording device and a second image recorded by a second image recording device;

determining, with a tracking system, an orientation of the optical inspection tool relative to the observation apparatus; and

transforming the second image based on the orientation of the optical inspection tool relative to the observation apparatus, wherein transforming the second image comprises:

generating a projected observation plane by projecting the second observation plane onto the first observation plane, wherein the projected observation plane has a projected third observation plane axis and a projected fourth observation plane axis and defining a projected second viewing axis which is aligned perpendicular to the projected third observation plane axis and the projected fourth observation plane axis, and wherein the projected third observation plane axis, the projected fourth observation plane axis, and projected second viewing axis define a projected coordinate system;

determining a rotation angle which indicates a rotation of the projected coordinate system about the projected second viewing axis such that the projected third observation plane axis is aligned parallel to and equally oriented with the first observation plane axis, and the projected fourth observation plane axis is aligned parallel to and equally oriented with the second observation plane axis; and

rotating the second image about the rotation angle about the projected second viewing axis.

10. The method of claim 9 , wherein:

transforming the second image comprises defining a reference plane,

the reference plane is a plane having a first reference plane axis and a second reference plane axis, and

the first and second reference plane axes are aligned perpendicular to a gravity.

11. The method of claim 10 , wherein transforming the second image comprises:

generating a projected first observation plane by projecting the first observation plane onto the reference plane, wherein the projected first observation plane has a projected first observation plane axis and a projected second observation plane axis and defines a projected first viewing axis; and

when the second viewing axis is aligned perpendicular to the reference plane, determining a first rotation angle α 1 and rotating the second image about the first rotation angle α 1 such that a rotated third observation plane axis of the rotated second observation plane is aligned parallel to and equally oriented with the projected first observation plane axis, and the projected fourth observation plane axis is aligned parallel to and equally oriented with the second observation plane axis.

12. The method of claim 10 , wherein transforming the second image comprises:

defining a horizontal plane and a vertical plane, wherein the horizontal plane is aligned parallel to and equally oriented with the reference plane and the vertical plane is aligned perpendicular to the reference plane;

generating a projected horizontal observation plane by projecting the second observation plane onto the horizontal plane and generating a projected vertical observation plane by projecting the second observation plane onto the vertical observation plane;

determining a first rotation angle α 1 such that a rotated projected third observation plane axis of the projected horizontal observation plane is aligned parallel to and equally oriented with the projected first observation plane axis;

determining a second rotation angle α 2 such that a rotated projected fourth observation plane axis of the projected vertical observation plane is directed away from and perpendicular to the reference plane in a direction opposite to the gravity;

determining a tilt angle β relative to the reference plane;

determining a third rotation angle α 3 based on the first rotation angle α 1 , the second rotation angle α 2 , and the tilt angle β; and

rotating the second image about the third rotation angle α 3 about the projected second viewing axis.

13. The method of claim 12 , wherein the third rotation angle α 3 is determined in accordance with

α 3 =g (β)·α 1 +(1 −g (β))·α 2 , and

wherein α 1 is the first rotation angle, α 2 is the second rotation angle, and g(β) is a function of the tilt angle β.

14. The method of claim 13 , wherein:

a value of a function g(β) of the tilt angle β is 0 when the tilt angle β is 0°,

the value of the function g(β) of the tilt angle β is 1 when the tilt angle β is 90°,

the function g(β) of the tilt angle β is monotonically increasing, and

the function g(β) of the tilt angle β is adjustable.

15. The method of claim 9 , wherein:

the observation apparatus is a microscope,

the optical inspection tool is an endoscope,

the target detection device is a camera, and

the at least one target is a marker.

16. The method of claim 9 , wherein transforming the second image comprises:

defining a vertical axis of the second image; and

reflecting the second image on the vertical axis.

17. The method of claim 9 , further comprising:

transforming the second image by training the visualization system, and

to transform the second image by training:

repeatedly manually rotating the second image about the projected second viewing axis corresponding to a rotation angle depending on the orientation of the optical inspection tool relative to the observation apparatus;

storing values of the rotation angle in a training database each time the second image is rotated about the rotation angle;

comparing the values previously stored in the training database with the values subsequently stored in the training database; and

automatically rotating the second image about the rotation angle based on the training of the visualization system.

Assignments (2)

Priority Claims (2)

Continuity (2)

References Cited (17)