Granted Patent B2

US 12,197,637 · App. 17/947,828 · Granted Jan 14, 2025

Inventors: Nils Frielinghaus (Heimstetten, DE); Christoffer Hamilton (Aschheim, DE)

Assignee: BRAINLAB AG

G06F3/012A61B90/361G02B27/017G06T19/006A61B2090/365G02B2027/0138G02B2027/014G02B2027/0178G02B2027/0187G06T2207/30004

Abstract

Claims (55)

1. A computer-implemented medical method for outputting, by an augmented reality device having a distance measurement unit and a display, augmentation information relating to positional tracking by a medical position tracking system, the method comprising executing, on at least one processor of at least one computer, the following steps:

a) a viewing direction of the augmented reality device is positionally registered with a tracking coordinate system associated with the position tracking system to generate an augmented reality device registration;

b) object position information is acquired which describes a position of a physical object in the tracking coordinate system, tracked by the position tracking system;

c) object presence information defining a presence of the physical object in a field of view of the augmented reality device is determined, using the distance measurement unit of the augmented reality device that determines a position of the augmented reality device relative to the physical object, and based on the augmented reality device registration and the object position information;

d) augmentation information is determined based on the object presence information and output by the display of the augmented reality device, wherein the augmentation information describes a tracking volume of the position tracking system by defining a visual rendering of the field of view of the position tracking system on the display of the augmented reality device, wherein a position of the field of view is transformed from the tracking coordinate system to a virtual coordinate space associated with the augmented reality device; and

wherein the position of the physical object in the tracking coordinate system is transformed from the tracking coordinate system to the virtual coordinate space associated with the augmented reality device, the virtual coordinate space usable for displaying the augmentation information on the display of the augmented reality device, wherein the augmentation information at least describing the tracking volume of the position tracking system is displayed as an overlay with respect to the physical object on the display of the augmented reality device and provides the visual rendering that indicates the position of the field of view.

2. The method according to claim 1 , wherein the augmentation information further describes at least one of the following:

a result of tracking the position of the physical object, whether a line of sight between the position tracking system and the physical object is obstructed;

at least one visual indication describing whether the physical object is meant to be tracked by the position tracking system, whether tracking the physical object is essential for correct functioning of an associated medical system;

at least one visual indication for identifying the physical object; and

at least one visual indication describing a warning to a user of the augmented reality device, a warning not to touch the physical object or that the physical object is close to being out of a line of sight between the position tracking system and the physical object.

3. The method according to claim 1 wherein

the position tracking system is an infra-red tracking system and an infrared-reflective marker device is attached to the physical object in a predetermined position relative to the physical object for allowing tracking of the physical object by the position tracking system, or

wherein the position tracking system is an electromagnetic tracking system and an electromagnetic marker device is attached to the physical object in a predetermined position relative to the physical object for allowing tracking of the physical object by the position tracking system.

4. The method according to claim 3 comprising the following steps:

marker device template data is acquired which describes a geometrical template of the marker device;

surface detection data which describes a surface geometry of the marker device is acquired based on a distance measurement by the distance measurement unit;

a relative position between the augmented reality device and the marker device is determined based on the surface detection data and the marker device template data; and

the viewing direction of the augmented reality device is positionally registered with the tracking coordinate system based on the determined relative position between the augmented reality device and the marker device.

5. The method according to claim 3 , comprising the following steps:

physical object template data is acquired which describes a geometrical template of the physical object;

surface detection data which describes a surface geometry of the physical object is acquired based on a distance measurement by the distance measurement unit;

a relative position between the augmented reality device and the physical object is determined based on the surface detection data and the physical object template data; and

the viewing direction of the augmented reality device is positionally registered with the tracking coordinate system based on the determined relative position between the augmented reality device and the physical object.

6. The method according to claim 1 wherein the augmented reality device comprises a visible image acquisition unit for acquiring an image in a visible wavelength range, and wherein the method comprises the following steps:

tracking system surface template data is acquired which describes a geometrical template of at least the surface of at least part of the position tracking system;

an image describing at least part of the tracking system is obtained from the visible image acquisition unit; and

the viewing direction of the augmented reality device is positionally registered with the tracking coordinate system based on the tracking system template data and the image describing the at least part of the position tracking system.

7. The method according to claim 1 wherein the method comprises the following steps:

tracking system surface template data is acquired which describes a geometrical template of at least part of the position tracking system;

tracking system surface detection data is acquired which describes a surface geometry of the at least part of the position tracking system based on a distance measurement by the distance measurement unit of the augmented reality device; and

the viewing direction of the augmented reality device is positionally registered with the tracking coordinate system based on the tracking system surface template data and the tracking system surface detection data.

8. A non-transitory computer-readable medium storing statements and instructions for execution by a computer to perform a method comprising steps of:

a) a viewing direction of an augmented reality device is positionally registered with a tracking coordinate system associated with a position tracking system to generate an augmented reality device registration;

b) object position information is acquired which describes a position of a physical object in the tracking coordinate system, tracked by the position tracking system;

c) object presence information defining a presence of the physical object in a field of view of the augmented reality device is determined, using a distance measurement unit of the augmented reality device that determines a position of the augmented reality device to the physical object, and based on the augmented reality device registration and the object position information;

d) augmentation information is determined based on the object presence information and output by a display of the augmented reality device, wherein the augmentation information describes a tracking volume of the position tracking system by defining a visual rendering of the field of view of the position tracking system on the display of the augmented reality device, wherein a position of the field of view is transformed from the tracking coordinate system to a virtual coordinate space associated with the augmented reality device; and

wherein the position of the physical object in the tracking coordinate system is transformed from the tracking coordinate system to the virtual coordinate space associated with the augmented reality device, the virtual coordinate space usable for displaying the augmentation information on the display of the augmented reality device, wherein the augmentation information at least describing the tracking volume of the position tracking system is displayed as an overlay with respect to the physical object on the display of the augmented reality device and provides the visual rendering that indicates the position of the field of view.

9. A medical system for determining augmentation information relating to positional tracking by a medical position tracking system, the medical system comprising:

at least one computer;

the medical position tracking system for tracking a position of a physical object; and

an augmented reality device having a distance measurement unit and an output unit for outputting augmentation information,

wherein the at least one computer includes at least one processor and memory storing instructions that, when executed, cause the at least one processor to perform following steps:

a) a viewing direction of the augmented reality device is positionally registered with a tracking coordinate system associated with the position tracking system to generate an augmented reality device registration;

b) object position information is acquired which describes a position of a physical object in the tracking coordinate system, tracked by the position tracking system;

c) object presence information defining a presence of the physical object in a field of view of the augmented reality device is determined, using the distance measurement unit of the augmented reality device that determines a position of the augmented reality device relative to the physical object, and based on the augmented reality device registration and the object position information;

d) augmentation information is determined based on the object presence information and output by a display of the augmented reality device, wherein the augmentation information describes a tracking volume of the position tracking system by defining a visual rendering of the field of view of the position tracking system on the display of the augmented reality device, wherein a position of the field of view is transformed from the tracking coordinate system to a virtual coordinate space associated with the augmented reality device, on the display of the augmented reality device; and

wherein the position of the physical object in the tracking coordinate system is transformed from the tracking coordinate system to the virtual coordinate space associated with the augmented reality device, the virtual coordinate space usable for displaying the augmentation information on the display of the augmented reality device, wherein the augmentation information at least describing the tracking volume of the position tracking system is displayed as an overlay with respect to the physical object on the display of the augmented reality device and provides the visual rendering that indicates the position of the field of view.

10. A computer implemented method, comprising:

positionally registering a field of view of an augmented reality device with a position tracking system that tracks objects in a tracking coordinate system, the tracking coordinate system being associated with the position tracking system;

acquiring object position information of a physical object in the tracking coordinate system and tracked by the position tracking system,

determining object presence information representing the position of the physical object in the field of view of the augmented reality device using a position determination unit associated with the augmented reality device; and

determining augmentation information using at least the object presence information, the augmentation information describing a tracking volume of the position tracking system to be displayed via the augmented reality device by defining a visual rendering of at least a part of the field of view of the position tracking system on a display of the augmented reality device, wherein a position of the tracking volume is transformed from the tracking coordinate system associated with the position tracking system to a virtual coordinate space of the augmented reality device,

the augmentation information describing at least a part of the tracking volume of the position tracking system as an overlay with respect to the physical object and

providing a visual rendering indicating the position of the field of view.

Assignments (2)

Continuity (2)

References Cited (12)