METHOD FOR REGISTRATION OF A VIRTUAL IMAGE IN AN AUGMENTED REALITY SYSTEM

1 . A method for registration of a virtual image in an augmented reality, AR, system comprising an AR device, the method comprising:

acquiring a first image by a camera of the AR system, wherein the first image depicts a first projection of a first laser beam onto a surface of a subject, the first projection being in the shape of a first line representing the intersection of a first plane with the surface of the subject, the first plane arranged such that it comprises or has a fixed spatial relation with respect to a target point;

determining, based at least on a shape and/or position of the first line in the first image, a first component of a position of the target point in AR device coordinates; and

adjusting a registration of a virtual image, which is to be overlaid with the field of view of the AR device, based on the first component of the position of the target point in AR device coordinates.

2 . The method of claim 1 ,

wherein the first image further depicts a second projection of a second laser beam onto the surface of the subject, the second projection being in the shape of a second line representing the intersection of a second plane with the surface of the subject;

wherein an intersection of the first plane and the second plane comprises the target point or has a fixed spatial relation with respect to the target point;

further comprising determining, based at least on a shape and/or position of the second line in the first image, and optionally also based on the shape and/or position of the first line in the first image, a second component of the position of the target point in AR device coordinates; and

wherein the adjusting of the registration of the virtual image is performed based on the first component and the second component of the target point in AR device coordinates.

3 . The method of claim 1 , further comprising determining a third component of the position of the target point in the AR device coordinates,

wherein adjusting the registration of the virtual image is performed based on the first component, the second component, and the third component of the position of the target point in AR device coordinates,

the method further comprising:

obtaining an image that depicts a third projection of a third laser beam onto the surface of the subject, wherein the third projection is in the shape of a third line representing the intersection of a third plane with the surface of the subject, the third plane being arranged such that the intersection of the first plane, the third plane, and optionally the second plane, comprises the target point or has a fixed spatial relation with respect to the target point; and

determining, based at least on a shape and/or position of the third line in the image, and optionally also based on the shape and/or position of the first line and/or the second line, the third component of the position of the target point in AR device coordinates,

wherein the image is the first image or a second image acquired by the camera of the AR system and depicting at least one of the first projection and the second projection.

4 . The method of claim 1 , wherein determining the first component and/or determining the second component and/or determining the third component of the position of the target point in AR device coordinates is further based on a shape of the surface of the subject obtained from 3D image data of the surface.

5 . The method of claim 4 , wherein determining the first component and/or the second component and/or the third component of the position of the target point in AR device coordinates comprises:

determining, based on the shape of the surface of the subject, a plurality of candidate lines and/or a plurality candidate crossing positions of crossings of candidate lines in the first image and/or in the second image;

matching the first line and/or the second line and/or the third line and/or one or more crossing positions of crossings of the first line and/or the second line and/or the third line in the first image and/or in the second image with the plurality of candidate lines and/or the plurality of candidate crossing positions, so as to obtain one or more matching candidate lines and/or one or more matching candidate crossing positions; and

determining the first component and/or the second component and/or the third component of the position of the target point in AR device coordinates based on the one or more matching candidate lines and/or the one or more matching candidate crossing positions.

6 . The method of claim 1 , comprising

reconstructing the first plane from the shape of the first line in the first image, wherein determining the first component of the position of the target point in AR device coordinates is based at least on the reconstructed first plane, and/or

reconstructing the second plane from the shape of the second line in the first image, wherein determining the second component of the position of the target point in AR device coordinates is based at least on the reconstructed second plane, and/or

reconstructing the third plane from the shape of the third line in the image, in particular the first image or the second image, wherein determining the third component of the position of the target point in AR device coordinates is based at least on the reconstructed third plane.

7 . The method of claim 1 ,

wherein the reconstructing of the first plane and/or the second plane and/or the third plane is further based on:

a/the shape of the surface of the subject obtained from 3D image data of the surface, and/or

a position of the first line and/or the second line and/or the third line in the first image and/or the second image, and/or

a relative position of the first line with respect to the second line and/or the third line in the first image, and/or

a relative position of the first line and/or the second line with respect to the third line in the second image, and/or

one or more crossing positions of the first line and/or the second line and/or the third line in the first image and/or in the second image.

8 . The method of claim 1 , wherein determining the first component and/or second component and/or third component of the position of the target point in AR device coordinates is further based on:

additional measurement data, from the camera and/or from an additional camera, and/or image data from a CT or MRI imaging device, and/or gyroscopic tracking data obtained by the augmented reality system and/or inside-out tracking data; and/or

information on the shape of the subject and/or arrangement of the subject relative to the camera and/or the target point; and/or

position information on a marker located on the subject and depicted in the first image and/or second image; and/or

information on an arrangement of the first plane and/or the second plane and/or the third plane, in particular an arrangement relative to each other and/or the subject and/or the camera and/or the target point; and/or

previously determined values of the first component and/or second component and/or third component of the position of the target point in AR device coordinates and/or information on at least one of a previously determined reconstructed first plane and/or second plane and/or third plane; and/or

information obtained by a calibration procedure.

9 . The method of claim 1 , comprising

continuously or at predetermined times and/or time intervals repeating the steps of acquiring the first image, and optionally obtaining the second image;

determining at least one of the first component, the second component, and the third component of the position of the target point in AR device coordinates;

determining whether at least one of the first component, second component, and third component of the position of the target point in AR device coordinates differs from a previously determined and stored first component, second component, and third component, respectively, of the position of the target point in AR device coordinates; and

in response to determining that this is the case, updating at least one of the previously determined and stored first component, second component, and third component.

10 . The method of claim 1 , further comprising

calculating, based on measurements performed in advance of placing the subject and based on the target position and orientation of the subject with respect to the target point or based on measurements determining the surface of the subject and the placement of the subject relative to the laser beams when the subject is placed, the expected projections of each of the laser beams, and

the AR device displaying, as part of the virtual image, the expected projections of the laser beams, to allow for a user of the AR device to determine whether the expected projections and the actual projections match, and/or

determining, from an image obtained by the camera, that the projections of each of the laser beams deviate from the expected projection, and, in case that the deviation exceeds a predetermined threshold, outputting an alert and/or taking automatic correction steps.

11 . The method of claim 1 , further comprising,

determining, by a calibration procedure, an initial first component, an initial second component, and an initial third component of the position of the target point in AR device coordinates, storing the initial first component, the initial second component, and the initial third component of the position of the target point in AR device coordinates as previously determined and stored first component, second component, and third component, respectively,

optionally wherein the calibration procedure comprises using, in addition to images obtained by the camera of the AR system, measurement data from other devices, the measurement data being data obtained prior to and/or during the calibration procedure.

12 . An augmented reality, AR, system comprising:

an augmented reality, AR, device configured to overlay a virtual image with a field of view of the AR device; and

a camera being part of the AR device and/or connected to the AR device,

wherein the AR system is configured with stored instructions and at least one processor to execute instructions causing the at least one processor to:

acquire a first image by a camera of the AR system, wherein the first image depicts a first projection of a first laser beam onto a surface of a subject, the first projection being in the shape of a first line representing the intersection of a first plane with the surface of the subject, the first plane arranged such that it comprises or has a fixed spatial relation with respect to a target point;

determine, based at least on a shape and/or position of the first line in the first image, a first component of a position of the target point in AR device coordinates; and

adjust a registration of a virtual image, which is to be overlaid with the field of view of the AR device, based on the first component of the position of the target point in AR device coordinates.

13 . A medical system comprising:

an AR system having an augmented reality, AR, device configured to overlay a virtual image with a field of view of the AR device; and

a camera being part of the AR device and/or connected to the AR device,

wherein the AR system is configured with stored instructions and at least one processor to execute the stored instructions and causing the at least one processor to:

acquire a first image by a camera of the AR system, wherein the first image depicts a first projection of a first laser beam onto a surface of a subject, the first projection being in the shape of a first line representing the intersection of a first plane with the surface of the subject, the first plane arranged such that it comprises or has a fixed spatial relation with respect to a target point;

determine, based at least on a shape and/or position of the first line in the first image a first component of a position of the target point in AR device coordinates; and

adjust a registration of a virtual image, which is to be overlaid with the field of view of the AR device, based on the first component of the position of the target point in AR device coordinates;

and further including a first laser configured to provide the first laser beam in the first plane, and optionally a second laser configured to provide the second laser beam in the second plane and/or a third laser to provide the third laser beam in the third plane.

14 . The system of claim 13 , further comprising a treatment machine comprising a treatment beam source and a patient support unit, and the lasers and the treatment machine are configured and/or arranged such that the target point, of the first, second, and third planes, coincides with the isocenter of the treatment beam source.

15 . (canceled)

16 . A non-transitory computer readable medium comprising instructions which, when executed by at least one processor, cause the at least one processor to:

acquire a first image by a camera of an AR system with an augmented reality device, wherein the first image depicts a first projection of a first laser beam onto a surface of a subject, the first projection being in the shape of a first line representing the intersection of a first plane with the surface of the subject, the first plane arranged such that it comprises or has a fixed spatial relation with respect to a target point;

determine, based at least on a shape and/or position of the first line in the first image, a first component of a position of the target point in AR device coordinates; and

adjust a registration of a virtual image, which is to be overlaid with the field of view of the AR device, based on the first component of the position of the target point in AR device coordinates.