Technique for optimizing rendering parameters of overlays of medical images
An overlay of an aligned medical image obtained from a medical scanner is rendered with a reference image. The reference image has a reference structure of a body. An intermediate representation of the reference structure is determined. The medical image has structures corresponding to the reference structures, and an intermediate representation of the structures in the medical image is determined. A rendering parameter is optimized by comparing the intermediate representations of the medical image and of the reference image. The medical image is aligned and overlayed with the reference image based on the optimized rendering parameter, and the aligned and overlayed image is rendered.
1 . A computer-implemented method for rendering of an overlay of an aligned medical image from data obtained from a medical scanner with a reference image, the method comprising:
receiving the reference image, wherein the reference image comprises at least one reference structure of a body;
determining an intermediate representation of the at least one reference structure in the received reference image;
receiving at least one medical image from a medical scanner, wherein the at least one medical image comprises at least partially structures of a patient's body corresponding to the at least one reference structure of the received reference image;
determining an intermediate representation of at least part of the structures in the received at least one medical image;
receiving at least one rendering parameter for rendering the received at least one medical image;
optimizing the received at least one rendering parameter for aligning and overlaying the received at least one medical image with the received reference image, wherein the optimizing comprises comparing the intermediate representation of the received at least one medical image with the intermediate representation of the received reference image;
aligning and overlaying the received at least one medical image with the received reference image based on the optimized at least one rendering parameter; and
rendering the aligned and overlayed at least one medical image with the at least one reference image.
2 . The method of claim 1 , wherein the medical scanner comprises:
a surgical camera;
an ultrasound scanner;
a positron emission tomography scanner;
an x-ray imaging scanner;
a computed tomography scanner; or
a magnetic resonance tomography scanner.
3 . The method of claim 1 , wherein the act of rendering comprises rendering the aligned and overlayed at least one medical image with the reference image using a device, the device comprising:
a screen;
a stereoscopic display;
a virtual reality display; or
an augmented reality headset.
4 . The method of claim 1 , wherein the reference image comprises:
a cryosection image data set;
a synthetically generated image; or
an image received from a medical scanner.
5 . The method of claim 1 , wherein the intermediate representation of the at least one reference structure and/or the intermediate representation of the at least partial structures of the at least one medical image is:
a segmentation;
a mesh;
an isosurface; or
a point cloud.
6 . The method of claim 1 , wherein the at least one rendering parameter comprises:
a view position;
a field of view;
a view orientation;
a depth of field; and/or
a lighting.
7 . The method of claim 1 , wherein at least one of the received reference image and the received at least one medical image comprises a series of non-static images.
8 . The method of claim 7 , wherein the series of non-static medical images is received from the medical scanner in real-time during an interventional procedure.
9 . The method of claim 1 , wherein comparing the intermediate representation of the received at least one medical image with the intermediate representation the received reference image comprises generating a function of a difference of at least one property of the intermediate representation of the received at least one medical image and of the intermediate representation of the received reference image, and wherein optimizing the at least one rendering parameter comprises determining at least one partial derivative of the generated function with respect to the at least one rendering parameter.
10 . The method of claim 9 , wherein the at least one property of the intermediate representation of the received at least one medical image and of the intermediate representation of the received reference image comprises an image color per discrete image location.
11 . The method of claim 10 , wherein the discrete image locations comprise pixels and/or voxels.
12 . The method of claim 9 , wherein optimizing the at least one rendering parameter comprises minimizing the generated function of the difference of the at least one property of the intermediate representation of the received at least one medical image and of the intermediate representation of the received reference image.
13 . The method of claim 1 , wherein receiving the reference image, determining the associated intermediate representation, receiving the at least one medical image, determining the associated intermediate representation, and receiving and optimizing the at least one rendering parameter are performed by a graphics processing unit.
14 . A system for rendering of an overlay of a medical image from data obtained from a medical scanner with a reference image, the system comprising:
a first interface configured to receive the reference image, wherein the reference image comprises at least one reference structure of a body;
a first processing unit configured to determine an intermediate representation of the at least one reference structure in the received reference image;
a second interface configured to receive the medical image from the medical scanner, wherein the medical image comprises, at least partially, structures of a patient's body corresponding to the at least one reference structure of the received reference image;
a second processing unit configured to determine an intermediate representation of at least part of the structures in the received medical image;
a third interface configured to receive at least one rendering parameter for rendering the received medical image;
a third processing unit configured to optimize the received at least one rendering parameter for aligning and overlaying the received medical image with the received reference image, wherein the optimization comprises comparison of the intermediate representation of the received medical image with the intermediate representation of the received reference image;
a fourth processing unit configured to align and overlay the received medical image with the received reference image based on the optimized at least one rendering parameter; and
a fourth interface configured to output the aligned and overlayed medical image with the reference image for rendering.
15 . The system of claim 14 , further comprising:
the medical scanner, the medical scanner comprising a surgical camera, an ultrasound scanner, a positron emission tomography scanner, an x-ray imaging scanner, a computed tomography scanner, or a magnetic resonance tomography scanner;
a renderer configured to render the aligned and overlayed medial image with the reference image; and
a display configured to display the rendered medial image with the reference image, the display comprising a screen, a stereoscopic display, a virtual reality display, or an augmented reality headset;
wherein the reference image comprises a cryosection image data set, a synthetically generated image, or an image received from the medical scanner or another medical scanner;
wherein the intermediate representation of the at least one reference structure and/or the intermediate representation of the structures of the at least one medical image is a segmentation, a mesh, an isosurface, or a point cloud; and
wherein the at least one rendering parameter comprises a view position, a field of view, a view orientation, a depth of field, and/or a lighting.
16 . The system of claim 14 , wherein at least one of the received reference image and the received medical image comprises a series of non-static images received from the medical scanner in real-time during an interventional procedure.
17 . The system of claim 14 , wherein the third processing unit is configured to compare the intermediate representation of the received medical image with the intermediate representation the received reference image by generation a function of a difference of at least one property of the intermediate representation of the received medical image and of the intermediate representation of the received reference image, and wherein the optimization of the at least one rendering parameter comprises determination of at least one partial derivative of the generated function with respect to the at least one rendering parameter.
18 . A system for rendering of an overlay of a medical image from data obtained from a medical scanner with a reference image, the system comprising:
a reference database for providing at least one reference image;
a medical scanner for providing at least one medical image;
a computer, wherein a first interface is configured to receive the at least one reference image from the reference database, and a second interface is configured to receive the at least one medical image from the medical scanner;
a processor is configured to optimize a rendering parameter for aligning and overlaying the received at least one medical image with the received at least one reference image, wherein the optimization comprises comparison of an intermediate representation of structure in the received medical image with an intermediate representation of the structure in the received reference image, the processor configured to align and overlay the received at least one medical image with the received at least one reference image based on the optimized rendering parameter; and
a renderer configured to render the aligned and overlaid at least one medical image with the at least one reference image.