Image registration and augmented reality system and method augmented reality thereof
Disclosed is an image registration and an augmented reality system and an augmented reality method thereof which is suitable for solving the problem of spatial localization of the temporomandibular joint (TMJ) in arthroscopic surgery. The system comprises a three-dimensional scanning model building device, a stereoscopic image photographing device, a projection device and an arithmetic unit. The three-dimensional scanning model was constructed by preoperative or intraoperative imaging of the patient, and the surface three-dimensional model constructed by the stereoscopic image photographing device was spatially aligned to remove the surface (skin layer) of the three-dimensional image to display the TMJ image. Through the calibration of the stereoscopic image photographing device and the projection device, accurate, three-dimensional TMJ image location information is projected onto the patient's body to achieve the purpose.
1. An image registration and augmented reality system, comprising:
a three-dimensional scanning model building device, scanning a cranium of a body to generate a plurality of anatomic images, and building a three-dimensional scanning model according to the plurality of anatomic images;
a stereoscopic image photographing device, capturing a plurality of skin surface images of the cranium and ears of the body, and building a surface three-dimensional model according to the plurality of skin surface images;
an arithmetic unit correcting and aligning the three-dimensional scanning model and the surface three-dimensional model according to a plurality of structural reference points on surfaces of a temporal portion and the ears of the body, and combining the aligned three-dimensional scanning model with the aligned surface three-dimensional model to visualize a three-dimensional image;
a projection device, calibrated and aligned according to a location of the stereoscopic image photographing device so as to project the three-dimensional image onto a corresponding skin surface of the body; and
a control device, electrically connected with the arithmetic unit and the projection device;
wherein after the three-dimensional image is projected onto the skin surface of the body, the control device controls the arithmetic unit to remove the surface three-dimensional model of the three-dimensional image, and display at least an internal tissue image of the body of the three-dimensional image through the projection device.
2. The image registration and augmented reality system of claim 1 , wherein the projection device is spatially and geometrically calibrated, so as to be located in a same coordinate system as that of the stereoscopic image photographing device.
3. The image registration and augmented reality system of claim 1 , wherein the internal tissue image of the body includes a cranium image and a mandibular image, such that medical staff is able to locate an entry point of surgical instruments according to the internal tissue image of the body.
4. The image registration and augmented reality system of claim 1 , wherein the three-dimensional scanning model building device includes anatomic image scanning devices for computerized tomography or nuclear magnetic resonance imaging.
5. An image registration and augmented reality system, comprising:
a three-dimensional scanning model building device, scanning a part of a body to generate a plurality of anatomic images, and building a three-dimensional scanning model according to the plurality of anatomic images;
a stereoscopic image photographing device, capturing a plurality of skin surface images of the part of the body, and building a surface three-dimensional model according to the plurality of skin surface images;
an arithmetic unit correcting and aligning the three-dimensional scanning model and the surface three-dimensional model according to a plurality of structural reference points on surfaces and around an organ having a three-dimensional structure of the part of the body, and combining the aligned three-dimensional scanning model with the aligned surface three-dimensional model to visualize a three-dimensional image;
a projection device, corrected according to a location of the stereoscopic image photographing device so as to project the three-dimensional image onto a corresponding skin surface of the body; and
a control device electrically connected with the arithmetic unit and the projection device;
wherein after the three-dimensional image is projected onto the skin surface of the body, the control device controls the arithmetic unit to remove the surface three-dimensional model of the three-dimensional image, and display at least an internal tissue image of the body of the three-dimensional image through the projection device.
6. The image registration and augmented reality system of claim 5 , wherein the three-dimensional image includes images of each tissue layer of the body.
7. The image registration and augmented reality system of claim 5 , wherein the projection device includes a monitor, a projector, or a head mounted display for displaying the three-dimensional scanning model, the surface three-dimensional model, the three-dimensional image and the body.
8. The image registration and augmented reality system of claim 5 wherein the surfaces of the part of the body include a temporal portion of the body.
9. The image registration and augmented reality system of claim 5 , wherein the organ having the three-dimensional structure of the part of the body includes ears of the body.
10. A method of image registration and augmented reality, comprising:
A. Scanning a cranium of a body by utilizing a three-dimensional scanning model building device to generate a plurality of anatomic images, and then building a three-dimensional scanning model according to the plurality of anatomic images;
B. Capturing a plurality of skin surface images of the cranium and ears of the body by utilizing a stereoscopic image photographing device and then building a surface three-dimensional model according to the plurality of skin surface images;
C. Correcting and aligning the three-dimensional scanning model and the surface three-dimensional model according to a plurality of structural reference points on surfaces of a temporal portion and the ears of the body through an arithmetic unit and then combining the aligned three-dimensional scanning model with the aligned surface three-dimensional model to visualize a three-dimensional image;
D. Projecting the three-dimensional image onto a corresponding skin surface of the body; and
E. Removing the surface three-dimensional model of the three-dimensional image, and displaying at least an internal tissue image of the body of the three-dimensional image.