Multi-modal image alignment method and system
A multi-modal image alignment method includes obtaining first points corresponding to a center vertex of a calibration object and second point groups corresponding to side vertices of the calibration object from two-dimensional images, obtaining third points corresponding to the center vertex from three-dimensional images, performing first optimizing computation using a first coordinate system associated with the two-dimensional images, the first points and the third points to obtain a first transformation matrix, processing on the three-dimensional images using the first transformation matrix to generate firstly-transformed images respectively, performing second optimizing computation using the firstly-transformed images, the first points and the second point groups to obtain a second transformation matrix, and transforming an image to be processed from a second coordinate system associated with the three-dimensional images to the first coordinate system using the first transformation matrix and the second transformation matrix.
1. A multi-modal image alignment system, comprising:
a calibration object, comprising:
a main body having a center vertex and a plurality of side vertices; and
a plurality of indicators respectively disposed on the center vertex and the plurality of side vertices;
a 2D image capturing device having a first 3D coordinate system, and configured to generate a plurality of 2D images associated with the calibration object;
a 3D image capturing device having a second 3D coordinate system, and configured to generate a plurality of 3D images associated with the calibration object; and
a processing device connected to the 2D image capturing device and the 3D image capturing device, and configured to obtain a coordinate transformation matrix according to the plurality of 2D images and the plurality of 3D images, and transform an image to be processed to the second 3D coordinate system or the first 3D coordinate system by using the coordinate transformation matrix.
2. The multi-modal image alignment system according to claim 1 , further comprising:
another 2D image capturing device connected to the processing device, having a third 3D coordinate system, and configured to generate a plurality of second 2D images associated with the calibration object;
wherein the processing device is further configured to obtain another coordinate transformation matrix according to the plurality of second 2D images and the plurality of 3D images, and perform transformation between the first 3D coordinate system and the third 3D coordinate system using the two coordinate transformation matrices.
3. The multi-modal image alignment system according to claim 1 , wherein the plurality of indicators have different colors or temperatures.
4. The multi-modal image alignment system according to claim 1 , wherein to obtain the coordinate transformation matrix, the processing device is further configured to:
obtain a plurality of first points corresponding to the center vertex and a plurality of second point groups corresponding to the plurality of side vertices from the plurality of 2D images;
obtain a plurality of third points corresponding to the center vertex from the plurality of 3D images;
perform first optimizing computation based on the first 3D coordinate system by using an initial first transformation matrix, the plurality of first points and the plurality of third points to obtain an optimized first transformation matrix;
process on the plurality of 3D images by using the optimized first transformation matrix to generate a plurality of firstly-transformed images respectively; and
perform second optimizing computation based on the first 3D coordinate system by using the plurality of firstly-transformed images, the plurality of first points, the plurality of second point groups and a predetermined specification parameter set of the calibration object to obtain an optimized second transformation matrix;
wherein the coordinate transformation matrix comprises the optimized first transformation matrix and the optimized second transformation matrix.
5. The multi-modal image alignment system according to claim 4 , wherein the plurality of first points and the plurality of third points respectively correspond to a plurality of calibration positions where the calibration object is placed, and to perform the first optimizing computation, the processing device is further configured to:
perform a distance calculation on the first point and the third point corresponding to each of the plurality of calibration positions to obtain a plurality of calculation results respectively corresponding to the plurality of calibration positions, wherein the distance calculation comprises:
obtaining a line connecting an origin of the first 3D coordinate system and the first point corresponding to each of the plurality of calibration positions;
transforming the third point corresponding to each of the plurality of calibration positions by using the initial first transformation matrix; and
calculating a distance between the third point transformed by the initial first transformation matrix and the line; and
iteratively adjusting the initial first transformation matrix based on a convergence function, and regarding the iteratively-adjusted initial first transformation matrix as the optimized first transformation matrix, wherein the convergence function is to minimize a sum of the plurality of calculation results.
6. The multi-modal image alignment system according to claim 4 , wherein to perform the second optimizing computation, the processing device is further configured to:
process on the plurality of firstly-transformed images by using an initial second transformation matrix to respectively generate a plurality of secondly-transformed images;
obtain a plurality of fourth point groups respectively from the plurality of secondly-transformed images according to the plurality of first points and the plurality of second point groups;
obtain a plurality of estimated specification parameter sets according to the plurality of fourth point groups respectively; and
iteratively adjust the initial second transformation matrix based on a convergence function, and regard the iteratively-adjusted initial second transformation matrix as the optimized second transformation matrix, wherein the convergence function is to minimize a difference level between the plurality of estimated specification parameter sets and the predetermined specification parameter set of the calibration object.
7. The multi-modal image alignment system according to claim 6 , wherein the plurality of secondly-transformed images, the plurality of first points, the plurality of second point groups respectively correspond to a plurality of calibration positions where the calibration object is placed, and to obtain the plurality of fourth point groups, the processing device is further configured to:
perform steps on a first point, a second point group and a secondly-transformed image corresponding to each of the plurality of calibration positions, wherein the steps comprise:
projecting the first point onto the secondly-transformed image to obtain a fifth point corresponding to each of the plurality of calibration positions; and
projecting a plurality of points in the second point groups onto the secondly-transformed image to obtain a plurality of sixth points corresponding to each of the plurality of calibration positions;
wherein the fifth point and the plurality of sixth points form one of the plurality of fourth point groups.
8. The multi-modal image alignment system according to claim 7 , wherein to obtain the plurality of estimated specification parameter sets, the processing device is further configured to:
perform specification parameter estimation on each of the plurality of fourth point groups to obtain one of the plurality of estimated specification parameter sets, wherein the specification parameter estimation comprises:
obtaining a plurality of connecting lines between the fifth point and the plurality of sixth points; and
calculating a plurality of estimated lengths of the plurality of connecting lines and a plurality of estimated angles between the plurality of connecting lines;
wherein the predetermined specification parameter set comprises a plurality of predetermined side lengths and a plurality of predetermined angles of the calibration object, the difference level between the plurality of estimated specification parameter sets and the predetermined specification parameter set is a weighted sum of a first value and a second value, the first value is a sum of differences respectively between the plurality of estimated lengths and the plurality of predetermined side lengths, and the second value is a sum of differences respectively between the plurality of estimated angles and the plurality of predetermined angles.
9. The multi-modal image alignment system according to claim 4 , wherein to obtaining the plurality of third points, the processing device is further configured to:
regard each of the plurality of 3D images as a target image;
obtain three planes in the target image, wherein the planes are adjacent to each other and three normal vectors of the three planes are perpendicular to each other; and
obtain a point of intersection of the planes as one of the plurality of third points.
10. The multi-modal image alignment system according to claim 1 , wherein the first 3D coordinate system is a camera coordinate system of a 2D image capturing device, the processing device is further configured to transform the image to be processed from the first 3D coordinate system to an image plane coordinate system of the 2D image capturing device using a parameter matrix of the 2D image capturing device, and the parameter matrix comprises a local length parameter and a projection center parameter of the 2D image capturing device.