Method and system of depth determination in model fusion for laparoscopic surgical guidance
The present teaching relates to method, system, medium, and implementations for estimating 3D coordinate of a 3D virtual model. Two pairs of feature points are obtained. Each of the pairs includes a respective 2D feature point on an organ observed in a 2D image, acquired during a medical procedure, and a respective corresponding 3D feature point from a 3D virtual model, constructed for the organ prior to the procedure based on a plurality of images of the organ. The first and the second 3D feature points have different depths. A 3D coordinate of a 3D feature point is determined based on the pairs of feature points so that a projection of the 3D virtual model from the 3D coordinate substantially matches the organ observed in the 2D image.
1 . A method implemented on at least one processor, a memory, and a communication platform for estimating a three-dimensional (3D) coordinate of a 3D virtual model, comprising:
accessing a 3D virtual model constructed for an organ of a patient based on a plurality of images of the organ prior to a medical procedure;
selecting manually two pairs of corresponding features points, including:
a first pair of corresponding feature points comprising:
a first two-dimensional (2D) feature point on the organ as observed in a 2D image acquired during the medical procedure; and
a first corresponding 3D feature point on the 3D virtual model, the first corresponding 3D feature point and the first 2D feature point corresponding to a first common point; and
a second pair of corresponding feature points comprising:
a second 2D feature point on the organ as observed in the 2D image; and
a second corresponding 3D feature point on the 3D virtual model, the second corresponding 3D feature point and the second 2D feature point corresponding to a second common point;
projecting the 3D virtual model along a line of sight determined based on the first pair and the second pair of corresponding feature points, resulting in a projection of the 3D virtual model,
wherein the projection of the 3D virtual model from the 3D coordinate substantially matches the organ as observed in the 2D image;
performing camera calibration, resulting in camera parameters;
identifying a scaling factor along the line of sight corresponding to a depth of the 3D coordinate;
identifying an image coordinate of the first 2D feature point in the 2D image; and
determining, without an additional manually selected feature point, and based on (1) the camera parameters, (2) the scaling factor, and (3) the image coordinate, a 3D coordinate of the first corresponding 3D feature point on the 3D virtual model.
2 . The method of claim 1 , wherein
the 3D virtual model has six degrees of freedom with first three degrees of freedom related to the 3D coordinate having values along X, Y, Z axes of a camera coordinate system.
3 . The method of claim 2 , wherein the 3D virtual model has second three degrees of freedom related to rotations of the 3D virtual model with respect to each of the X, Y, and Z axes.
4 . The method of claim 3 , further comprising determining the second three degrees of freedom via:
adjusting rotation of the 3D virtual model with respect to at least one of the X, Y, and Z axes when projecting the 3D virtual model from the 3D coordinate on to the 2D image; and
selecting a best combination of rotations that yields a best match between the projection of the 3D virtual model and the patient's organ observed in the 2D image.
5 . The method of claim 4 , wherein the step of selecting is by a user via at least one of:
a visual inspection of the projection created using each combination, and
a quantitative measure automatically computed characterizing a degree of match between the projection of the 3D virtual model and the patient's organ observed in the 2D image.
6 . The method of claim 2 , wherein the line of sight is formed between the first 2D feature point and a focal point of a camera associated with a medical instrument deployed in the medical procedure.
7 . The method of claim 2 , wherein the 3D coordinate of the first 3D feature point is determined based on:
[
X
1
Y
1
Z
1
]
=
M
c
a
m
e
r
a
[
x
1
y
1
1
]
.
s
1
where (X_1, Y_1, Z_1) are three values along X, Y, Z axes, (x_1, y_1) is the image coordinate of the first 2D feature point in the 2D image, and s 1 is the scaling factor along the line of sight corresponding to the depth Z.
8 . The method of claim 5 , wherein a determination of the scaling factor corresponding to the depth Z comprises:
determining a minimum depth value for Z;
determining a distance d between the first and the second 3D feature points;
determining a maximum depth value for Z based on d to form a range of depth for Z;
projecting, at each of the depths within the range, the 3D virtual model on to the 2D image plane; and
selecting a depth value within the range that yields a best match between the projection of the 3D virtual model and the patient's organ observed in the 2D image.
9 . A machine readable and non-transitory medium having information recorded thereon for estimating a three-dimensional (3D) coordinate of a 3D virtual model, wherein the information, when read by the machine, causes the machine to perform the following steps:
accessing a 3D virtual model constructed for an organ of a patient based on a plurality of images of the organ prior to a medical procedure;
selecting manually two pairs of corresponding features points, including:
a first pair of corresponding feature points comprising:
a first two-dimensional (2D) feature point on the organ as observed in a 2D image acquired during the medical procedure; and
a first corresponding 3D feature point on the 3D virtual model, the first corresponding 3D feature point and the first 2D feature point corresponding to a first common point; and
a second pair of corresponding feature points comprising:
a second 2D feature point on the organ as observed in the 2D image; and
a second corresponding 3D feature point on the 3D virtual model,
the second corresponding 3D feature point and the second 2D feature point corresponding to a second common point;
projecting the 3D virtual model along a line of sight determined based on the first pair and the second pair of corresponding feature points, resulting in a projection of the 3D virtual model,
wherein the projection of the 3D virtual model from the 3D coordinate substantially matches the organ as observed in the 2D image;
performing camera calibration, resulting in camera parameters;
identifying a scaling factor along the line of sight corresponding to a depth of the 3D coordinate;
identifying an image coordinate of the first 2D feature point in the 2D image; and
determining, without an additional manually selected feature point, and based on (1) the camera parameters, (2) the scaling factor, and (3) the image coordinate, a 3D coordinate of the first corresponding 3D feature point on the 3D virtual model.
10 . The medium of claim 9 , wherein
the 3D virtual model has six degrees of freedom with first three degrees of freedom related to the 3D coordinate having values along X, Y, Z axes of a camera coordinate system.
11 . The medium of claim 10 , wherein the 3D virtual model has second three degrees of freedom related to rotations of the 3D virtual model with respect to each of the X, Y, and Z axes.
12 . The medium of claim 11 , wherein the information, when read by the machine, further causes the machine to perform the step of determining the second three degrees of freedom via:
adjusting rotation of the 3D virtual model with respect to at least one of the X, Y, and Z axes when projecting the 3D virtual model from the 3D coordinate on to the 2D image; and
selecting a best combination of rotations that yields a best match between the projection of the 3D virtual model and the patient's organ observed in the 2D image.
13 . The medium of claim 12 , wherein the step of selecting is by a user via at least one of:
a visual inspection of the projection created using each combination, and
a quantitative measure automatically computed characterizing a degree of match between the projection of the 3D virtual model and the patient's organ observed in the 2D image.
14 . The medium of claim 10 , wherein the line of sight is formed between the first 2D feature point and a focal point of a camera associated with a medical instrument deployed in the medical procedure.
15 . The medium of claim 10 , wherein the 3D coordinate of the first 3D feature point is determined haced on:
[
X
1
Y
1
Z
1
]
=
M
c
a
m
e
r
a
[
x
1
y
1
1
]
.
s
1
where (X_1, Y_1, Z_1) are three values along X, Y, Z axes, (x_1, y_1) is the image coordinate of the first 2D feature point in the 2D image, and s 1 is the scaling factor along the line of sight corresponding to the depth Z.
16 . The medium of claim 15 , wherein a determination of the scaling factor corresponding to the depth Z comprises:
determining a minimum depth value for Z;
determining a distance d between the first and the second 3D feature points;
determining a maximum depth value for Z based on d to form a range of depth for Z;
projecting, at each of the depths within the range, the 3D virtual model on to the 2D image plane; and
selecting a depth value within the range that yields a best match between the projection of the 3D virtual model and the patient's organ observed in the 2D image.
17 . A system for estimating a three-dimensional (3D) coordinate of a 3D virtual model, comprising:
an anatomical structure mesh generation unit configured for constructing a 3D virtual model for an organ of a patient based on a plurality of images of the organ prior to a medical procedure; and
a one-mark based model-to-video alignment unit configured for selecting manually two pairs of corresponding feature points, by:
selecting a first pair of corresponding feature points comprising:
a first two-dimensional (2D) feature point on the organ as observed in a 2D image acquired during the medical procedure; and
a first corresponding 3D feature point on the 3D virtual model, the first corresponding 3D feature point and the first 2D feature point corresponding to a first common point; and
a second pair of corresponding feature points comprising:
a second 2D feature point on the organ as observed in the 2D image; and
a second corresponding 3D feature point on the 3D virtual model, the second corresponding 3D feature point and the second 2D feature point corresponding to a second common point;
projecting the 3D virtual model along a line of sight determined based on the first pair and the second pair of corresponding feature points, resulting in a projection of the 3D virtual model,
wherein the projection of the 3D virtual model from the 3D coordinate substantially matches the organ as observed in the 2D image;
performing camera calibration, resulting in camera parameters;
identifying a scaling factor along the line of sight corresponding to a depth of the 3D coordinate;
identifying an image coordinate of the first 2D feature point in the 2D image; and
determining, without an additional manually selected feature point, and based on (1) the camera parameters, (2) the scaling factor, and (3) the image coordinate, a 3D coordinate of the first corresponding 3D feature point on the 3D virtual model.
18 . The system of claim 17 , wherein the first 3D feature point is on the line of sight formed between the first 2D feature point and a focal point of a camera associated with a medical instrument deployed in the medical procedure.
19 . The system of claim 18 , wherein the one-mark based model-to-video alignment unit is configured to determine the 3D coordinate of the first 3D feature point based on:
[
X
1
Y
1
Z
1
]
=
M
c
a
m
e
r
a
[
x
1
y
1
1
]
.
s
1
where (X_1, Y_1, Z_1) are three values along X, Y, Z axes, (x_1, y_1) is the image coordinate of the first 2D feature point in the 2D image, and s 1 is the scaling factor along the line of sight corresponding to the depth Z.
20 . The system of claim 19 , wherein the one-mark based model-to-video alignment unit is configured to determine the scaling factor corresponding to the depth Z by:
determining a minimum depth value for Z;
determining a distance d between the first and the second 3D feature points;
determining a maximum depth value for Z based on d to form a range of depth for Z;
projecting, at each of the depths within the range, the 3D virtual model on to the 2D image plane; and
selecting a depth value within the range that yields a best match between the projection of the 3D virtual model and the patient's organ observed in the 2D image.