Method of robust surface and depth estimation
Systems and methods related to estimating the distance of a body structure from a medical device are disclosed. An example method includes illuminating the body structure with a light source of a medical device, capturing a first input image of the body structure with a digital camera positioned on the medical device, representing the first image with a first plurality of pixels, wherein the first plurality of pixels includes one or more pixels displaying a local intensity maxima, defining a first pixel group from the one or more pixels displaying a local intensity maxima, wherein the first pixel group corresponds to a plurality of surface points of the body structure and wherein the first pixel group further includes a first image intensity. The method further includes calculating a relative distance from the digital camera to a first surface point of the plurality of surface points.
1. A method of estimating the distance of a body structure from a medical device, the method comprising:
illuminating the body structure with a light source positioned on the distal end region of a medical device;
capturing a first input image of the body structure with a digital camera positioned on the distal end region of the medical device;
representing the first image with a first plurality of pixels, wherein the first plurality of pixels includes one or more pixels displaying a local intensity maxima;
defining a first pixel group from the one or more pixels displaying a local intensity maxima, wherein the first pixel group corresponds to a plurality of surface points of the body structure;
wherein the first pixel group further includes a first image intensity; and
calculating a relative distance r from the digital camera to a first surface point of the plurality of surface points by assuming and {right arrow over (S)} are parallel at the first surface point, that A is constant, and solving the relation for r:
I
∝
LA
v
⇀
·
S
→
r
4
wherein:
I is an image intensity;
L is an illumination intensity;
A is a surface albedo factor;
is a vector from the first surface point to the camera;
{right arrow over (S)} is a vector normal to the first surface point.
2. The method of claim 1 , further comprising calculating the relative distances from the digital camera to each of the surface points of the plurality of surface points.
3. The method of claim 2 , further comprising calculating a refined surface albedo factor using the relative distances from the digital camera to each of the surface points and a pixel intensity average over pixels with a similar hue.
4. The method of claim 3 , further comprising calculating refined relative distances from the digital camera to each of the surface points using the refined surface albedo factor.
5. The method of claim 4 , wherein the refined distances and the refined surface albedo factor are constrained by a map of three-dimensional position uncertainty vectors derived from one or more registered frames.
6. The method of claim 5 , further comprising calculating a second surface point position from a weighted average of the refined distance values, the refined surface albedo factor and a previous estimate of the second surface point position.
7. The method of claim 6 , wherein the weighted average of the refined distance values, the refined surface albedo factor and the previous estimate of the second surface point position is inversely proportional to the magnitude of an uncertainty vector.
8. The method of claim 7 , further comprising calculating an uncertainty vector , weighted averages and positions according to the relations:
w
p
=
U
m
U
m
+
❘
"\[LeftBracketingBar]"
U
⇀
p
❘
"\[RightBracketingBar]"
,
w
m
=
❘
"\[LeftBracketingBar]"
U
⇀
p
❘
"\[RightBracketingBar]"
U
m
+
❘
"\[LeftBracketingBar]"
U
⇀
p
❘
"\[RightBracketingBar]"
,
p
⇀
′
=
w
p
p
⇀
+
w
m
m
⇀
wherein:
w p =a weighted average of previous distance values;
w m =a weighted average of a new measurement node coordinate;
=a previous model node coordinate;
=an updated model node coordinate;
m =a new model node uncertainty measurement;
=a previous model node uncertainty vector; and
=a new measurement node coordinate.
9. The method of claim 8 , further comprising generating a surface texture map associated with the surface position map, wherein the surface texture map is generated using the relation:
❘
"\[LeftBracketingBar]"
U
⇀
p
❘
"\[RightBracketingBar]"
=
w
m
+
w
p
v
^
w
p
+
w
m
U
m
2
+
❘
"\[LeftBracketingBar]"
U
⇀
p
❘
"\[RightBracketingBar]"
2
.
10. A method of estimating the distance of a body structure from a medical device, the method comprising:
illuminating the body structure with a light source positioned on the distal end region of the medical device;
using a digital camera positioned on the distal end of region of the medical device to capture a first image of the body structure at a first time point, wherein the image capture device is positioned at a first position when it captures the first image at the first time point;
representing the first image with a first plurality of pixels, wherein the first plurality of pixels includes one or more pixels displaying a local intensity maxima;
defining a first pixel group from the one or more pixels displaying a local intensity maxima, wherein the first pixel group corresponds to a plurality of surface points of the body structure;
wherein the first pixel group further includes a first image intensity; and
calculating a relative distance r from the digital camera to a first surface point of the plurality of surface points by assuming and {right arrow over (S)} are parallel at the first surface point, that A is constant, and solving the relation for r:
I
∝
LA
v
⇀
·
S
→
r
4
wherein:
I is an image intensity;
L is an illumination intensity;
A is a surface albedo factor;
is a vector from the first surface point to the camera;
is a vector normal to the first surface point.
11. The method of claim 10 , further comprising calculating the relative distances from the digital camera to each of the surface points of the plurality of surface points.
12. The method of claim 11 , further comprising calculating a refined surface albedo factor using the relative distances from the digital camera to each of the surface points and a pixel intensity average over pixels with a similar hue.
13. The method of claim 12 , further comprising calculating refined relative distances from the digital camera to each of the surface points using the refined surface albedo factor.
14. The method of claim 13 , wherein the refined distances and the refined surface albedo factor are constrained by a map of three-dimensional position uncertainty vectors derived from one or more registered frames.
15. The method of claim 14 , further comprising calculating a second surface point position from a weighted average of the refined distance values, the refined surface albedo factor and a previous estimate of the second surface point position.
16. The method of claim 15 , wherein the weighted average of the refined distance values, the refined surface albedo factor and the previous estimate of the second surface point position is inversely proportional to the magnitude of an uncertainty vector.
17. The method of claim 16 , further comprising calculating an uncertainty vector p , weighted averages and positions according to the relations:
w
p
=
U
m
U
m
+
❘
"\[LeftBracketingBar]"
U
⇀
p
❘
"\[RightBracketingBar]"
,
w
m
=
❘
"\[LeftBracketingBar]"
U
⇀
p
❘
"\[RightBracketingBar]"
U
m
+
❘
"\[LeftBracketingBar]"
U
⇀
p
❘
"\[RightBracketingBar]"
,
p
⇀
′
=
w
p
p
⇀
+
w
m
m
⇀
wherein:
w p =a weighted average of previous distance values;
w m =a weighted average of a new measurement node coordinate;
=a previous model node coordinate;
=an updated model node coordinate;
U m =a new model node uncertainty measurement;
=a previous model node uncertainty vector; and
=a new measurement node coordinate.
18. The method of claim 16 , further comprising generating a surface texture map associated with the surface position map, wherein the surface texture map is generated using the relation:
❘
"\[LeftBracketingBar]"
U
⇀
p
❘
"\[RightBracketingBar]"
=
w
m
U
p
^
+
w
p
v
^
w
p
+
w
m
U
m
2
+
❘
"\[LeftBracketingBar]"
U
⇀
p
❘
"\[RightBracketingBar]"
2
.
19. A system for estimating the distance of a body structure from a medical device, comprising:
a processor; and
a non-transitory computer-readable storage medium comprising code configured to perform a method for estimating the distance of a body structure from a medical device, the method comprising:
illuminating the body structure with a light source positioned on the distal end region of a medical device;
capturing a first input image of the body structure with a digital camera positioned on the distal end region of the medical device;
representing the first image with a first plurality of pixels, wherein the first plurality of pixels includes one or more pixels displaying a local intensity maxima;
defining a first pixel group from the one or more pixels displaying a local intensity maxima, wherein the first pixel group corresponds to a plurality of surface points of the body structure;
wherein the first pixel group further includes a first image intensity; and
calculating a relative distance r from the digital camera to a first surface point of the plurality of surface points by assuming and {right arrow over (S)} are parallel at the first surface point, that A is constant, and solving the relation for r:
I
∝
LA
v
⇀
·
S
→
r
4
wherein:
I is an image intensity;
L is an illumination intensity;
A is a surface albedo factor;
is a vector from the first surface point to the camera;
{right arrow over (S)} is a vector normal to the first surface point.
20. The method of claim 19 , further comprising calculating the relative distances from the digital camera to each of the surface points of the plurality of surface points.