Systems and methods for generating three-dimensional measurements using endoscopic video data
Presented herein are systems and methods for performing three-dimensional measurements of a surgical space using two-dimensional endoscopic images. According to an aspect, video data taken from an endoscopic imaging device can be used to generate a three-dimensional model of the surgical space represented by the video data. In one or more examples, two-dimensional images from the video data can be used to generate a three-dimensional model of the surgical space. In one or more examples, the one or more two-dimensional images of the surgical space can include a fiducial marker as part of the image. Using both the depth information and a size reference provided by the fiducial marker, the systems and methods herein can generate a three-dimensional model of the surgical space. The generated three-dimensional model can then be used to perform a variety of three-dimensional measurements in a surgical cavity in an accurate and efficient manner.
1. A method for measuring three-dimensional distances using endoscopic images, the method comprising:
receiving video data captured from an endoscopic imaging device configured to image an internal area of a patient;
capturing one or more two-dimensional images of the internal area from the received video data, wherein an image of the one or more two-dimensional images comprises a visualization of an object, and wherein the object comprises one or more fiducial markers configured to indicate a pre-determined dimension of the object or the fiducial marker, or both;
generating a three-dimensional model of the internal area based on the captured one or more two-dimensional images;
determining a size of a three-dimensional area of the three-dimensional model based on the one or more fiducial markers of the object configured to indicate the pre-determined dimension;
identifying a first point on the one or more two-dimensional images;
identifying a second point on the one or more two-dimensional images; and
determining a three-dimensional measurement between the first point and the second point in the area based on the determined size of the three-dimensional area and the generated three-dimensional model of the area.
2. The method of claim 1 , wherein capturing the one or more two-dimensional images comprises:
in response to a user requesting a measurement, receiving a first indication to mark a start time point in the received video data;
receiving a second indication to mark a stop time point in the received video data;
extracting a first two-dimensional image corresponding to the first indication; and
extracting a second two-dimensional image corresponding to the second indication.
3. The method of claim 1 , wherein capturing the one or more two-dimensional images comprises:
receiving a first indication at a graphical user interface on a computing device display;
receiving a second indication at the graphical user interface on the computing device display;
extracting a first two-dimensional image corresponding to the first indication; and
extracting a second two-dimensional image corresponding to the second indication.
4. The method of claim 1 , wherein capturing the one or more two-dimensional images comprises:
capturing a first image with the endoscopic imaging device located at a first position in the area; and
capturing a second image with the imaging device located at a second position in the area.
5. The method of claim 4 , wherein generating a three-dimensional model of the area based on the captured one or more two-dimensional images comprises applying a structure-from-motion procedure to the first and second images to generate the three-dimensional model of the area.
6. The method of claim 1 , wherein generating a three-dimensional model of the area based on the captured one or more two-dimensional images comprises:
capturing a first two-dimensional image of the area;
receiving data from a time-of-flight sensor, wherein the received data corresponds to the imaged area; and
generating the three-dimensional model of the area based on the captured first two-dimensional image of the area and the received data from the time-of-flight sensor.
7. The method of claim 1 , wherein the endoscopic imaging device comprises a stereo camera, and wherein capturing the one or more two-dimensional images comprises capturing a first two-dimensional image and a second two-dimensional image using the stereo camera of the endoscopic imaging device.
8. The method of claim 7 , wherein generating a three-dimensional model of the area based on the captured one or more two-dimensional images comprises applying a structure-from-motion procedure to the first and second two-dimensional images to generate the three-dimensional model of the area.
9. The method of claim 1 , wherein identifying the first point on the one or more two-dimensional images comprises segmenting the object from the first image of the one or more two-dimensional images, and wherein identifying the second point on the one or more two-dimensional images comprises segmenting the object from the second image of the one or more two-dimensional images.
10. The method of claim 9 , wherein segmenting the object from the first and second images of the one or more two-dimensional images comprises applying one or more machine learning models to the first and second images, wherein the one or more machine learning models are created using a supervised training process that comprises using one or more annotated images to train the machine learning model.
11. The method of claim 1 , wherein determining a three-dimensional measurement between the first point and the second point in the area based on the determined size of the area and the generated three-dimensional model of the area comprises determining a distance along a surface between the first position and the second position.
12. The method of claim 1 , wherein determining a three-dimensional measurement between the first point and the second point in the area based on the determined size of the area and the generated three-dimensional model of the area comprises determining the point-to-point distance between the first position and the second position.
13. A system for measuring three-dimensional distances using endoscopic images, the system comprising:
a memory;
one or more processors;
wherein the memory stores one or more programs that when executed by the one or more processors, cause the one or more processors to:
receive video data captured from an endoscopic imaging device configured to image an internal area of a patient;
capture one or more two-dimensional images of the internal area from the received video data, wherein an image of the one or more two-dimensional images comprises a visualization of an object, and wherein the object comprises one or more fiducial markers configured to indicate a pre-determined dimension of the object or the fiducial marker, or both;
generate a three-dimensional model of the internal area based on the captured one or more two-dimensional images;
determine a size of a three-dimensional area of the three-dimensional model based on the one or more fiducial markers of the object configured to indicate the pre-determined dimension;
identify a first point on the one or more two-dimensional images;
identify a second point on the one or more two-dimensional images; and
determine a three-dimensional measurement between the first point and the second point in the area based on the determined size of the three-dimensional area and the generated three-dimensional model of the area.
14. The system of claim 13 , wherein capturing the one or more two-dimensional images comprises:
in response to a user requesting a measurement, receiving a first indication to mark a start time point in the received video data;
receiving a second indication to mark a stop time point in the received video data;
extracting a first two-dimensional image corresponding to the first indication; and
extracting a second two-dimensional image corresponding to the second indication.
15. The system of claim 13 , wherein capturing the one or more two-dimensional images comprises:
receiving a first indication at a graphical user interface on a computing device display;
receiving a second indication at the graphical user interface on the computing device display;
extracting a first two-dimensional image corresponding to the first indication; and
extracting a second two-dimensional image corresponding to the second indication.
16. The system of claim 13 , wherein capturing the one or more two-dimensional images comprises:
capturing a first image with the endoscopic imaging device located at a first position in the area; and
capturing a second image with the imaging device located at a second position in the area.
17. The system of claim 16 , wherein generating a three-dimensional model of the area based on the captured one or more two-dimensional images comprises applying a structure-from-motion procedure to the first and second images to generate the three-dimensional model of the area.
18. The system of claim 13 , wherein generating a three-dimensional model of the area based on the captured one or more two-dimensional images comprises:
capturing a first two-dimensional image of the area;
receiving data from a time-of-flight sensor, wherein the received data corresponds to the imaged area; and
generating the three-dimensional model of the area based on the captured first two-dimensional image of the area and the received data from the time-of-flight sensor.
19. The system of claim 13 , wherein the endoscopic imaging device comprises a stereo camera, and wherein capturing the one or more two-dimensional images comprises capturing a first two-dimensional image and a second two-dimensional image using the stereo camera of the endoscopic imaging device.
20. The system of claim 19 , wherein generating a three-dimensional model of the area based on the captured one or more two-dimensional images comprises applying a structure-from-motion procedure to the first and second two-dimensional images to generate the three-dimensional model of the area.
21. The system of claim 13 , wherein identifying the first point on the one or more two-dimensional images comprises segmenting the object from the first image of the one or more two-dimensional images, and wherein identifying the second point on the one or more two-dimensional images comprises segmenting the object from the second image of the one or more two-dimensional images.
22. The system of claim 21 , wherein segmenting the object from the first and second images of the one or more two-dimensional images comprises applying one or more machine learning models to the first and second images, wherein the one or more machine learning models are created using a supervised training process that comprises using one or more annotated images to train the machine learning model.
23. The system of claim 13 , wherein determining a three-dimensional measurement between the first point and the second point in the area based on the determined size of the area and the generated three-dimensional model of the area comprises determining a distance along a surface between the first position and the second position.
24. The system of claim 13 , wherein determining a three-dimensional measurement between the first point and the second point in the area based on the determined size of the area and the generated three-dimensional model of the area comprises determining the point-to-point distance between the first position and the second position.
25. A non-transitory computer readable storage medium storing one or more programs for measuring three-dimensional distances, for execution by one or more processors of an electronic device that when executed by the device, cause the device to:
receive video data captured from an endoscopic imaging device configured to image an internal area of a patient;
capture one or more two-dimensional images of the internal area from the received video data, wherein an image of the one or more two-dimensional images comprises a visualization of an object, and wherein the object comprises one or more fiducial markers configured to indicate a pre-determined dimension of the object or the fiducial marker, or both;
generate a three-dimensional model of the internal area based on the captured one or more two-dimensional images;
determine a size of a three-dimensional area of the three-dimensional model based on the one or more fiducial markers of the object configured to indicate the pre-determined dimension;
identify a first point on the one or more two-dimensional images;
identify a second point on the one or more two-dimensional images; and
determine a three-dimensional measurement between the first point and the second point in the area based on the determined size of the three-dimensional area and the generated three-dimensional model of the area.
26. The non-transitory computer readable storage medium of claim 25 , wherein capturing the one or more two-dimensional images comprises:
in response to a user requesting a measurement, receiving a first indication to mark a start time point in the received video data;
receiving a second indication to mark a stop time point in the received video data;
extracting a first two-dimensional image corresponding to the first indication; and
extracting a second two-dimensional image corresponding to the second indication.
27. The non-transitory computer readable storage medium of claim 25 , wherein capturing the one or more two-dimensional images comprises:
receiving a first indication at a graphical user interface on a computing device display;
receiving a second indication at the graphical user interface on the computing device display;
extracting a first two-dimensional image corresponding to the first indication; and
extracting a second two-dimensional image corresponding to the second indication.
28. The non-transitory computer readable storage medium of claim 25 , wherein capturing the one or more two-dimensional images comprises:
capturing a first image with the endoscopic imaging device located at a first position in the area; and
capturing a second image with the imaging device located at a second position in the area.
29. The non-transitory computer readable storage medium of claim 28 , wherein generating a three-dimensional model of the area based on the captured one or more two-dimensional images comprises applying a structure-from-motion procedure to the first and second images to generate the three-dimensional model of the area.
30. The non-transitory computer readable storage medium of claim 25 , wherein generating a three-dimensional model of the area based on the captured one or more two-dimensional images comprises:
capturing a first two-dimensional image of the area;
receiving data from a time-of-flight sensor, wherein the received data corresponds to the imaged area; and
generating the three-dimensional model of the area based on the captured first two-dimensional image of the area and the received data from the time-of-flight sensor.
31. The non-transitory computer readable storage medium of claim 25 , wherein the endoscopic imaging device comprises a stereo camera, and wherein capturing the one or more two-dimensional images comprises capturing a first two-dimensional image and a second two-dimensional image using the stereo camera of the endoscopic imaging device.
32. The non-transitory computer readable storage medium of claim 31 , wherein generating a three-dimensional model of the area based on the captured one or more two-dimensional images comprises applying a structure-from-motion procedure to the first and second two-dimensional images to generate the three-dimensional model of the area.
33. The non-transitory computer readable storage medium of claim 25 , wherein identifying the first point on the one or more two-dimensional images comprises segmenting the object from the first image of the one or more two-dimensional images, and wherein identifying the second point on the one or more two-dimensional images comprises segmenting the object from the second image of the one or more two-dimensional images.
34. The non-transitory computer readable storage medium of claim 33 , wherein segmenting the object from the first and second images of the one or more two-dimensional images comprises applying one or more machine learning models to the first and second images, wherein the one or more machine learning models are created using a supervised training process that comprises using one or more annotated images to train the machine learning model.
35. The non-transitory computer readable storage medium of claim 25 , wherein determining a three-dimensional measurement between the first point and the second point in the area based on the determined size of the area and the generated three-dimensional model of the area comprises determining a distance along a surface between the first position and the second position.
36. The non-transitory computer readable storage medium of claim 25 , wherein determining a three-dimensional measurement between the first point and the second point in the area based on the determined size of the area and the generated three-dimensional model of the area comprises determining the point-to-point distance between the first position and the second position.