System and method for medical instrument navigation and targeting
Certain aspects relate to systems and techniques for medical instrument navigation and targeting. In one aspect, a system includes a medical instrument having an elongate body and at least one sensor, a display, a processor, and a memory storing a model of a mapped portion of a luminal network and a position of a target with respect to the model. The processor may be configured to: determine, based on data from the at least one sensor, a position and orientation of a distal end of the medical instrument with respect to the model, and cause, on at least a portion of the display, a rendering of the model, the position of the target, and the position and orientation of the distal end of the medical instrument. The rendering may be based on a viewpoint directed at the target and different from a viewpoint of the medical instrument.
1. A system, comprising:
a medical instrument having an elongate body and at least one sensor;
a display;
a processor; and
a memory storing computer-executable instructions to cause the processor to:
access a three-dimensional model of a luminal network, the model based on a preoperative mapping of a portion of the luminal network,
determine a position of a target with respect to the model,
determine, based on data from the at least one sensor, a position and orientation of a distal end of the medical instrument with respect to the model,
cause, on at least a portion of the display, a rendering of the model, the position of the target with respect to the model, and the position and orientation of the distal end of the medical instrument with respect to the model, wherein the rendering comprises a virtual viewpoint directed at the target, the rendering being simulated from a virtual viewpoint perspective at a virtual location with respect to the model, and
cause within the portion of the display a rendering of a modified view of the model, the modified view being rendered from a modified virtual viewpoint in which the virtual viewpoint perspective is rotated around the target with the target as a center of the rotation.
2. The system of claim 1 , further comprising a user input element, wherein the memory further stores computer-executable instructions to cause the processor to:
receive, via the user input element, a first type of user interaction with the user input element,
generate a first type of rendering instruction based on the first type of user interaction with the user input element, and
in response to generating the first type of rendering instruction, cause within the portion of the display the rendering of the modified view of the model, the position of the target, and the position and orientation of the distal end of the medical instrument.
3. The system of claim 2 , wherein:
the first type of rendering instruction comprises a rotation instruction to rotate the viewpoint directed at the target, and
the modified view being rendered from the modified virtual viewpoint in which the virtual viewpoint perspective is rotated around the target with the target as the center of the rotation is performed in response to the rotation instruction.
4. The system of claim 2 , wherein the memory further stores computer-executable instructions to cause the processor to:
receive, via the user input element, a second type of user interaction with the user input element,
generate a second type of rendering instruction based on the second type of user interaction with the user input element, the second type of rendering instruction comprising an elevation instruction to alter an angle between a reference plane centered at the target and the viewpoint directed at the target, and
in response to generating the second type of rendering instruction, cause within the portion of the display a rendering of the of the model having an altered the angle between the reference plane and the viewpoint directed at the target.
5. The system of claim 4 , wherein:
the user input element comprises a button,
the first user interaction type comprises an actuation of the button for a greater than a threshold period of time, and
the second user interaction type comprises an actuation of the button for less than the threshold period of time.
6. The system of claim 2 , wherein:
the first type of rendering instruction comprising a toggle perspective instruction to toggle the viewpoint of the rendering from the viewpoint different from the viewpoint of the medical instrument to the viewpoint of the medical instrument, and
the modified view comprises a rendering of the model from the viewpoint of the medical instrument.
7. The system of claim 1 , wherein the memory further stores computer-executable instructions to cause the processor to:
cause, based on the position and orientation of the distal end of the medical instrument, a rendering of a graphical indicator extending from the distal end of the medical instrument in an insertion direction of the medical instrument,
determine, based on the data from the at least one sensor, a change in at least one of the position and orientation of the distal end of the medical instrument,
determine, in response to determining the change in at least one of the position and orientation of the distal end of the medical instrument, that the graphical indicator intersects the target, and
cause a rendering of a change to the graphical indicator in response to determining that the line intersects the target.
8. The system of claim 1 , wherein the memory further stores computer-executable instructions to cause the processor to:
determine, based on the data from the at least one sensor, that the distal end of the medical instrument is within a threshold distance from the target; and
cause a rendering of a graphical indicator in response to determining that the distal end of the medical instrument is within a threshold distance from the target, the graphical indicator extending from the distal end of the medical instrument in an insertion direction of the medical instrument.
9. The system of claim 8 , wherein the graphical indicator is selected from: a line, a cylinder, or a cone.
10. The system of claim 8 , wherein the graphical indicator is a cone, the aperture of the cone is based on an estimated error range of the orientation of the distal end of the medical instrument.
11. The system of claim 8 , wherein the memory further stores computer-executable instructions to cause the processor to:
determine, in response to determining the change in at least one of the position and orientation of the distal end of the medical instrument, that the graphical indicator intersects a center of the target, and
cause a rendering of a change in the color of the graphical indicator in response to determining that the graphical indicator intersects the center of the target.
12. The system of claim 1 , wherein:
the model comprises a skeleton comprising a plurality of segments, wherein each of the segments is defined with respect to a center line of a corresponding lumen of the mapped portion of the luminal network,
the memory further stores computer-executable instructions to cause the processor to:
identify a segment in which the distal end of the medical instrument is located,
determine a depth of the distal end of the medical instrument along the identified segment, and
determine the position of the distal end of the medical instrument based on the identified segment and the depth.
13. The system of claim 12 , wherein the memory further stores computer-executable instructions to cause the processor to:
determine a distance between the distal end of the medical instrument and the target based on the determined position of the distal end of the medical instrument and the position of the target.
14. The system of claim 12 , wherein the memory further stores computer-executable instructions to cause the processor to:
determine that the position of the distal end of the medical instrument is within a volume defined by the model, and
restrict the determined position of the distal end of the medical instrument to be located along the skeleton.
15. The system of claim 12 , wherein:
the at least one sensor comprises one of: an electromagnetic (EM) sensor or a shape sensing fiber, and
the memory further stores computer-executable instructions to cause the processor to configured generate the position data based only on data received from one of the EM sensor of the shape sensing fiber.
16. A non-transitory computer readable storage medium having stored thereon instructions that, when executed, cause at least one computing device to:
access a three-dimensional model of a luminal network, the model based on a preoperative mapping of a portion of the luminal network;
determine a position of a target with respect to the model;
determine, based on data from at least one sensor of a medical instrument, a position and orientation of a distal end of the medical instrument with respect to the model;
cause, on at least a portion of a display, a rendering of the model, a position of the target with respect to the model, and the position and orientation of the distal end of the medical instrument with respect to the model, wherein the rendering comprises a virtual viewpoint directed at the target, the rendering being simulated from a virtual viewpoint perspective at a virtual location with respect to the model; and
cause within the portion of the display a rendering of a modified view of the model, the modified view being rendered from a modified virtual viewpoint in which the virtual viewpoint perspective is rotated around the target with the target as a center of the rotation.
17. The non-transitory computer readable storage medium of claim 16 , further having stored thereon instructions that, when executed, cause at least one computing device to:
receive, via the user input element, a first type of user interaction with the user input element;
generate a first type of rendering instruction based on the first type of user interaction with the user input element; and
in response to generating the first type of rendering instruction, cause within the portion of the display the rendering of the modified view of the model, the position of the target, and the position and orientation of the distal end of the medical instrument.
18. The non-transitory computer readable storage medium of claim 17 , wherein:
the first type of rendering instruction comprises a rotation instruction to rotate the viewpoint directed at the target, and
the modified view being rendered from the modified virtual viewpoint in which the virtual viewpoint perspective is rotated around the target with the target as the center of the rotation is performed in response to the rotation instruction.
19. The non-transitory computer readable storage medium of claim 17 , further having stored thereon instructions that, when executed, cause at least one computing device to:
receive, via the user input element, a second type of user interaction with the user input element;
generate a second type of rendering instruction based on the second type of user interaction with the user input element, the second type of rendering instruction comprising an elevation instruction to alter an angle between a reference plane centered at the target and the viewpoint directed at the target; and
in response to generating the second type of rendering instruction, cause within the portion of the display a rendering of the of the model having an altered the angle between the reference plane and the viewpoint directed at the target.
20. The non-transitory computer readable storage medium of claim 19 , wherein:
the user input element comprises a button,
the first user interaction type comprises an actuation of the button for a greater than a threshold period of time, and
the second user interaction type comprises an actuation of the button for less than the threshold period of time.
21. The non-transitory computer readable storage medium of claim 17 , wherein:
the first type of rendering instruction comprising a toggle perspective instruction to toggle the viewpoint of the rendering from the viewpoint different from the viewpoint of the medical instrument to the viewpoint of the medical instrument, and
the modified view comprises a rendering of the of the model from the viewpoint of the medical instrument.
22. The non-transitory computer readable storage medium of claim 16 , further having stored thereon instructions that, when executed, cause at least one computing device to:
cause, based on the position and orientation of the distal end of the medical instrument, a rendering of a graphical indicator extending from the distal end of the medical instrument in an insertion direction of the medical instrument;
determine, based on the data from the at least one sensor, a change in at least one of the position and orientation of the distal end of the medical instrument;
determine, in response to determining the change in at least one of the position and orientation of the distal end of the medical instrument, that the graphical indicator intersects the target; and
cause a rendering of a change to the graphical indicator in response to determining that the line intersects the target.
23. The non-transitory computer readable storage medium of claim 16 , further having stored thereon instructions that, when executed, cause at least one computing device to:
determine, based on the data from the at least one sensor, that the distal end of the medical instrument is within a threshold distance from the target; and
cause a rendering of a graphical indicator in response to determining that the distal end of the medical instrument is within a threshold distance from the target, the graphical indicator extending from the distal end of the medical instrument in an insertion direction of the medical instrument.
24. The non-transitory computer readable storage medium of claim 23 , further having stored thereon instructions that, when executed, cause at least one computing device to:
determine, in response to determining the change in at least one of the position and orientation of the distal end of the medical instrument, that the graphical indicator intersects a center of the target, and
cause a rendering of a change in the color of the graphical indicator in response to determining that the graphical indicator intersects the center of the target.
25. The non-transitory computer readable storage medium of claim 24 , wherein:
the model comprises a skeleton comprising a plurality of segments, wherein each of the segments is defined with respect to a center line of a corresponding lumen of the mapped portion of the luminal network,
the non-transitory computer readable storage medium further has stored thereon instructions that, when executed, cause at least one computing device to:
identify a segment in which the distal end of the medical instrument is located;
determine a depth of the distal end of the medical instrument along the identified segment; and
determine the position of the distal end of the medical instrument based on the identified segment and the depth.
26. The non-transitory computer readable storage medium of claim 25 , further having stored thereon instructions that, when executed, cause at least one computing device to:
determine a distance between the distal end of the medical instrument and the target based on the determined position of the distal end of the medical instrument and the position of the target.
27. The non-transitory computer readable storage medium of claim 25 , further having stored thereon instructions that, when executed, cause at least one computing device to:
cause, on at least another portion of the display, a rendering of the position of the distal end of the medical instrument based on the determined position of the distal end of the medical instrument.
28. The non-transitory computer readable storage medium of claim 25 , further having stored thereon instructions that, when executed, cause at least one computing device to:
determine that the position of the distal end of the medical instrument is within a volume defined by the model, and
restrict the determined position of the distal end of the medical instrument to be located along the skeleton.
29. The non-transitory computer readable storage medium of claim 25 , wherein:
the at least one sensor comprises one of: an electromagnetic (EM) sensor or a shape sensing fiber, and
the memory further stores computer-executable instructions to cause the processor to configured generate the position data based only on data received from one of the EM sensor of the shape sensing fiber.
30. The system of claim 2 , wherein the memory further stores computer-executable instructions to cause the processor to:
receive, via the user input element, a second type of user interaction with the user input element,
generate a second type of rendering instruction based on the second type of user interaction with the user input element, the second type of rendering instruction comprising toggle instruction to toggle between: (i) the virtual viewpoint positioned at a virtual location different from the position of the distal end of the medical instrument and (ii) a virtual viewpoint positioned at a virtual location that is the same as the position of the distal end of the medical instrument, and
in response to generating the second type of rendering instruction, cause within the portion of the display a rendering of the of the model from the virtual viewpoint positioned at the virtual location that is the same as the position of the distal end of the medical instrument.