Devices, systems, methods, and storage media for controlling a bendable medical device
Some devices, systems, and methods obtain a fluoroscopic image and corresponding C-arm positional information of the fluoroscopic image, wherein the fluoroscopic image depicts a distal end of a bendable medical device, and wherein the C-arm positional information indicates a rotation angle of a C-arm of a C-arm scanner; obtain an orientation of the distal end in an image plane of the fluoroscopic image, wherein the image plane is defined in a detector reference frame; obtain an instructed direction; determine, in the image plane, a movement direction of the distal end that corresponds to the instructed direction; map the movement direction from the image plane to a distal-end reference frame of the distal end, wherein the mapping is based on an orientation of the distal end in an intermediate reference frame and on the C-arm positional information; and control the distal end to move according to the mapped movement direction.
1. A system comprising:
one or more computer-readable media; and
one or more processors in communication with the one or more computer-readable media, wherein the one or more processors and the one or more computer-readable media are configured to cooperate to perform operations that comprise:
obtaining a fluoroscopic image that was generated by a C-arm scanner and corresponding C-arm positional information for the fluoroscopic image, wherein the fluoroscopic image depicts a distal end of a bendable medical device, and wherein the C-arm positional information indicates a rotation angle of a C-arm of the C-arm scanner;
determining an orientation of the distal end in an image plane of the fluoroscopic image, wherein the image plane is defined in a detector reference frame of the C-arm scanner;
obtaining an instructed direction;
determining, in the image plane, a movement direction of the distal end that corresponds to the instructed direction, wherein the determining is based on the orientation of the distal end in the image plane, and wherein the movement direction lies in the imaging plane;
mapping the movement direction from the image plane, which is defined in the detector reference frame, to a distal-end reference frame of the distal end, thereby generating a mapped movement direction, wherein the detector reference frame is different from the distal-end reference frame, wherein the mapping is based on an orientation of the distal end in an intermediate reference frame and on the C-arm positional information, and wherein the mapped movement direction, which is defined in the distal-end reference frame, lies in the image plane; and
controlling the distal end to move according to the mapped movement direction.
2. The system of claim 1 , wherein the intermediate reference frame is a reference frame of a proximal end of the bendable medical device.
3. The system of claim 1 , wherein the intermediate reference frame is a reference frame of a position-tracking system.
4. The system of claim 1 , wherein the orientation of the distal end in the image plane is determined based on the orientation of the distal end in the intermediate reference frame and on the C-arm positional information.
5. The system of claim 1 , wherein the orientation of the distal end in the image plane is determined based on a computer-vision analysis of the fluoroscopic image.
6. The system of claim 1 , wherein the mapped movement direction lies entirely in the image plane.
7. The system of claim 1 , wherein the instructed direction is right or, alternatively, left.
8. The system of claim 1 , further comprising an input device,
wherein the input device includes a first button and a second button,
wherein the input device is configured to send a signal that indicates “right” as the instructed direction in response to activation of the first button, and
wherein the input device is configured to send a signal that indicates “left” as the instructed direction in response to activation of the second button.
9. The system of claim 1 , wherein the distal end of the bendable medical device includes three bending segments, which are bendable independent from each other;
wherein the one or more processors and the one or more computer-readable media are further configured to cooperate to identify a bending segment, of the three bending segments, that corresponds to the instructed direction; and
wherein controlling the distal end to move according to the mapped movement direction includes controlling the bending segment that corresponds to the instructed direction to bend in the mapped movement direction.
10. The system of claim 9 , wherein controlling the distal end to move according to the mapped movement direction includes controlling the two bending segments, of the three bending segments, that do not correspond to the instructed direction to retain their current poses.
11. The system of claim 9 , further comprising an input device,
wherein the three bending segments include a first bending segment, a second bending segment, and a third bending segment,
wherein the input device includes a first button, a second button, a third button, a fourth button, a fifth button, and a sixth button,
wherein the input device is configured to send a signal that corresponds to the first bending segment and that indicates “right” as the instructed direction in response to activation of the first button,
wherein the input device is configured to send a signal that corresponds to the first bending segment and that indicates “left” as the instructed direction in response to activation of the second button,
wherein the input device is configured to send a signal that corresponds to the second bending segment and that indicates “right” as the instructed direction in response to activation of the third button,
wherein the input device is configured to send a signal that corresponds to the second bending segment and that indicates “left” as the instructed direction in response to activation of the fourth button,
wherein the input device is configured to send a signal that corresponds to the third bending segment and that indicates “right” as the instructed direction in response to activation of the fifth button, and
wherein the input device is configured to send a signal that corresponds to the third bending segment and that indicates “left” as the instructed direction in response to activation of the sixth button.
12. The system of claim 1 , further comprising the bendable medical device,
wherein the distal end of the bendable medical device includes three bending segments, which are bendable independent from each other,
wherein the three bending segments include a first bending segment, a second bending segment, and a third bending segment, each of which includes a respective distal end,
wherein the distal end of the first bending segment includes a first radiopaque marker,
wherein the distal end of the second bending segment includes a second radiopaque marker,
wherein the distal end of the third bending segment includes a third radiopaque marker, and
wherein the first radiopaque marker, the second radiopaque marker, and the third radiopaque marker are composed of different radiopaque materials that cause the first radiopaque marker, the second radiopaque marker, and the third radiopaque marker to have different appearances in the fluoroscopic image.
13. The system of claim 1 , further comprising the bendable medical device,
wherein the distal end of the bendable medical device includes three bending segments, which are bendable independent from each other,
wherein the three bending segments include a first bending segment, a second bending segment, and a third bending segment, each of which includes a respective distal end,
wherein the distal end of the first bending segment includes a first radiopaque marker that includes one or more respective openings,
wherein the distal end of the second bending segment includes a second radiopaque marker that includes one or more respective openings,
wherein the distal end of the third bending segment includes a third radiopaque marker that includes one or more respective openings, and
wherein the one or more respective openings of the first radiopaque marker, the second radiopaque marker, and the third radiopaque marker have different shapes.
14. A method for controlling a bendable medical device, the method comprising:
obtaining a fluoroscopic image that was generated by a C-arm scanner and corresponding C-arm positional information for the fluoroscopic image, wherein the fluoroscopic image depicts a distal end of a bendable medical device, and wherein the C-arm positional information indicates a rotation angle of a C-arm of the C-arm scanner;
obtaining an orientation of the distal end in an image plane of the fluoroscopic image, wherein the image plane is defined in a detector reference frame of the C-arm scanner;
obtaining an instructed direction;
determining, in the image plane, a movement direction of the distal end that corresponds to the instructed direction, wherein the determining is based on the orientation of the distal end in the image plane, and wherein the movement direction lies in the imaging plane;
mapping the movement direction from the image plane, which is defined in the detector reference frame, to a distal-end reference frame of the distal end, thereby generating a mapped movement direction, wherein the detector reference frame is different from the distal-end reference frame, wherein the mapping is based on an orientation of the distal end in an intermediate reference frame and on the C-arm positional information, and wherein the mapped movement direction, which is defined in the distal-end reference frame, lies in the image plane; and
controlling the distal end to move according to the mapped movement direction.
15. The method of claim 14 , wherein the intermediate reference frame is a reference frame of a proximal end of the bendable medical device or is a reference frame of a position-tracking system.
16. The method of claim 14 , wherein the mapped movement direction lies entirely in the image plane.
17. The method of claim 14 , wherein the mapping of the movement direction includes generating a rotation matrix that describes an orientation of the distal-end reference frame relative to the intermediate reference frame or generating a rotation matrix that describes an orientation of the detector reference frame relative to the intermediate reference frame.
18. The method of claim 17 , wherein the mapping of the movement direction includes both generating the rotation matrix that describes the orientation of the distal-end reference frame relative to the intermediate reference frame and generating the rotation matrix that describes the orientation of the detector reference frame relative to the intermediate reference frame.
19. The method of claim 18 , wherein the mapping of the movement direction includes generating a rotation matrix that describes an orientation of the distal-end reference frame relative to the detector reference frame based on the rotation matrix that describes the orientation of the distal-end reference frame relative to the intermediate reference frame and on the rotation matrix that describes the orientation of the detector reference frame relative to the intermediate reference frame.
20. A method for controlling a bendable medical device, the method comprising:
obtaining a fluoroscopic image that was generated by a C-arm scanner and corresponding C-arm positional information for the fluoroscopic image, wherein the fluoroscopic image depicts a distal end of a bendable medical device, and wherein the C-arm positional information indicates a rotation angle of a C-arm of the C-arm scanner;
obtaining distal-end positional information, which indicates an orientation of the distal end;
generating a reference-frame mapping between a detector reference frame of the C-arm scanner and a distal-end reference frame of the distal end, wherein the reference-frame mapping is generated based on an orientation of the distal end in an intermediate reference frame and on the C-arm positional information, wherein the detector reference frame is different from the distal-end reference frame, and wherein the intermediate reference frame is different from both the detector reference frame and the distal-end reference frame;
generating an inter-view-relationship indicator, which indicates a relationship between the detector reference frame and the distal-end reference frame, based on the reference-frame mapping; and
displaying the fluoroscopic image and the inter-view-relationship indicator.
21. The method of claim 20 , further comprising:
obtaining an endoscopic image that was captured from an imaging device at the distal end of the bendable medical device; and
displaying the endoscopic image with the fluoroscopic image and the inter-view-relationship indicator.
22. The method of claim 21 , wherein the inter-view-relationship indicator is superimposed on the endoscopic image.
23. The method of claim 20 , wherein generating the reference-frame mapping between the detector reference frame and the distal-end reference frame includes generating a rotation matrix that describes an orientation of the distal-end reference frame relative to the intermediate reference frame or generating a rotation matrix that describes an orientation of the detector reference frame relative to the intermediate reference frame.
24. The method of claim 23 , wherein generating the reference-frame mapping between the detector reference frame and the distal-end reference frame includes both generating the rotation matrix that describes the orientation of the distal-end reference frame relative to the intermediate reference frame and generating the rotation matrix that describes the orientation of the detector reference frame relative to the intermediate reference frame.
25. The method of claim 24 , wherein the intermediate reference frame is a reference frame of a proximal end of the bendable medical device or is a reference frame of a position-tracking system.