Methods of guiding manual movement of medical systems
A computer-assisted medical system includes a user device wearable by an operator. The user device includes a display device configured to present imagery overlaid in an environment of a manipulator assembly, and a sensor configured to detect one or more landmarks in the environment. The medical system includes a controller configured to execute instructions to perform operations The operations include receiving, from the sensor, position or orientation information for the one or more landmarks in the environment, and directing a manual movement of a portion of the manipulator assembly by causing the display device to present the imagery overlaid in the environment based on the received position or orientation information.
1. A computer-assisted medical system comprising:
a user device wearable by an operator, the user device comprising:
a display device configured to present imagery overlaid in an environment of a manipulator assembly, and
a sensor configured to detect one or more landmarks in the environment; and
a controller configured to execute instructions to perform operations, the operations comprising:
receiving, from the sensor, position or orientation information for the one or more landmarks in the environment, and
directing a manual movement of a portion of the manipulator assembly by causing the display device to present guidance for performing the manual movement within the imagery overlaid in the environment based on the received position or orientation information, wherein:
causing the display device to present the imagery comprises causing the display device to present the imagery overlaid with a floor surface in the environment to direct the manual movement of the portion of the manipulator assembly along the floor surface, or
the imagery indicates a path from a current location of the portion of the manipulator assembly toward a recommended location of the portion of the manipulator assembly.
2. The medical system of claim 1 , wherein the operations further comprise:
directing a manual movement of a portion of the medical system relative to the manipulator assembly by causing the display device to present the imagery overlaid in the environment based on the received position or orientation information, the portion of the medical system being distinct from the manipulator assembly.
3. The medical system of claim 1 , wherein:
the imagery is first imagery, and
the operations further comprise directing a manual movement of a patient relative to the manipulator assembly by causing the display device to present second imagery overlaid in the environment based on the received position or orientation information.
4. The medical system of claim 1 , wherein:
the operations further comprise receiving user input information and determining a recommended configuration based on the user input information, and
directing the manual movement comprises directing the manual movement further based on the recommended configuration.
5. The medical system of claim 1 , wherein:
directing the manual movement of the portion of the manipulator assembly comprises directing a user to reorient the portion of the manipulator assembly relative to an alignment indicator in the imagery; or
the imagery overlaid in the environment is indicative of one or more waypoints for the manual movement of the portion of the manipulator assembly.
6. The medical system of claim 1 , wherein the imagery overlaid in the environment comprises a map having a first region indicative of undesirable locations for the portion of the manipulator assembly or indicative of desirable locations for the portion of the manipulator assembly.
7. The medical system of claim 1 , wherein the imagery overlaid in the environment comprises:
a map having a first region indicative of desirable locations for the portion of the manipulator assembly and a second region indicative of undesirable locations for the portion of the manipulator assembly; or
a map indicative of a desirability value for each location of a plurality of locations; or
an annotation indicative of a direction of motion or a magnitude of motion toward a desirable location for the portion of the manipulator assembly.
8. The medical system of claim 1 , wherein the imagery further comprises an indicator of an obstacle along the floor surface or proximate the path.
9. The medical system of claim 1 , wherein the imagery:
indicates a recommended port location for a manipulator of the manipulator assembly, or indicates a keep out region for the manipulator assembly, or indicates a recommended configuration of the manipulator assembly, or comprises a representation of an internal anatomy of a patient.
10. The medical system of claim 1 , wherein:
the operator is a first operator,
the user device is a first user device wearable by the first operator,
the display device is a first display device,
the imagery is first imagery,
the one or more landmarks are one or more first landmarks, and
the medical system further comprises a second user device wearable by a second operator, the second user device comprising a second display device configured to present second imagery overlaid in the environment of the manipulator assembly, and a second sensor configured to detect one or more second landmarks in the environment, and
the operations further comprise causing the second display device to present the second imagery overlaid in the environment based on a same reference frame as used for causing the first display device to present the first imagery.
11. The medical system of claim 1 , wherein:
the operator is a first operator,
the user device is a first user device,
the imagery is a first imagery, and
the operations further comprise causing the display device of the first user device to present second imagery captured by a second user device worn by a second operator.
12. The medical system of claim 1 , wherein:
the operator is a first operator,
the user device is a first user device,
the medical system comprises a plurality of user devices, the plurality of user devices comprising the first user device and a second user device wearable by a second operator,
the imagery is a first imagery, and
the operations further comprise directing manual movement of a part of the medical system distinct from the manipulator assembly by causing a second display device of the second user device to present second imagery overlaid in the environment.
13. The medical system of claim 1 , wherein:
the sensor is configured to detect movement of a user, and
directing the manual movement of the portion of the manipulator assembly comprises directing the manual movement of the portion of the manipulator assembly based on the detected movement of the user.
14. The medical system of claim 13 , wherein the operations further comprise determining a desired setup configuration for the medical system based on the detected movement of the user.
15. The medical system of claim 1 , wherein:
the sensor is configured to detect a user gesture, and
the operations further comprise updating the imagery in response to the user gesture,
the imagery comprises an indicator overlaid in the environment at a first target location for the portion of the manipulator assembly, and
updating the imagery comprises updating the imagery such that the indicator is moved relative to the environment in response to the user gesture.
16. The medical system of claim 1 , wherein:
the sensor is configured to detect an actual port location,
the operations further comprise updating the imagery in response to the actual port location.
17. The medical system of claim 1 , wherein the operations further comprise:
based on updated position or orientation information for the one or more landmarks, updating the imagery to direct an updated manual movement of a portion of the manipulator assembly; or
causing the display device to present the imagery further based on a kinematic configuration of a manipulator assembly.
18. The medical system of claim 1 , wherein the operations comprise:
before directing the manual movement, generating the imagery based on prior user-directed motion of the manipulator assembly, generating the imagery based on a type of a medical procedure to be performed by the manipulator assembly, generating the imagery based on a target or actual location of a port on a patient, or generating the imagery based on a target or actual configuration of the manipulator assembly, or generating the imagery based on a predefined plan indicative of recommended positions or orientations of the one or more landmarks in the environment.
19. The medical system of claim 1 , wherein:
the operations further comprise receiving contextual information, the contextual information selected from the group consisting of: equipment information, operator information, obstacle information, and patient information, and
directing the manual movement comprises directing the manual movement further based on the contextual information.
20. The medical system of claim 1 , further comprising:
the manipulator assembly, wherein the manipulator assembly comprises a teleoperated manipulator; and
an input device for receiving user commands to move the teleoperated manipulator.
21. A method of setting up a computer-assisted medical system comprising a manipulator assembly, the method comprising:
receiving, from a sensor of a user device of the computer-assisted medical system, position or orientation information for one or more landmarks in an environment, and
directing a manual movement of a portion of the manipulator assembly by causing a display device of the user device to present guidance for performing the manual movement within imagery overlaid in the environment based on the received position or orientation information, wherein:
causing the display device to present the imagery comprises causing the display device to present the imagery overlaid with a floor surface in the environment to direct the manual movement of the portion of the manipulator assembly along the floor surface, or
the imagery indicates a path from a current location of the portion of the manipulator assembly toward a recommended location of the portion of the manipulator assembly.
22. The method of claim 21 , further comprising:
directing a manual movement of a portion of the medical system relative to the manipulator assembly by causing the display device to present the imagery overlaid in the environment, the portion of the medical system being distinct from the manipulator assembly.
23. The method of claim 22 , wherein the imagery overlaid in the environment comprises:
a first region indicative of undesirable locations for the portion of the manipulator assembly; or
a second region indicative of desirable locations for the portion of the manipulator assembly; or
a desirability value for the portion of the manipulator assembly in each location of a plurality locations in the environment.
24. The method of claim 21 , further comprising:
creating the imagery based on data comprising information selected from the group consisting of: a target location of a port for an entry by the manipulator assembly, an actual location of the port, procedure information, equipment information, operator information, obstacle information, patient information, and a predefined plan for a medical procedure to be performed by the manipulator assembly.
25. A non-transitory machine-readable medium comprising a plurality of machine-readable instructions which, when executed by one or more processors associated with a medical system comprising a manipulator assembly, are adapted to cause the one or more processors to perform a method comprising:
receiving, from a sensor of a user device of the medical system, position or orientation information for one or more landmarks in an environment, and
directing a manual movement of a portion of the manipulator assembly by causing a display device of the user device to present guidance for performing the manual movement within imagery overlaid in the environment based on the received position or orientation information, wherein:
causing the display device to present the imagery comprises causing the display device to present the imagery overlaid with a floor surface in the environment to direct the manual movement of the portion of the manipulator assembly along the floor surface, or
the imagery indicates a path from a current location of the portion of the manipulator assembly toward a recommended location of the portion of the manipulator assembly.
26. The non-transitory machine-readable medium of claim 25 , wherein the method further comprises:
creating the imagery based on data comprising information selected from the group consisting of: a target location of a port for an entry by the manipulator assembly, an actual location of the port, procedure information, equipment information, operator information, obstacle information, patient information, and a predefined plan for a medical procedure to be performed by the manipulator assembly.