Surgical workflow deviation detection involving head-mounted device
A head-mounted device (HMD) and method for assisting a user during a surgical procedure. The HMD has a camera and display supported by a head-mountable structure. The display is positionable in front of eyes of the user. A controller accesses workflow data defining workflow steps associated with the surgical procedure. The controller obtains data from the camera related to an environment of the surgical procedure. The camera data is monitored to detect a deviation or a potential deviation from one or more of the workflow steps. The controller generates an instruction/recommendation to assist the user in taking action to address the deviation or the potential deviation. The instruction/recommendation is specifically tailored to workflow responsibilities of the user. A computer-generated image related to the instruction or the recommendation is provided on the display.
1. A head-mounted device (HMD) for assisting a HMD user among a plurality of users during a surgical procedure, the HMD comprising:
a head-mountable structure configured to be worn by the HMD user;
a camera supported by the head-mountable structure;
a display supported by the head-mountable structure and being positionable in front of eyes of the HMD user; and
a controller configured to:
access pre-scripted workflow data defining a plurality of workflow steps associated with the surgical procedure;
obtain, from the camera, camera data related to an environment of the surgical procedure;
process the camera data to determine locations of a plurality of surgical objects;
monitor the camera data to detect a deviation or a potential deviation from one or more of the workflow steps of the plurality of workflow steps;
determine if the deviation or potential deviation is associated with the HMD user among the plurality of user, and upon determining that the deviation or potential deviation is associated with the HMD user, generate an instruction or a recommendation to assist the HMD user in taking action to address the deviation or the potential deviation, wherein the instruction or the recommendation is specifically tailored to workflow responsibilities of the HMD user;
produce a computer-generated image related to the instruction or the recommendation; and
provide the computer-generated image on the display in front of the eyes of the HMD user determined to be associated with the deviation or potential deviation.
2. The HMD of claim 1 , wherein the instruction or the recommendation relates to a surgical object in the environment.
3. The HMD of claim 2 , wherein the surgical object is a surgical tool.
4. The HMD of claim 3 , wherein the instruction or the recommendation relates to replacement or changing of the surgical tool.
5. The HMD of claim 3 , wherein the instruction or the recommendation identifies a location of the surgical tool in the environment.
6. The HMD of claim 2 , wherein the instruction or the recommendation relates to rearrangement or repositioning of the surgical object.
7. The HMD of claim 6 , wherein:
the surgical object is an anatomy of a patient; and
the instruction or the recommendation relates to rearrangement or repositioning the anatomy.
8. The HMD of claim 2 , wherein the surgical object is a planned implant.
9. The HMD of claim 8 , wherein the instruction or the recommendation relates to changing the planned implant.
10. The HMD of claim 1 , the instruction or the recommendation relates to a surgical plan associated with the surgical procedure.
11. The HMD of claim 10 , further comprising an input device configured to enable the HMD user to modify the surgical plan.
12. The HMD of claim 11 , wherein the input device comprises a sensor configured to detect a gesture command from the HMD user to modify the surgical plan.
13. The HMD of claim 1 , wherein the controller is configured to produce the computer-generated image as a list or menu of items selectable by the HMD user.
14. The HMD of claim 1 , wherein:
the camera data comprises real-world video of the environment of the surgical procedure; and
the controller is configured to combine the computer-generated image with the real-world video.
15. The HMD of claim 1 , wherein:
the display is transparent to enable the HMD user to see-through the display to observe the environment; and
the controller is configured to superimpose the computer-generated image onto the display.
16. The HMD of claim 1 , wherein the camera supported by the head-mountable structure is included in a machine vision system to determine locations of the plurality of surgical objects.
17. A method of operating a head-mounted device (HMD) for assisting a HMD user among a plurality of users during a surgical procedure, the HMD including a head-mountable structure configured to be worn by the HMD user, a camera supported by the head-mountable structure, a display supported by the head-mountable structure and being positionable in front of eyes of the user, and a controller, the method comprising the controller performing the following:
accessing pre-scripted workflow data defining a plurality of workflow steps associated with the surgical procedure;
obtaining, from the camera, camera data related to an environment of the surgical procedure;
processing the camera data to determine locations of a plurality of surgical objects;
monitoring the camera data to detect a deviation or a potential deviation from one or more of the workflow steps;
determining if the deviation or potential deviation is associated with the HMD user among the plurality of user, and
upon determining that the deviation or potential deviation is associated with the HMD user, generating an instruction or a recommendation to assist the HMD user in taking action to address the deviation or the potential deviation, and specifically tailoring the instruction or the recommendation to workflow responsibilities of the HMD user;
producing a computer-generated image related to the instruction or the recommendation; and
providing the computer-generated image on the display in front of the eyes of the HMD user determined to be associated with the deviation or potential deviation.
18. A surgical system comprising:
a plurality of head-mounted devices (HMDs) configured to be worn by a user, for assisting users during a surgical procedure, each HMD comprising: a head-mountable structure; a camera supported by the head-mountable structure; a display supported by the head-mountable structure and being positionable in front of eyes of the user; and a controller;
a surgical workflow control system configured to:
assign each one of the HMDs to one of the users;
access pre-scripted workflow data defining a plurality of workflow steps associated with the surgical procedure;
obtain, from one or more cameras of the HMDs, camera data related to an environment of the surgical procedure;
process the camera data to determine locations of a plurality of surgical objects;
monitor the camera data to detect a deviation or a potential deviation from one or more of the workflow steps of the plurality of workflow steps;
identify which user is implicated by the deviation or the potential deviation;
generate an instruction or a recommendation to assist the implicated user in taking action to address the deviation or the potential deviation, wherein the instruction or the recommendation is specifically tailored to workflow responsibilities of the implicated user; and
communicate with the controller of the HMD assigned to the implicated user to instruct the HMD to produce a computer-generated image related to the instruction or the recommendation and provide the computer-generated image on the display of the HMD assigned to the implicated user.
19. The surgical system of claim 18 , the instruction or the recommendation is related to a surgical object in the environment.
20. The surgical system of claim 18 , wherein the surgical object is a surgical tool, and the instruction or the recommendation is related to replacing or changing of the surgical tool or a location of the surgical tool in the environment.
21. The surgical system of claim 18 , wherein the surgical object is an anatomy of a patient, and the instruction or the recommendation relates to rearranging or repositioning the anatomy.
22. The surgical system of claim 18 , wherein the surgical object is a planned implant, and the instruction or the recommendation relates to changing the planned implant.
23. The surgical system of claim 18 further comprising one or both of: a surgical navigation system comprising an optical localizer and trackers to determine locations of a plurality of surgical objects and generate camera video data; and a machine vision system comprising at least one camera to determine locations of the plurality of surgical objects and generate camera video data.