System and method for tracking a portion of the user as a proxy for non-monitored instrument
A tracking system of operating room (OR) personnel and is configured to extrapolate the movement, position, orientation, and context of one or more of the active OR participants to determine the specific operation procedure in progress, the surgical instrument being used by the OR participant, and/or anticipated movement of the OR personnel. The system may further be configured to render a virtual element that includes an anticipated position of a surgical instrument, recommendations, guidance, warning, and surgical location information, as augmented reality (AR) content. The AR content is integrated in real-time with the live-feed of the surgical site, and aids the OR personnel in perioperative surgical procedures.
1 . A surgical system comprising:
a camera system;
an active sensor corresponding to a surgical staff member, wherein the active sensor captures active tracking data; and
a surgical hub communicatively coupled to the camera system and the active sensor, wherein the surgical hub comprises a memory and a control circuit, and wherein the control circuit is configured to:
identify a unique identifier corresponding to the surgical staff member, wherein the unique identifier comprises visual indicators located on an exterior surface of an article worn by an operating room participant, and wherein the unique identifier is visible to the camera system;
receive contextual data from an external source, wherein the contextual data comprises hand positions for surgical instruments used in a surgical procedure, wherein the hand positions comprise finger shapes and hand orientations, and wherein the contextual data comprises situational awareness information for the surgical procedure;
receive passive tracking data from the camera system associated with the surgical staff member;
receive the active tracking data from the active sensor associated with the surgical staff member;
determine that the surgical staff member is using a first surgical instrument that is not tracked by the surgical hub;
compare the passive tracking data and the active tracking data to the hand positions for the surgical instruments used in the surgical procedure, wherein the finger shapes and the hand orientations are used to identify a user using the surgical instrument for the surgical procedure;
determine a specific surgical instrument corresponding to the passive tracking data and the contextual data;
generate a virtual element signal, wherein the virtual element signal is associated with a virtual representation of the specific surgical instrument; and
display the virtual element signal on an augmented reality device, wherein the augmented reality device is configured to overlay the virtual element signal that is associated with the virtual representation of the specific surgical instrument on a real surgical environment, and wherein the real surgical environment is associated with a surgical field.
2 . The surgical system of claim 1 , wherein the article comprises a surgical glove.
3 . The surgical system of claim 2 , wherein the active sensor is embedded in the surgical glove, and wherein the active sensor is configured to measure a location, force, or proximity, or combinations thereof, to structural elements of each finger of the surgical staff member located in the surgical glove.
4 . The surgical system of claim 3 , wherein the active sensor is communicatively coupled to the control circuit with elastically deformable electrical conductors.
5 . The surgical system of claim 4 , wherein the elastically deformable electrical conductors are printed on the surgical glove.
6 . The surgical system of claim 1 , wherein a first article comprises a tracking sleeve worn on a forearm of the surgical staff member, wherein the active sensor is embedded in the tracking sleeve, and wherein the active sensor is configured to measure active movement signals corresponding to a movement of a first person.
7 . The surgical system of claim 6 , wherein the active movement signals comprise mechanomyogram (MMG) signals and electromyogram (EMG) signals.
8 . The surgical system of claim 6 , wherein a second article comprises a passive tracking surgical glove, and wherein the passive tracking data is captured based on the movement of a passive tracking glove and associated based on the active tracking data captured from tracking sleeves.
9 . The surgical system of claim 1 , wherein the unique identifier comprises a unique identifier for each finger of the surgical staff member.
10 . The surgical system of claim 1 , wherein the unique identifier comprises a first unique identifier for a left hand and a second unique identifier for a right hand of the surgical staff member.
11 . The surgical system of claim 1 , wherein the control circuit is further configured to:
determine a user input initiated by the surgical staff member corresponding to an initiation sequence that registers the unique identifier with an identity of the surgical staff member.
12 . A method for tracking movement of operating room staff members, the method comprising:
receiving, by a surgical hub, contextual data from an external source, wherein the contextual data comprises hand positions for surgical instruments used in a surgical procedure, wherein the hand positions comprise finger shapes and hand orientations, and wherein the contextual data comprises situational awareness information for the surgical procedure;
identifying, by the surgical hub, a first unique identifier associated with a first surgical staff member and a second unique identifier associated with a second surgical staff member;
receiving, by the surgical hub, passive tracking data from a camera system associated with the first surgical staff member and the second surgical staff member, wherein the passive tracking data is determined by fiducial markers;
receiving, by the surgical hub, active tracking data from an active sensor associated with the first surgical staff member and the second surgical staff member;
determining, by the surgical hub, that the first surgical staff member or the second surgical staff member is using a surgical instrument that is not tracked by the surgical hub;
comparing, by the surgical hub, the passive tracking data and the active tracking data to the hand positions for the surgical instruments used in the surgical procedure, wherein the finger shapes and the hand orientations are used to identify a user using the surgical instrument for the surgical procedure;
determining, by the surgical hub, a specific surgical instrument corresponding to the passive tracking data and the contextual data;
generating, by the surgical hub, a virtual element signal, wherein the virtual element signal is associated with a virtual representation of the specific surgical instrument;
transmitting, by the surgical hub, the virtual element signal to be displayed on an augmented reality device; and
displaying, by the surgical hub, the virtual element signal on the augmented reality device, wherein the augmented reality device is configured to overlay the virtual element signal that is associated with the virtual representation of the specific surgical instrument on a real surgical environment, and wherein the real surgical environment is associated with a surgical field.
13 . The method of claim 12 , wherein the active sensor comprises at least one strain gauge that corresponds to each finger of the first surgical staff member and the second surgical staff member.
14 . The method of claim 13 , further comprising:
determining, by the surgical hub, that the first surgical staff member is holding the surgical instrument;
determining, by the surgical hub, that a hand-off of the surgical instrument was initiated based on a change in grip position of the first surgical staff member and the second surgical staff member;
determining, by the surgical hub, that the second surgical staff member is beginning to grip the surgical instrument; and
determining, by the surgical hub, that the surgical instrument has completed the hand-off from the first surgical staff member to the second surgical staff member.
15 . The method of claim 12 , wherein at least one of the first unique identifier or the second unique identifier comprise at least a unique color, pattern, texture, or quick response (QR) code, or combinations thereof, and wherein the first unique identifier is scanned as part of an initiation sequence that registers the first unique identifier with an identity of the first surgical staff member.
16 . The method of claim 12 , wherein a fiducial marker is tracked with a light source that is outside of a visible spectrum, wherein the fiducial marker is embedded in a surgical glove, and wherein the light source is part of a wrist-mounted camera.
17 . The method of claim 12 , wherein the active sensor is embedded in a surgical glove, and wherein the active sensor is communicatively coupled to a control circuit on the surgical glove.
18 . The method of claim 17 , wherein the control circuit is enclosed in a removable housing.
19 . The method of claim 17 , wherein the active sensor and the control circuit receive power from a piezoelectric ceramic power cell.
20 . The surgical system of claim 1 , wherein the augmented reality device is worn by a surgeon performing the surgical procedure in the surgical field.
21 . The method of claim 12 , wherein the augmented reality device is worn by a surgeon performing the surgical procedure in the surgical field.