Mixing directly visualized with rendered elements to display blended elements and actions happening on-screen and off-screen
Apparatuses, systems, and method for mixed reality visualization are disclosed herein. In one aspect, a method for mixed reality visualization includes capturing, by a first camera of a first visualization system, an image of an object in a surgical field, wherein a first portion of the object is outside of a field of view of the first camera; tracking, by a tracking system, a position of a second portion of the object; determining, by a surgical hub, an attribute of the object based on the tracked position of the second portion of the object, wherein the attribute of the object is related to the first portion of the object outside of a field of view of the camera; and displaying, by an augmented reality display device, the captured image of the object in the surgical field and a graphic based on the attribute of the object.
1 . A method for mixed reality visualization of a surgical system, the method comprising:
obtaining an image of a surgical instrument in a surgical field via an imaging device, wherein a first portion of the surgical instrument is outside a field of view of the imaging device and wherein a second portion of the surgical instrument is within the field of view of the imaging device;
tracking a position of the second portion of the surgical instrument within the field of view of the imaging device;
determining, based on the position of the second portion of the surgical instrument, an interaction between the surgical instrument and an object, wherein the interaction is outside the field of view of the imaging device;
determining an operating parameter associated with the first portion of the surgical instrument that is outside the field of view of the imaging device, wherein the operating parameter is determined based on the interaction between the surgical instrument and the object;
rendering an element based on the operating parameter and the interaction, wherein the element is associated with the first portion of the surgical instrument that is outside the field of view of the imaging device; and
displaying a blended image, wherein the blended image comprises the obtained image and the element.
2 . The method of claim 1 , wherein the position is tracked based on data associated with at least one of: a structured light sensor, a light detection and ranging (LIDAR) sensor, radio frequency identification (RFID), audio beaconing, or non-visual light tracking.
3 . The method of claim 1 , wherein the object is a second surgical instrument, and wherein the element indicates that the interaction is an unintended collision between the surgical instrument and the second surgical instrument occurring outside the field of view of the imaging device.
4 . The method of claim 1 , wherein the surgical instrument comprises a fiducial marker, and wherein tracking the position of the second portion of the surgical instrument is tracked via the fiducial marker.
5 . The method of claim 4 , wherein the method further comprises:
determining the operating parameter further based at least on the fiducial marker, wherein the operating parameter is associated with a range of motion of the surgical instrument, and wherein the element indicates the range of motion of the surgical instrument.
6 . A surgical system for mixed reality visualization, the surgical system comprising:
a processor configured to:
obtain an image of a surgical instrument in a surgical field via an imaging device, wherein a first portion of the surgical instrument is outside a field of view of the imaging device and wherein a second portion of the surgical instrument is within the field of view of the imaging device;
track a position of the second portion of the surgical instrument within the field of view of the imaging device;
determine, based on the position of the second portion of the surgical instrument, an interaction between the surgical instrument and an object;
determine an operating parameter associated with the first portion of the surgical instrument that is outside the field of view of the imaging device, wherein the operating parameter is determined based on the interaction between the surgical instrument and the object;
render an element based on the operating parameter and the interaction, wherein the element is associated with the first portion of the surgical instrument that is outside the field of view of the imaging device; and
display a blended image, wherein the blended image comprises the obtained image and the element.
7 . The surgical system of claim 6 , wherein the position is tracked based on data associated with at least one of: a structured light sensor, a light detection and ranging (LIDAR) sensor, radio frequency identification (RFID), audio beaconing, or non-visual light tracking.
8 . The surgical system of claim 6 , wherein the object is a second surgical instrument, and wherein the element indicates that the interaction is an unintended collision between the surgical instrument and the second surgical instrument occurring outside the field of view of the imaging device.
9 . The surgical system of claim 6 , wherein the surgical instrument comprises a fiducial marker, and wherein the position of the second portion the surgical instrument is tracked via the fiducial marker.
10 . The surgical system of claim 9 , wherein the processor is further configured to;
determine the operating parameter further based at least on the fiducial marker, wherein the operating parameter is associated with a range of motion of the surgical instrument, and wherein the element indicates the range of motion of the surgical instrument.
11 . The method of claim 1 , wherein the operating parameter is at least one of a speed, a force, a firing speed, a firing force, an activation status, a power level, an activation time, or an energy mode associated with the surgical instrument.
12 . The surgical system of claim 6 , wherein the operating parameter is at least one of a speed, a force, a firing speed, a firing force, an activation status, a power level, an activation time, or an energy mode associated with the surgical instrument.
13 . The method of claim 1 , wherein the operating parameter is a force associated with the interaction, and wherein the element indicates the force of the interaction.
14 . The method of claim 1 , wherein the operating parameter is a speed associated with the interaction, and wherein the element indicates the speed of the interaction.