Mixing directly visualized with rendered elements to display blended elements and actions happening on-screen and off-screen

Inventors: Frederick E. Shelton, IV (Hillsboro, OH); Shane R. Adams (Lebanon, OH); Matthew D. Cowperthwait (Cincinnati, OH); Kevin M. Fiebig (Cincinnati, OH); Jason L. Harris (Lebanon, OH); Cory G. Kimball (Hamilton, OH); Monica L. Z. Rivard (Cincinnati, OH); Leonardo N. Rossoni (Rahway, NJ); Risto Kojcev (Santa Clara, CA); Felix J. Bork (Schnürpflingen, DE)

Assignee: Cilag GmbH International

G06T11/00 A61B34/20 A61B34/25 A61B34/30 A61B34/32 A61B34/76 A61B90/36 A61B90/361 A61B90/37 A61B90/39 G06F3/14 G06F3/1454 G06F3/147 G06T7/0012 G06T7/20 G06T19/006 G06V20/20 G08B21/182 G16H20/40 G16H40/67 H04L63/105 H04L67/12 H04W24/10 H04W76/14 A61B2034/102 A61B2034/2055 A61B2034/2072 A61B2090/365 A61B2090/368 A61B2090/371 A61B2090/372 A61B2090/373 A61B2090/3937 A61B2090/3975 G06T2207/10028 G06T2207/30024 G06T2210/41

View Patent ↗

Loading inventors, assignments & file history…

Monitor This Case

Get email alerts when status or documents change.

Order Certified Copies

Most orders are placed with the USPTO same day — all within 24 business hours.

Order via The Patent Place →

Pre-filled with this patent's details

Quick Facts

Patent No.: US 12670670
App. No.: 17/688,653
Granted: Jun 30, 2026
Kind: B2

External Links

USPTO Patent Center↗ Full Patent PDF↗

Abstract

Apparatuses, systems, and method for mixed reality visualization are disclosed herein. In one aspect, a method for mixed reality visualization includes capturing, by a first camera of a first visualization system, an image of an object in a surgical field, wherein a first portion of the object is outside of a field of view of the first camera; tracking, by a tracking system, a position of a second portion of the object; determining, by a surgical hub, an attribute of the object based on the tracked position of the second portion of the object, wherein the attribute of the object is related to the first portion of the object outside of a field of view of the camera; and displaying, by an augmented reality display device, the captured image of the object in the surgical field and a graphic based on the attribute of the object.

Claims (29)

1 . A method for mixed reality visualization of a surgical system, the method comprising:

obtaining an image of a surgical instrument in a surgical field via an imaging device, wherein a first portion of the surgical instrument is outside a field of view of the imaging device and wherein a second portion of the surgical instrument is within the field of view of the imaging device;

tracking a position of the second portion of the surgical instrument within the field of view of the imaging device;

determining, based on the position of the second portion of the surgical instrument, an interaction between the surgical instrument and an object, wherein the interaction is outside the field of view of the imaging device;

determining an operating parameter associated with the first portion of the surgical instrument that is outside the field of view of the imaging device, wherein the operating parameter is determined based on the interaction between the surgical instrument and the object;

rendering an element based on the operating parameter and the interaction, wherein the element is associated with the first portion of the surgical instrument that is outside the field of view of the imaging device; and

displaying a blended image, wherein the blended image comprises the obtained image and the element.

2 . The method of claim 1 , wherein the position is tracked based on data associated with at least one of: a structured light sensor, a light detection and ranging (LIDAR) sensor, radio frequency identification (RFID), audio beaconing, or non-visual light tracking.

3 . The method of claim 1 , wherein the object is a second surgical instrument, and wherein the element indicates that the interaction is an unintended collision between the surgical instrument and the second surgical instrument occurring outside the field of view of the imaging device.

4 . The method of claim 1 , wherein the surgical instrument comprises a fiducial marker, and wherein tracking the position of the second portion of the surgical instrument is tracked via the fiducial marker.

5 . The method of claim 4 , wherein the method further comprises:

determining the operating parameter further based at least on the fiducial marker, wherein the operating parameter is associated with a range of motion of the surgical instrument, and wherein the element indicates the range of motion of the surgical instrument.

6 . A surgical system for mixed reality visualization, the surgical system comprising:

a processor configured to:

obtain an image of a surgical instrument in a surgical field via an imaging device, wherein a first portion of the surgical instrument is outside a field of view of the imaging device and wherein a second portion of the surgical instrument is within the field of view of the imaging device;

track a position of the second portion of the surgical instrument within the field of view of the imaging device;

determine, based on the position of the second portion of the surgical instrument, an interaction between the surgical instrument and an object;

determine an operating parameter associated with the first portion of the surgical instrument that is outside the field of view of the imaging device, wherein the operating parameter is determined based on the interaction between the surgical instrument and the object;

render an element based on the operating parameter and the interaction, wherein the element is associated with the first portion of the surgical instrument that is outside the field of view of the imaging device; and

display a blended image, wherein the blended image comprises the obtained image and the element.

7 . The surgical system of claim 6 , wherein the position is tracked based on data associated with at least one of: a structured light sensor, a light detection and ranging (LIDAR) sensor, radio frequency identification (RFID), audio beaconing, or non-visual light tracking.

8 . The surgical system of claim 6 , wherein the object is a second surgical instrument, and wherein the element indicates that the interaction is an unintended collision between the surgical instrument and the second surgical instrument occurring outside the field of view of the imaging device.

9 . The surgical system of claim 6 , wherein the surgical instrument comprises a fiducial marker, and wherein the position of the second portion the surgical instrument is tracked via the fiducial marker.

10 . The surgical system of claim 9 , wherein the processor is further configured to;

determine the operating parameter further based at least on the fiducial marker, wherein the operating parameter is associated with a range of motion of the surgical instrument, and wherein the element indicates the range of motion of the surgical instrument.

11 . The method of claim 1 , wherein the operating parameter is at least one of a speed, a force, a firing speed, a firing force, an activation status, a power level, an activation time, or an energy mode associated with the surgical instrument.

12 . The surgical system of claim 6 , wherein the operating parameter is at least one of a speed, a force, a firing speed, a firing force, an activation status, a power level, an activation time, or an energy mode associated with the surgical instrument.

13 . The method of claim 1 , wherein the operating parameter is a force associated with the interaction, and wherein the element indicates the force of the interaction.

14 . The method of claim 1 , wherein the operating parameter is a speed associated with the interaction, and wherein the element indicates the speed of the interaction.