Granted Patent B2

US 12,316,965 · App. 18/174,818 · Granted May 27, 2025

Inventors: Tarik Yardibi (Cambridge, MA); Steen Hansen (Copenhagen, DK)

Assignee: Cilag GmbH International

H04N23/683H04N13/15H04N23/555H04N23/6811H04N23/6812H04N23/84H04N25/78H04N2013/0081

Abstract

Claims (43)

1. A system comprising:

an image sensor comprising a pixel array that detects electromagnetic radiation and reads out a plurality of data frames; and

an image sensor processor that receives the plurality of data frames read out by the image sensor, wherein the plurality of data frames comprises a plurality of color data frames and a plurality of advanced data frames;

wherein the image signal processor renders a video stream comprising a plurality of advanced overlay data frames, wherein each of the plurality of advanced overlay data frames comprises a color data frame and a false color overlay generated based on an advanced data frame; and

wherein the image signal processor executes an adaptive persistence algorithm to stabilize movement of the false color overlay across the plurality of advanced overlay data frames.

2. The system of claim 1 , wherein the image signal processor stabilizes the movement of the false color overlay to compensate for one or more of movement of the image sensor, noise within the plurality of data frames, fluctuations in light across the plurality of data frames, or algorithm uncertainties in generating the false color overlay.

3. The system of claim 1 , wherein the adaptive persistence algorithm comprises applying persistence on movement detections within the plurality of advanced overlay data frames to stabilize the movement of the false color overlay across the plurality of advanced overlay data frames.

4. The system of claim 3 , wherein applying the persistence on the movement detections comprises executing one or more of:

a hysteresis-based tracking algorithm;

a Kalman filter-based tracking algorithm;

a particle filter-based tracking algorithm; or

a SLAM (simultaneous localization and mapping) based tracking algorithm.

5. The system of claim 1 , further comprising an inertial measurement unit (IMU) associated with the image sensor, wherein the IMU measures movement of the image sensor in real-time; and

wherein the image signal processor further receives sensor data from the IMU; and

wherein the adaptive persistence algorithm comprises stabilizing the movement of the false color overlay at least based on the sensor data from the IMU.

6. The system of claim 5 , wherein the image signal processor is further configured to automatically adjust one or more of a hysteresis or a memory of the adaptive persistence algorithm in real-time.

7. The system of claim 6 , wherein the image signal processor adjusts the one or more of the hysteresis or the memory of the adaptive persistence algorithm in real-time based at least on a velocity of movement of the image sensor determined based on the sensor data from the IMU.

8. The system of claim 6 , wherein the image signal processor adjusts the one or more of the hysteresis or the memory of the adaptive persistence algorithm in real-time based at least on a velocity of movement of the image sensor determined based on motion tracking of objects within sequential data frames output by the image sensor.

9. The system of claim 1 , wherein the image sensor comprises two image sensors that simultaneously output a plurality of stereo pairs of data frames; and

wherein the system further comprises a controller that provides the plurality of stereo pairs of data frames to an algorithm configured to generate a plurality of disparity maps, wherein each of the plurality of disparity maps is calculated based on a stereo pair of data frames.

10. The system of claim 9 , wherein the controller is further configured to provide relative positions of the two image sensors and the plurality of disparity maps to an algorithm configured to calculate a three-dimensional topographical map of a scene using triangulation.

11. The system of claim 1 , further comprising an emitter comprising a plurality of electromagnetic sources comprising a white light source and further comprising one or more of:

a plurality of excitation sources configured to emit electromagnetic radiation within a waveband selected for fluorescing a tissue or reagent; or

a plurality of multispectral sources configured to emit electromagnetic radiation within a waveband selected for eliciting a spectral response from a tissue.

12. The system of claim 1 , wherein each of the plurality of advanced data frames comprises one or more of:

a fluorescence data frame detected in response to an emitter emitting a fluorescence excitation wavelength of electromagnetic radiation selected to fluoresce one or more of a tissue or a reagent; or

a multispectral data frame detected in response to the emitter emitting a multispectral wavelength of electromagnetic radiation selected to elicit a spectral response from a tissue and/or penetrate through a tissue.

13. The system of claim 1 , wherein the image signal processor communicates with a deep learning algorithm configured to identify a target object within a scene based on one or more of the plurality of advanced data frames; and

wherein the false color overlay highlights the target object within the scene, and wherein the target object comprises one or more of an arterial tissue, a venous tissue, a nervous tissue, a cancerous tissue, a tissue perfusion, or a ureter tissue.

14. The system of claim 13 , wherein the false color overlay is rendered as a heatmap indicating a likelihood that a region comprises the target object.

15. The system of claim 13 , wherein executing the adaptive persistence algorithm to stabilize movement of the false color overlay comprises:

averaging a rolling window of the plurality of advanced data frames over time, wherein a quantity of data frames within the rolling window is determined at least in part on movement of the image sensor; and

rendering the false color overlay based on the averaged advanced data frames to compensate for one or more of noise, light fluctuations, or algorithm uncertainties.

16. The system of claim 13 , wherein executing the adaptive persistence algorithm to stabilize movement of the false color overlay comprises:

identifying one or more pixels across a portion of the plurality of advanced data frames that is inconsistently identified as comprising the target object by the deep learning algorithm;

averaging readings for each of the one or more pixels across the portion of the plurality of advanced data frames; and

stabilizing the false color overlay across a corresponding portion of the advanced overlay data frames based on the averaged readings for each of the one or more pixels.

17. The system of claim 1 , further comprising an inertial measurement unit (IMU) associated with the image sensor, wherein the IMU measures movement of the image sensor in real-time; and

wherein stabilizing the movement of the false color overlay across the plurality of advanced overlay data frames comprises using x, y, z acceleration values retrieved from the IMU to automatically adjust one or more of a hysteresis or a memory of the adaptive persistence algorithm.

18. The system of claim 1 , wherein stabilizing the movement further comprises stabilizing based on a temporal gradient of the plurality of data frames over time.

19. The system of claim 1 , further comprising calculating an estimated movement of the image sensor based on differences two or more sequential data frames of the plurality of data frames;

wherein stabilizing the movement of the false color overlay across the plurality of advanced overlay data frames comprises utilizing the estimated movement of the image sensor to automatically adjust one or more of a hysteresis or a memory of the adaptive persistence algorithm.

20. The system of claim 1 , wherein the image signal processor automatically adjusts a forgetting factor for the adaptive persistence algorithm based on a velocity of movement of the image sensor.

Assignments (1)

Continuity (1)

References Cited (164)