Granted Patent B2

US 12,464,251 · App. 18/564,310 · Granted Nov 4, 2025

Inventors: Zhen He (Cupertino, CA); Jeffrey M. DiCarlo (Austin, TX); Max J. Trejo (San Jose, CA)

Assignee: Intuitive Surgical Operations, Inc.

H04N23/71A61B5/0071H04N23/11H04N23/72H04N23/73H04N23/74H04N23/76A61B2505/05

Abstract

Claims (94)

1 . An apparatus comprising:

one or more processors; and

memory storing executable instructions that, when executed by the one or more processors, cause the apparatus to:

obtain an image frame captured by an image capture system in accordance with an auto-exposure parameter set to a first setting, the image frame including a first image component associated with visible light and a second image component associated with non-visible light;

determine, based on the first image component, a first auto-exposure gain;

determine, based on the second image component, a second auto-exposure gain; and

determine, based on the first and second auto-exposure gains, a second setting for the auto-exposure parameter, the second setting configured to be used by the image capture system to capture a subsequent image frame;

wherein the determining the first auto-exposure gain includes:

detecting a first auto-exposure value for the first image component, the first auto-exposure value representing one or more auto exposure-related characteristics of the first image component,

accessing a first auto-exposure target for the first image component, the first auto-exposure target representing a target for the first auto-exposure value, and

determining the first auto-exposure gain based on the first auto-exposure target and the first auto-exposure value; and

wherein the determining the second auto-exposure gain includes:

detecting a second auto-exposure value for the second image component, the second auto-exposure value representing one or more auto exposure-related characteristics of the second image component,

accessing a second auto-exposure target for the second image component, the second auto-exposure target representing a target for the second auto-exposure value, and

determining the second auto-exposure gain based on the second auto-exposure target and the second auto-exposure value.

2 . The apparatus of claim 1 , wherein:

the non-visible light comprises fluorescence excitation illumination emitted at a wavelength included in an infrared light region; and

the second image component associated with the non-visible light shows fluorescence illumination emitted by a fluorescence imaging agent excited by the fluorescence excitation illumination.

3 . The apparatus of claim 1 , wherein:

the accessing the first auto-exposure target includes accessing data representative of a first user-selected setting indicative of a user preference for a luminance with which the first image component is to be presented; and

the accessing the second auto-exposure target includes accessing data representative of a second user-selected setting indicative of a user preference for a luminance with which the second image component is to be presented.

4 . The apparatus of claim 1 , wherein:

the determining the first auto-exposure gain is performed using a first auto-exposure gain algorithm; and

the determining the second auto-exposure gain is performed using a second auto-exposure gain algorithm different from the first auto-exposure gain algorithm.

5 . The apparatus of claim 4 , wherein:

the first auto-exposure gain algorithm comprises:

determining a component auto-exposure value for a set of pixels of the first image component, and

determining the first auto-exposure gain based on the component auto-exposure value; and

the second auto-exposure gain algorithm comprises:

differentiating, within a set of pixels of the second image component, signal pixels of the second image component from background pixels of the second image component,

determining one or more of a signal auto-exposure value for the signal pixels or a background auto-exposure value for the background pixels, and

determining the second auto-exposure gain based on one or more of the signal auto-exposure value or the background auto-exposure value.

6 . The apparatus of claim 4 , wherein:

the first auto-exposure gain algorithm comprises:

determining a component auto-exposure value for a set of pixels of the first image component, and

determining the first auto-exposure gain based on the component auto-exposure value; and

the second auto-exposure gain algorithm comprises:

identifying, within the second image component, a signal region including pixels having auto-exposure values exceeding an auto-exposure value threshold,

determining the second auto-exposure gain based on the auto-exposure values of the pixels of the identified signal region, and

adjusting the auto-exposure value threshold based on a size of the signal region and in a manner that targets maintaining the size of the signal region at or above a signal region size threshold.

7 . The apparatus of claim 1 , wherein the auto-exposure parameter comprises an exposure time parameter corresponding to how long a shutter exposes an image sensor of the image capture system.

8 . The apparatus of claim 1 , wherein the auto-exposure parameter comprises an illumination intensity parameter corresponding to an illumination source of the image capture system.

9 . The apparatus of claim 1 , wherein the auto-exposure parameter comprises a gain parameter corresponding to one or more of an analog gain of the image capture system, an RGB gain of the image capture system, or a Bayer gain of the image capture system.

10 . The apparatus of claim 1 , wherein:

the instructions, when executed by the one or more processors, cause the apparatus to determine:

a first tentative parameter setting for the auto-exposure parameter, the first tentative parameter setting associated with the first image component and based on the first auto-exposure gain, and

a second tentative parameter setting for the auto-exposure parameter, the second tentative parameter setting associated with the second image component and based on the second auto-exposure gain; and

the determining the second setting for the auto-exposure parameter comprises merging the first and second tentative parameter settings.

11 . The apparatus of claim 1 , wherein the auto-exposure parameter comprises a shared auto-exposure parameter configured to be set with a single setting that influences both the first image component associated with the visible light and the second image component associated with the non-visible light.

12 . The apparatus of claim 1 , wherein the auto-exposure parameter comprises a non-shared auto-exposure parameter configured to be set with a dual setting that independently influences the first image component associated with the visible light and the second image component associated with the non-visible light.

13 . The apparatus of claim 1 , wherein:

the auto-exposure parameter is included in a set of auto-exposure parameters used by the image capture system to capture a series of image frames including the image frame and the subsequent image frame;

the instructions, when executed by the one or more processors, cause the apparatus to determine, based on the first and second auto-exposure gains and in a predetermined order, updated settings for each auto-exposure parameter in the set of auto-exposure parameters; and

the predetermined order in which the updated settings are determined is configured to maximize a signal integrity and minimize a noise level captured for the subsequent image frame.

14 . The apparatus of claim 13 , wherein an exposure time parameter corresponding to how long a shutter exposes an image sensor of the image capture system comes earlier in the predetermined order than a gain parameter corresponding to one or more of an analog gain of the image capture system, an RGB gain of the image capture system, or a Bayer gain of the image capture system.

15 . The apparatus of claim 1 , wherein the instructions, when executed by the one or more processors, cause the apparatus to update the auto-exposure parameter to the second setting such that the subsequent image frame will be captured in accordance with the auto-exposure parameter set to the second setting.

16 . A method comprising:

determining, by a computing device and based on a first image component that is included in an image frame captured by an image capture system and that is associated with visible light, a first auto-exposure gain;

determining, by the computing device and based on a second image component that is included in the image frame and that is associated with non-visible light, a second auto-exposure gain; and

determining, by the computing device and based on the first and second auto-exposure gains, a setting for an auto-exposure parameter that is used by the image capture system to capture a subsequent image frame;

wherein the determining the first auto-exposure gain includes:

detecting a first auto-exposure value for the first image component, the first auto-exposure value representing one or more auto exposure-related characteristics of the first image component,

accessing a first auto-exposure target for the first image component, the first auto-exposure target representing a target for the first auto-exposure value, and

determining the first auto-exposure gain based on the first auto-exposure target and the first auto-exposure value; and

wherein the determining the second auto-exposure gain includes:

detecting a second auto-exposure value for the second image component, the second auto-exposure value representing one or more auto exposure-related characteristics of the second image component,

accessing a second auto-exposure target for the second image component, the second auto-exposure target representing a target for the second auto-exposure value, and

determining the second auto-exposure gain based on the second auto-exposure target and the second auto-exposure value.

17 . The method of claim 16 , further comprising independently determining, by the computing device, a first tentative parameter setting for the auto-exposure parameter and a second tentative parameter setting for the auto-exposure parameter;

wherein:

the first tentative parameter setting is associated with the first image component and is based on the first auto-exposure gain,

the second tentative parameter setting is associated with the second image component and is based on the second auto-exposure gain, and

the determining the setting for the auto-exposure parameter comprises merging the first and second tentative parameter settings.

18 . A system comprising:

a first illumination source configured to illuminate tissue within a body with visible light during a performance of a medical procedure;

a second illumination source configured to illuminate the tissue with non-visible light during the performance of the medical procedure;

an image capture device configured to capture an image frame in accordance with an auto-exposure parameter set to a first setting, the image frame depicting an internal view of the body that features the tissue illuminated by the first and second illumination sources; and

one or more processors configured to:

determine, based on a first image component that is included in the image frame and that is associated with the visible light, a first auto-exposure gain;

determine, based on a second image component that is included in the image frame and that is associated with the non-visible light, a second auto-exposure gain; and

determine, based on the first and second auto-exposure gains, a second setting for the auto-exposure parameter, the second setting configured to be used by the image capture device to capture a subsequent image frame;

wherein the determining the first auto-exposure gain includes:

detecting a first auto-exposure value for the first image component, the first auto-exposure value representing one or more auto exposure-related characteristics of the first image component,

accessing a first auto-exposure target for the first image component, the first auto-exposure target representing a target for the first auto-exposure value, and

determining the first auto-exposure gain based on the first auto-exposure target and the first auto-exposure value; and

wherein the determining the second auto-exposure gain includes:

detecting a second auto-exposure value for the second image component, the second auto-exposure value representing one or more auto exposure-related characteristics of the second image component,

accessing a second auto-exposure target for the second image component, the second auto-exposure target representing a target for the second auto-exposure value, and

determining the second auto-exposure gain based on the second auto-exposure target and the second auto-exposure value.

19 . The system of claim 18 , wherein:

the one or more processors are configured to determine:

a first tentative parameter setting for the auto-exposure parameter, the first tentative parameter setting associated with the first image component and based on the first auto-exposure gain, and

a second tentative parameter setting for the auto-exposure parameter, the second tentative parameter setting associated with the second image component and based on the second auto-exposure gain; and

the determining the second setting for the auto-exposure parameter comprises merging the first and second tentative parameter settings.

Assignments (1)

Continuity (2)

References Cited (13)