IP Library Granted Patent US 12,025,559
Granted Patent B2
US 12,025,559 · App. 17/313,513 · Granted Jul 2, 2024

Minimizing image sensor input/output in a pulsed laser mapping imaging system

Inventors: Joshua D. Talbert (Salt Lake City, UT); Donald M. Wichern (Ogden, UT)
Assignee: Cilag GmbH International
G01N21/6456A61B1/00006A61B1/000095A61B1/043A61B1/0638A61B1/0655A61B5/0261A61B5/20A61B5/4887A61B5/489A61B5/4893G01S17/89G02B5/208G06T7/0012G06T7/20G06T7/521G06T7/70H04N13/239H04N23/125H04N23/74H04N23/743A61B1/063A61B1/07G01S7/483G06T2207/10012G06T2207/10016G06T2207/10024G06T2207/10028G06T2207/10036G06T2207/10064G06T2207/10068G06T2207/10152G06T2207/20212G06T2207/30096G06T2207/30101G06T2207/30104H04N23/555
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,025,559
App. No.
17/313,513
Granted
Jul 2, 2024
Kind
B2
Abstract

Minimizing image sensor input/output pads in a pulsed laser mapping imaging system is disclosed. A system includes an emitter for emitting pulses of electromagnetic radiation and an image sensor comprising a pixel array for sensing reflected electromagnetic radiation. The system includes a plurality of bidirectional pads comprising an output state for issuing data and an input state for receiving data. The system includes a controller configured to synchronize timing of the emitter and the image sensor. The system is such that at least a portion of the pulses of electromagnetic radiation emitted by the emitter comprises a laser mapping pattern.

Claims (57)

1. A system comprising:

an image sensor comprising a pixel array for sensing reflected electromagnetic radiation;

a sensor data pad in communication with the image sensor that is capable of issuing and receiving data;

an emitter for emitting pulses of electromagnetic radiation, wherein the emitter comprises:

a visible source for emitting a visible pulse of electromagnetic radiation, and

a mapping source for emitting a mapping pulse of electromagnetic radiation for determining a dimension, a distance, or a topology of one or more objects within a light deficient environment; and

a controller in electronic communication with the image sensor and the emitter;

wherein the image sensor captures mapping data in response to an emission by the mapping source of the emitter.

2. The system of claim 1 , wherein the mapping source is a laser mapping source and comprises one or more sources of electromagnetic radiation for pulsing one or more of: a grid array comprising one or more of vertical hashing or horizontal hashing, a raster grid of discrete points, an occupancy grid map, or a dot array.

3. The system of claim 1 , wherein the mapping pattern comprises one or more of: a grid array comprising vertical hashing and horizontal hashing, a raster grid of discrete points, an occupancy grid map, or a dot array.

4. The system of claim 1 , wherein the mapping data comprises information for generating a three-dimensional topology of a scene.

5. The system of claim 1 , wherein the mapping data comprises information for tracking a location of an object within a light deficient environment, wherein the emitter is the only source of electromagnetic radiation within the light deficient environment.

6. The system of claim 1 , wherein the mapping data comprises information for calculating the dimension, the distance, or the topology of one or more objects within the light deficient environment, wherein the emitter is the only source of electromagnetic radiation within the light deficient environment.

7. The system of claim 1 , wherein the emitter comprises a plurality of sources of electromagnetic radiation, wherein the plurality of sources of electromagnetic radiation comprises:

the mapping source for pulsing the mapping pattern;

a red source for pulsing a red wavelength of electromagnetic radiation;

a green source for pulsing a green wavelength of electromagnetic radiation; and

a blue source for pulsing a blue wavelength of electromagnetic radiation;

wherein each of the mapping source, the red source, the green source, and the blue source can be independently actuated by the controller.

8. The system of claim 7 , wherein the plurality of sources of electromagnetic radiation of the emitter further comprises one or more of:

a fluorescence source for emitting a fluorescence excitation wavelength of electromagnetic radiation for fluorescing a reagent; or

a hyperspectral source for emitting a hyperspectral wavelength of electromagnetic radiation for eliciting a spectral response from a tissue.

9. The system of claim 7 , wherein the plurality of sources of electromagnetic radiation of the emitter further comprises one or more of:

an electromagnetic source for emitting electromagnetic radiation having a wavelength from about 770 nm to about 790 nm;

an electromagnetic source for emitting electromagnetic radiation having a wavelength from about 795 nm to about 815 nm;

an electromagnetic source for emitting electromagnetic radiation having a wavelength from about 513 nm to about 545 nm;

an electromagnetic source for emitting electromagnetic radiation having a wavelength from about 565 nm to about 585 nm; or

an electromagnetic source for emitting electromagnetic radiation having a wavelength from about 900 nm to about 1000 nm.

10. The system of claim 1 , wherein:

the image sensor captures imaging data for generating a plurality of exposure frames, wherein each of the plurality of exposure frames corresponds with a pulse of electromagnetic radiation emitted by the emitter; and

the pixel array of the image sensor senses reflected electromagnetic radiation to generate the plurality of exposure frames during a readout period of the pixel array, wherein the readout period is a duration of time when active pixels in the pixel array are read.

11. The system of claim 1 , wherein the emitter is configured to emit a sequence of pulses of electromagnetic radiation repeatedly sufficient for generating a video stream comprising a plurality of image frames, wherein each image frame in the video stream comprises data from a plurality of exposure frames captured by the image sensor during a readout period of the pixel array.

12. The system of claim 11 , wherein each image frame comprises:

a red exposure frame captured by the image sensor in response to a red source of the emitter pulsing a red wavelength of electromagnetic radiation;

a green exposure frame captured by the image sensor in response to a green source of the emitter pulsing a green wavelength of electromagnetic radiation; and

a blue exposure frame captured by the image sensor in response to a blue source of the emitter pulsing a blue wavelength of electromagnetic radiation;

wherein the red exposure frame, the green exposure frame, and the blue exposure frame are combined to generate a color RGB image frame.

13. The system of claim 12 , wherein one or more of the image frames further comprises a mapping exposure frame overlaid thereon, wherein the mapping exposure frame is captured by the image sensor in response to the mapping source pulsing the mapping pattern, and wherein the mapping exposure frame comprises an indication of one or more of:

a three-dimensional topology of a scene;

a dimension of one or more objects within the scene;

a location of an object within the scene; or

a distance between two or more objects within the scene.

14. The system of claim 1 , further comprising an endoscope for imaging within a light deficient environment, wherein the emitter is the only source of electromagnetic radiation within the light deficient environment, and wherein the mapping data comprises information for calculating one or more of:

a distance from the endoscope to an object within the light deficient environment;

an angle from the endoscope to an object within the light deficient environment; or

a three-dimensional surface topology of an object within the light deficient environment.

15. The system of claim 1 , wherein the mapping data comprises information for calculating real-time measurements to an accuracy of less than one millimeter.

16. The system of claim 1 , wherein the mapping source comprises a plurality of laser mapping sources for emitting tool-specific laser mapping patterns for each of a plurality of tools within a scene.

17. The system of claim 1 , wherein the mapping pattern comprises a first output and a second output that are independent of one another, wherein the first output is emitted to illuminate a light deficient environment and the second output is for tracking a position of a tool within the light deficient environment, wherein the emitter is the only source of electromagnetic radiation within the light deficient environment.

18. The system of claim 1 , wherein the controller provides the mapping data to a third-party system for calculating a three-dimensional topology of a scene by way of light detection and ranging (LIDAR) mapping.

19. The system of claim 1 , wherein:

the controller receives one or more of fluorescence imaging data or hyperspectral imaging data from the image sensor;

the controller provides the mapping data and the one or more of the fluorescence imaging data or the hyperspectral imaging data to a third-party system trained to identify a tissue within the scene; and

the controller receives a dimension and a location of the tissue from the third-party system, wherein the third-party system calculates the dimension and the location of the tissue based on the mapping data and the one or more of the fluorescence imaging data or the hyperspectral imaging data.

20. The system of claim 1 , wherein the sensor data pad is a bidirectional pad, and wherein the bidirectional pad is in:

the output state when pixel data is output from the image sensor, wherein the pixel data is generated by exposing the pixel array; and

the input state when receiving control commands for the operation of the image sensor.

Assignments (3)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 6, 2021
From: ETHICON LLC
To: CILAG GMBH INTERNATIONAL
Reel/Frame 056983/0569 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 6, 2021
From: TALBERT, JOSHUA D.; WICHERN, DONALD M.
To: DEPUY SYNTHES PRODUCTS, INC.
Reel/Frame 056163/0232 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 6, 2021
From: DEPUY SYNTHES PRODUCTS, INC.
To: ETHICON LLC
Reel/Frame 056163/0272 →