Videostroboscopy of vocal cords with a hyperspectral, fluorescence, and laser mapping imaging system
Videostroboscopy with a hyperspectral, fluorescence, and laser mapping image system is described. A system includes an emitter for emitting pulses of electromagnetic radiation and an image sensor comprising a pixel array for sensing reflected electromagnetic radiation. The system includes a controller configured to cause the emitter to emit the pulses of electromagnetic radiation at a strobing frequency determined based on a vibration frequency of vocal cords of a user. The system is such that at least a portion of the pulses of electromagnetic radiation emitted by the emitter comprises one or more of a hyperspectral emission, a fluorescence emission, and/or a laser mapping pattern.
1. A system for medical imaging, the system comprising:
an emitter for emitting a plurality of pulses of electromagnetic radiation;
an image sensor comprising a pixel array for sensing reflected electromagnetic radiation; and
a controller in electronic communication with the emitter and the image sensor, wherein the controller is configured to cause the emitter to emit the plurality of pulses of electromagnetic radiation at a strobing frequency;
wherein the strobing frequency is determined based on a vibration frequency of vocal cords of a patient;
wherein the image sensor is configured to sense the reflected electromagnetic radiation from the plurality of pulses of electromagnetic radiation to generate a plurality of exposure frames;
wherein at least a portion of the plurality of pulses of electromagnetic radiation emitted by the emitter comprises a visible emission and two or more specialty emissions comprising:
a hyperspectral emission comprising one or more wavelengths of electromagnetic radiation within a waveband selected to elicit a spectral response from a tissue structure,
a fluorescence excitation emission of electromagnetic radiation, or
a laser mapping emission;
wherein the plurality of pulses of electromagnetic radiation are emitted in a pulse pattern comprising the visible emission and the two or more specialty emissions;
wherein at least a portion of the plurality of exposure frames comprises a visible exposure frame and two or more specialty exposure frames comprising:
a hyperspectral exposure frame corresponding to the hyperspectral emission,
a fluorescence exposure frame corresponding to the fluorescence excitation emission, or
a laser mapping frame corresponding to the laser mapping emission.
2. The system of claim 1 , wherein the controller is further configured to continuously determine the vibration frequency of the vocal cords of the patient and adjust the strobing frequency of the emitter in accordance with the vibration frequency of the vocal cords.
3. The system of claim 1 , wherein the controller comprises one or more processors for executing instructions stored in non-transitory computer readable storage medium, the instructions comprising:
determining the vibration frequency of the vocal cords of the patient;
calculating the strobing frequency based on the vibration frequency;
causing the emitter to emit the pulses of electromagnetic radiation at the strobing frequency; and
suppressing at least a portion of the pulses of electromagnetic radiation emitted by the emitter.
4. The system of claim 3 , wherein the instructions are such that suppressing at least the portion of the pulses of electromagnetic radiation emitted by the emitter comprises one or more of:
suppressing a pulse of electromagnetic radiation emitted during a pixel readout portion of a readout period of the image sensor; and
suppressing a pulse of electromagnetic radiation emitted during a blanking portion of the readout period of the image sensor;
wherein the pulses of electromagnetic radiation are suppressed such that an aggregate amount of illumination which is the sum of the amplitude and the duration of all non-suppressed pulses for a first blanking period of the image sensor is equal to the aggregate amount of illumination in a successive blanking period of the image sensor.
5. The system of claim 1 , further comprising a microphone for sensing sound emitted by the patient such that the vibration frequency of the vocal cords of the patient can be calculated based on the sound.
6. The system of claim 1 , wherein the controller is configured to suspend a readout period of the image sensor for a suspended period and cause the emitter to emit one or more pulses of electromagnetic radiation during the suspended period.
7. The system of claim 1 , wherein two or more partial exposure frames captured by the image sensor are combined to generate a combined exposure frame, wherein the combined exposure frame comprises data for a single wavelength of electromagnetic radiation.
8. The system of claim 7 , wherein the single wavelength of electromagnetic radiation comprises one or more of:
a red wavelength of visible light;
a green wavelength of visible light;
a blue wavelength of visible light;
the hyperspectral emission;
the fluorescence excitation emission; or
the laser mapping emission.
9. The system of claim 7 , wherein the combined exposure frame is processed to be substantially artifact free by normalizing the combined exposure frame using digital gain.
10. The system of claim 1 , wherein two or more exposure frames captured by the pixel array of the image sensor are combined to generate a combined exposure frame representing pixel data responsive to a single wavelength of electromagnetic radiation pulsed by the emitter, wherein the combined exposure frame is processed to generate a normalized exposure frame by normalizing using digital gain.
11. The system of claim 1 , wherein each of the plurality of exposure frames corresponds to one or more pulses of electromagnetic radiation emitted by the emitter.
12. The system of claim 11 , wherein the pixel array of the image sensor senses reflected electromagnetic radiation to generate the plurality of exposure frames during a readout period of the pixel array, wherein the readout period comprises a duration of time when active pixels in the pixel array are read.
13. The system of claim 1 , wherein the emitter is configured to emit, during a pulse duration, a plurality of sub-pulses of electromagnetic radiation having a sub-duration shorter than the pulse duration.
14. The system of claim 1 , wherein one or more of the pulses of electromagnetic radiation emitted by the emitter comprises electromagnetic radiation emitted at two or more wavelengths simultaneously as a single pulse or a single sub-pulse.
15. The system of claim 1 , wherein at least a portion of the pulses of electromagnetic radiation emitted by the emitter is the fluorescence excitation emission that results in the fluorescence exposure frame created by the image sensor, and wherein the controller is configured to provide the fluorescence exposure frame to a corresponding system that analyzes the florescence exposure frame to determine tissue information comprising one or more of:
a location of the tissue structure within a scene,
a disease within the tissue structure, or
a condition of the tissue structure.
16. The system of claim 15 , wherein the fluorescence excitation emission comprises one or more of:
electromagnetic radiation having a wavelength from about 770 nm to about 795 nm; or
electromagnetic radiation having the wavelength from about 790 nm to about 815 nm.
17. The system of claim 16 , wherein the controller is further configured to:
receive the tissue information from the corresponding system;
generate an overlay frame comprising the tissue information;
generate a color image frame from the visible exposure frame; and
combine the overlay frame with the color image frame depicting the scene to visualize the tissue information within the scene.
18. The system of claim 17 , wherein sensing the reflected electromagnetic radiation by the pixel array comprises generating the laser mapping exposure frame by sensing reflected electromagnetic radiation resulting from the emitter pulsing the laser mapping emission, and wherein the controller is further configured to:
provide the laser mapping exposure frame to the corresponding system that determines one or more structural characteristics of one or more objects within the scene, wherein the one or more objects comprises one or more of the tissue structure, one or more tissue structures, or one or more tools;
provide a location of the one or more or objects to the corresponding system; and
receive the one or more structural characteristics of the one or more objects from the corresponding system;
wherein the structural characteristics comprise one or more of:
one or more distances between the one or more objects within the scene,
a surface topology of the one or more objects within the scene,
one or more dimensions of the one or more objects within the scene, or
position information of the one or more objects within the scene.
19. The system of claim 18 , wherein the tissue structure or the one or more tissue structures comprises one or more of a nerve, a ureter, a blood vessel, an artery, a blood flow, or a tumor.
20. The system of claim 1 , wherein the controller is configured to synchronize timing of the pulses of electromagnetic radiation during a blanking period of the image sensor,
wherein the blanking period corresponds to a time between a readout of a last row of active pixels in the pixel array and a beginning of a next subsequent readout of active pixels in the pixel array.
21. The system of claim 1 , wherein two or more pulses of electromagnetic radiation emitted by the emitter result in two or more instances of reflected electromagnetic radiation that are sensed by the pixel array to generate two or more exposure frames that are combined to form an image frame.
22. The system of claim 1 , wherein the emitter is configured to emit a sequence of pulses of electromagnetic radiation repeatedly sufficient for generating a video stream comprising a plurality of image frames, wherein each image frame in the video stream comprises data from a plurality of exposure frames, and wherein each of the exposure frames corresponds to one or more pulses of electromagnetic radiation.
23. The system of claim 1 , wherein the emitter repeats the pattern of varying wavelengths of electromagnetic radiation.
24. The system of claim 1 , wherein sensing reflected electromagnetic radiation by the pixel array comprises generating the laser mapping exposure frame by sensing reflected electromagnetic radiation resulting from the emitter pulsing the laser mapping emission, wherein the laser mapping exposure frame comprises information for determining real time measurements of an object comprising one or more of a tissue structure or tool, wherein the real time measurements comprise one or more of:
a distance from an endoscope to the object;
an angle between an endoscope and the object; or
surface topology information about the object.
25. The system of claim 1 , wherein at least a portion of the pulses of electromagnetic radiation emitted by the emitter comprises a plurality of tool-specific laser mapping emissions for each of a plurality of tools within a scene.
26. The system of claim 1 , wherein the laser mapping emission emitted by the emitter comprises a first output and a second output that are independent from one another, wherein the first output is for light illumination and the second output is for tool tracking.
27. The system of claim 1 , wherein at least a portion of the pulses of electromagnetic radiation emitted by the emitter is the hyperspectral emission that results in the hyperspectral exposure frame created by the image sensor, and wherein the controller is configured to provide the hyperspectral exposure frame to a corresponding system that analyzes the hyperspectral exposure frame to determine a classification of the target within a scene based on the hyperspectral exposure frame.
28. The system of claim 27 , wherein the hyperspectral emission comprises:
electromagnetic radiation within a waveband from about 513 nm to about 545 nm and electromagnetic radiation within a waveband from about 900 nm to about 1000 nm; or
electromagnetic radiation within a waveband from about 565 nm to about 585 nm and electromagnetic radiation within a waveband from about 900 nm to about 1000 nm.
29. The system of claim 28 , wherein the controller is further configured to:
receive the classification of the target from the corresponding system;
generate an overlay frame comprising the classification of the target;
generate a color image frame from the visible exposure frame; and
combine the overlay frame with the color image frame depicting the scene to visualize the classification of the target within the scene.
30. The system of claim 29 , wherein sensing the reflected electromagnetic radiation by the pixel array comprises generating the laser mapping exposure frame by sensing reflected electromagnetic radiation resulting from the emitter pulsing the laser mapping emission, and wherein the controller is further configured to:
provide the laser mapping exposure frame to the corresponding system that determines one or more structural characteristics of one or more objects within the scene, wherein the one or more objects comprises one or more of the tissue structure, one or more tissue structures, or one or more tools;
provide a location of the one or more objects to the corresponding system; and
receive the one or more structural characteristics of the one or more objects from the corresponding system;
wherein the structural characteristics comprise one or more of:
one or more distances between the one or more objects within the scene,
a surface topology of the one or more objects within the scene,
one or more dimensions of the one or more objects within the scene, or
position information of the one or more objects within the scene.
31. The system of claim 30 , wherein the one or more objects comprises the tissue structure comprising one or more of a nerve, a ureter, a blood vessel, an artery, a blood flow, or a tumor.
32. A method comprising:
determining a vibration frequency of vocal cords of a patient;
pulsing a plurality of pulses of electromagnetic radiation by an emitter at a strobing frequency that substantially matches the vibration frequency of the vocal cords;
generating a plurality of exposure frames captured by a pixel array of an image sensor,
wherein the pixel array is configured to sense reflected electromagnetic radiation from the plurality of pulses of electromagnetic radiation;
wherein at least a portion of the plurality of pulses of electromagnetic radiation emitted by the emitter comprises a visible emission and two or more specialty emissions comprising:
a hyperspectral emission comprising one or more wavelengths of electromagnetic radiation within a waveband selected to elicit a spectral response from a tissue, wherein the waveband comprises one or more of:
electromagnetic radiation having a wavelength range from about 513 nm to about 545 nm,
electromagnetic radiation having a wavelength range from about 565 nm to about 585 nm, or
electromagnetic radiation having a wavelength range from about 900 nm to about 1000 nm,
a fluorescence excitation emission of electromagnetic radiation; or
a laser mapping emission;
wherein the plurality of pulses of electromagnetic radiation are emitted in a pulse pattern comprising the visible emission and the two or more specialty emissions;
wherein at least a portion of the plurality of exposure frames comprises a visible exposure frame and two or more specialty exposure frames comprising:
a hyperspectral exposure frame corresponding to the hyperspectral emission,
a fluorescence exposure frame corresponding to the fluorescence excitation emission, or
a laser mapping frame corresponding to the laser mapping emission.
33. The method of claim 32 , further comprising:
combining two or more exposure frames of the plurality of exposure frames to generate a combined exposure frame; and
processing the combined exposure frame to generate a normalized exposure frame by normalizing the combined exposure frame using digital gain;
wherein the two or more exposure frames are sensed by the pixel array in response to a single wavelength of electromagnetic radiation pulsed by the emitter.
34. The method of claim 32 , further comprising:
suppressing a pulse of electromagnetic radiation emitted during a pixel readout portion of a readout period of the image sensor; and
suppressing a pulse of electromagnetic radiation emitted during a blanking portion of the readout period of the image sensor;
wherein the pulses of electromagnetic radiation are suppressed such that an aggregate amount of illumination which is the sum of the amplitude and the duration of all non-suppressed pulses for a first blanking period of the image sensor is equal to the aggregate amount of illumination in a successive blanking period of the image sensor.
35. The method of claim 32 , further comprising suspending a readout period of the image sensor for a suspended period and causing the emitter to emit one or more pulses of electromagnetic radiation during the suspended period.