Image rotation in an endoscopic hyperspectral, fluorescence, and laser mapping imaging system

1. A system comprising:

an endoscope;

a handpiece unit;

an emitter for emitting pulses of electromagnetic radiation, comprising a plurality of electromagnetic sources, the sources comprising:

a visible source for pulsing a visible wavelength of electromagnetic radiation, and

a specialty source comprising a mapping source and one or more of:

a fluorescence source for pulsing a fluorescence excitation wavelength of electromagnetic radiation, or

a spectral source for pulsing electromagnetic radiation within a waveband comprising one or more wavelengths selected to elicit a spectral response from a tissue;

one or more image sensors each comprising a pixel array for sensing electromagnetic radiation; and

a controller in electronic communication with the emitter and the one or more image sensors;

a rotation sensor for sensing rotation of the handpiece unit and/or the endoscope;

wherein the one or more image sensors sense reflected electromagnetic radiation in response to pulses of reflected electromagnetic radiation from the emitter to generate a plurality of exposure frames, and wherein the plurality of exposure frames comprises a mapping exposure frame and one or more of a spectral exposure frame or a fluorescence exposure frame.

2. The system of claim 1 , wherein the mapping exposure frame comprises data for calculating one or more of a topography of a scene, a dimension of one or more objects within the scene, a location of one or more tools within the scene, or a distance between the one or more objects and the endoscope.

3. The system of claim 1 , wherein the spectral exposure frame is sensed in response to a multispectral emission pulsed by the spectral source, and wherein the multispectral emission comprises electromagnetic radiation comprising a wavelength within a multispectral range, wherein the multispectral range comprises one or more of:

wavelengths from about 510 nm to about 590 nm; or

wavelengths from about 900 nm to about 1000 nm.

4. The system of claim 1 , wherein the fluorescence exposure frame is sensed in response to a fluorescence excitation emission pulsed by the fluorescence source, and wherein the fluorescence excitation emission causes a reagent to fluoresce.

5. The system of claim 4 , wherein the fluorescence excitation emission comprises electromagnetic radiation comprising a wavelength within a fluorescence range, wherein the fluorescence range comprises one or more of:

wavelengths from about 770 nm to about 795 nm; or

wavelengths from about 790 nm to about 815 nm.

6. The system of claim 1 , wherein:

the controller synchronizes timing of the emitter and the one or more image sensors such that the one or more image sensors sense the plurality of frames in response to the emitter pulsing the plurality of emissions of electromagnetic radiation; and

wherein the controller is configured to execute instructions, wherein the instructions comprise providing the mapping exposure frame to a corresponding laser mapping system configured to calculate one or more of a topography of a scene, a dimension of one or more objects within the scene, a location of one or more tools within the scene, or a distance between the one or more objects and the endoscope based on the mapping exposure frame.

7. The system of claim 6 , wherein the instructions executed by the controller further comprise one or more of:

providing the spectral exposure frame to a corresponding multispectral system configured to identify one or more tissue structures within the scene based on data from the multispectral frame; or

providing the fluorescence exposure frame to a corresponding fluorescence system configured to identify a location of a reagent within the scene based on data from the fluorescence frame.

8. The system of claim 7 , wherein the plurality of frames sensed by the one or more image sensors further comprises a color image frame sensed in response to an emission of visible electromagnetic radiation, and wherein the instructions executed by the controller further comprise:

generating an overlay frame comprising the color image frame, data extracted from the laser mapping frame, and one or more of:

an indication of the one or more tissue structures within the scene as determined based on the multispectral frame; or

an indication of the location of the reagent within the scene as determined based on the fluorescence frame; and

providing the overlay frame to a display for real-time visualization of the scene.

9. The system of claim 1 , wherein the rotation sensor comprises one or more of a rotation-detecting Hall-effect sensor, a diametrically-polarized magnetic annulus, a gyroscope, or a potentiometer.

10. The system of claim 1 , further comprising one or more processors for executing an image signal processing pipeline comprising instructions for generating output image frames, wherein the instructions comprise:

receiving an angle of rotation for rotating at least one frame of the plurality of frames to maintain a constant image horizon, wherein the angle of rotation is determined based on output from the rotation sensor;

identifying integer coordinates for pixel data in the at least one frame of the plurality of frames;

applying a rotation kernel to the integer coordinates to transform the integer coordinates to real-number pixel coordinates.

11. The system of claim 10 , wherein the instructions for the image signal processing pipeline further comprise:

identifying one or more pixels within the at least one frame of the plurality of frames that is void after applying the rotation kernel; and

filling the void one or more pixels by applying nearest neighbor substitution, bilinear interpolation, or bicubic interpolation.

12. The system of claim 1 , further comprising one or more processors for executing an image signal processing pipeline comprising instructions for generating output image frames, wherein the instructions comprise:

receiving an angle of rotation from the rotation sensor, wherein the rotation sensor senses the angle of rotation of the handpiece unit relative to the endoscope;

determining whether to apply image transformation to maintain a constant image horizon based on whether the same angle of rotation is sensed by the rotation sensor across a threshold number of frames; and

in response to the same angle of rotation being sensed across the threshold number of frames, applying the image transformation to at least one frame of the plurality of frames to maintain the constant image horizon.

13. The system of claim 1 , wherein the rotation sensor senses an angle of rotation of the handpiece unit relative to the endoscope, and wherein the one or more image sensors are disposed at a distal end of the endoscope.

14. The system of claim 1 ,

wherein the visible source comprises one or more of a white source for emitting white electromagnetic radiation, a red source for emitting red electromagnetic radiation, a blue source for emitting blue electromagnetic radiation, or a green source for emitting green electromagnetic radiation; and

wherein the mapping source emits one or more of vertical hashing, horizontal hashing, a raster grid of discrete points, an occupancy grid map, or a dot array.

15. The system of claim 14 , wherein:

the spectral source comprises one or more multispectral sources for emitting electromagnetic radiation comprising a wavelength within a multispectral range, wherein the multispectral range comprises one or more of:

wavelengths from about 510 nm to about 590 nm; or

wavelengths from about 900 nm to about 1000 nm; and

the fluorescence sources comprises one or more fluorescence sources for emitting electromagnetic radiation comprising the fluorescence excitation wavelength within a fluorescence range, wherein the fluorescence range comprises one or more of:

wavelengths from about 770 nm to about 795 nm; or

wavelengths from about 790 nm to about 815 nm.

16. The system of claim 14 , wherein the spectral source further comprises:

a first multispectral source for emitting electromagnetic radiation comprising a wavelength within a range from 495 nm to 565 nm;

a second multispectral source for emitting electromagnetic radiation comprising a wavelength within a range from 555 nm to 595 nm; and

a third multispectral source for emitting electromagnetic radiation comprising a wavelength within a range from 900 nm to 1000 nm.

17. The system of claim 14 , wherein the fluorescence source further comprises:

a first fluorescence source for emitting electromagnetic radiation comprising a wavelength within a range from 770 nm to 795 nm; and

a second fluorescence source for emitting electromagnetic radiation comprising a wavelength within a range from 790 nm to 815 nm.

18. The system of claim 1 , wherein the spectral exposure frame comprises data for identifying a spectral response of a tissue structure, wherein the tissue structure comprises one or more of a nerve, a ureter, an artery, a blood vessel, or a tumor.

19. The system of claim 1 , further comprising:

one or more filters for filtering the fluorescence excitation wavelength to prevent the one or more image sensors from sensing the fluorescence excitation wavelength;

wherein the one or more image sensors sense a fluorescence relaxation wavelength of the tissue structure and/or the reagent to generate the fluorescence frame.

20. The system of claim 1 , further comprising:

an image signal processing pipeline for performing rotation transformations on exposure frames captured by the image sensor based on the angle of rotation of a lumen of the endoscope relative to the handpiece of the endoscope;

wherein the image signal processing pipeline rotates one or more of the plurality of exposure frames counter to the angle of rotation to maintain a consistent image horizon of a scene being imaged by the one or more image sensors.