Hyperspectral imaging in a light deficient environment

1. An endoscopic system for use in a light deficient environment for capturing color and spectral visualization data, the system comprising:

an endoscope comprising:

a tube;

an image sensor comprising a pixel array that detects electromagnetic radiation and reads out image data; and

a lens assembly comprising at least one optical element corresponding to the image sensor;

a light engine that cycles a plurality of sources of electromagnetic radiation on and off according to a pulse cycle to emit a plurality of pulses of electromagnetic radiation, wherein the plurality of sources comprises:

a first spectral source that emits electromagnetic radiation within a first spectral waveband; and

a second spectral source that emits electromagnetic radiation within a second spectral waveband;

wherein the first spectral waveband is different from the second spectral waveband; and

a controller in communication with the image sensor and the light engine, wherein the controller is configured to:

receive a user input requesting that an output video stream comprises color imaging data, dimensional information, and an overlay that highlights a target tissue;

identify two or more sources of the plurality of sources to be pulsed to generate the color imaging data and to identify the target tissue with an algorithm;

determine a recurring frequency for pulsing each of the two or more sources to generate the video stream according to the user input;

calculate the pulse cycle for the light engine based on the recurring frequency for pulsing each of the two or more sources; and

instruct the light engine to cycle the plurality of sources on and off according to the pulse cycle;

wherein the dimensional information comprises one or more of a dimension, surface area, or relative position of one or more of a surgical tool, anatomical structure, or object within a scene.

2. The endoscopic system of claim 1 , wherein the controller is configured to calculate the pulse cycle further based on an optimized a pulse duration for each of the two or more sources for capturing the color imaging data, the dimensional information, and for identifying the target tissue with the algorithm, wherein the pulse duration for each of the plurality of sources of the light engine is variable.

3. The endoscopic system of claim 1 , wherein the controller calculates the pulse cycle such that at least one source of the plurality of sources emits a plurality of sub-pulses of electromagnetic radiation during a single pulse duration corresponding with a single blanking period of the image sensor.

4. The endoscopic system of claim 3 , wherein two or more of the plurality of sub-pulses of electromagnetic radiation are emitted for a different duration.

5. The endoscopic system of claim 3 , wherein two or more of the plurality of sub-pulses of electromagnetic radiation are emitted for an equal duration; and

wherein the single pulse duration comprises two or more wavelengths of electromagnetic radiation emitted simultaneously.

6. The endoscopic system of claim 3 , wherein the light engine cycles the plurality of sources such that the single pulse duration comprises two or more different wavelengths of electromagnetic radiation and further such that two or more of the plurality of sub-pulses are emitted for a different duration.

7. The endoscopic system of claim 1 , wherein the two or more sources of electromagnetic radiation pulsed according to the pulse cycle pattern comprises:

a visible source that pulses electromagnetic radiation within a visible waveband, wherein pulses by the visible source correspond with the image sensor outputting the color imaging data; and

one or more of the first spectral source or the second spectral source, wherein pulses by the one or more of the first spectral source or the second spectral source correspond with the image sensor outputting data for identifying the target tissue with the algorithm.

8. The endoscopic system of claim 1 , wherein the controller receives color image data and spectral image data from the image sensor, and wherein the controller combines the color image data and the spectral image data to generate overlay image data, and wherein the spectral image data is sensed by the pixel array in response to the light engine pulsing one or more of:

the first spectral waveband;

the second spectral waveband; or

a third spectral waveband;

wherein at least one of the first spectral waveband, the second spectral waveband, or the third spectral waveband is selected for multispectral imaging.

9. The endoscopic system of claim 1 , wherein the controller is further configured to automatically adjust the plurality of pulses of electromagnetic radiation emitted according to the pulse cycle based on a threshold, wherein the threshold determines proper illumination for visualizing the scene with the image sensor.

10. The endoscopic system of claim 1 , wherein the pixel array detects the electromagnetic radiation and reads out the image data in response to each of a plurality of pulse durations, and wherein the light engine emits one or more pulses of electromagnetic radiation during each pulse duration of the plurality of pulse durations.

11. The endoscopic system of claim 10 , wherein the pixel array reads out the image data for generating a plurality of exposure frames, and wherein each of the plurality of exposure frames comprises to one pulse duration of the plurality of pulse durations.

12. The endoscopic system of claim 11 , wherein data from two or more of the plurality of exposure frames are displayed to a user as an overlay image, and wherein the overlay image comprises a depiction of the one or more of the dimension, surface area, or relative position of the one or more of the surgical tool, anatomical structure, or object within the scene.

13. The endoscopic system of claim 1 , wherein the pixel array detects the electromagnetic radiation and reads out the image data for generating an exposure frame in response to each pulse of the plurality of pulses of electromagnetic radiation emitted by the light engine, and wherein one or more exposure frames are displayed to a user as a single image on display as part of the video stream.

14. The endoscopic system of claim 1 , wherein images outputs according to the video stream are assigned a visible color for use on a display, and wherein the visible color is 8-bit or 16-bit or n-bit.

15. The endoscopic system of claim 1 , wherein the plurality of sources comprises a source tuned to emit only electromagnetic radiation within a waveband from about 425 nm to about 475 nm.

16. The endoscopic system of claim 1 , wherein the plurality of sources comprises a source tuned to emit only electromagnetic radiation within a waveband from about 520 nm to about 545 nm.

17. The endoscopic system of claim 1 , wherein the plurality of sources comprises a source tuned to emit only electromagnetic radiation within a waveband from about 625 nm to about 645 nm.

18. The endoscopic system of claim 1 , wherein the plurality of sources comprises a source tuned to emit only electromagnetic radiation within a waveband from about 760 nm to about 795 nm.

19. The endoscopic system of claim 1 , wherein the plurality of sources comprises a source tuned to emit only electromagnetic radiation within a waveband from about 795 nm to about 815 nm.

20. The endoscopic system of claim 1 , wherein the plurality of sources comprises a source tuned to emit only electromagnetic radiation within a waveband from about 370 nm to about 420 nm.

21. The endoscopic system of claim 1 , wherein the plurality of sources comprises a source tuned to emit only electromagnetic radiation within a waveband from about 600 nm to about 670 nm.

22. The endoscopic system of claim 1 , wherein the plurality of sources of the light engine comprises:

the first spectral source tuned to emit the first spectral waveband, wherein the first spectral waveband comprises electromagnetic radiation from about 513 nm to about 545 nm;

the second spectral source tuned to emit the second spectral waveband, wherein the second spectral waveband comprises electromagnetic radiation from about 565 nm to about 585 nm;

a third spectral source tuned to emit a third spectral waveband, wherein the third spectral waveband comprises electromagnetic radiation from about 900 nm to about 1000 nm;

a fourth spectral source tuned to emit a fourth spectral waveband, wherein the fourth spectral waveband comprises electromagnetic radiation from about 770 nm to about 795 nm; and

a fifth spectral source tuned to emit a fifth spectral waveband, wherein the fifth spectral waveband comprises electromagnetic radiation from about 790 nm to about 815 nm.

23. The endoscopic system of claim 1 , further comprising a polarization filter that is located in a path of the plurality of pulses of electromagnetic radiation emitted by the light engine.

24. The endoscopic system of claim 23 , further comprising a waveguide that transmits the plurality of pulses of electromagnetic radiation from the light engine to a distal end of the tube of the endoscope, and wherein the polarization filter is located at a proximal end of the waveguide.

25. The endoscopic system of claim 23 , further comprising a waveguide that transmits the plurality of pulses of electromagnetic radiation from the light engine to a distal end of the tube of the endoscope, and wherein the polarization filter is located at a distal end of the waveguide.

26. The endoscopic system of claim 1 , further comprising a lens assembly and an electromagnetic radiation filter.

27. The endoscopic system of claim 1 , wherein the plurality of sources of the light engine comprises a laser mapping source configured to emit light diffracted into a laser map pattern, and wherein the laser map pattern comprises one or more of vertical hashing, horizontal hashing, a raster grid of discrete points, an occupancy grid map, or a dot array.

28. The endoscopic system of claim 1 , wherein the image sensor reads out a plurality of exposure frames that each correspond in time with at least one pulse of the plurality of pulses of electromagnetic radiation emitted by the light engine; and

wherein at least a portion of the plurality of exposure frames is provided to the algorithm to identify the target tissue within the scene.

29. The endoscopic system of claim 1 , wherein the controller is further configured to:

provide imaging data to the algorithm to identify a location of the target tissue within the scene; and

render an overlay frame highlighting the target tissue based on an output received from the algorithm, wherein the highlight of the target tissue is encoded to any color selected by either an overlay algorithm or a user.

30. The endoscopic system of claim 1 , wherein at least one of the first spectral waveband or the second spectral waveband is a fluorescence excitation wavelength that causes a reagent to fluoresce, and wherein the endoscopic system further comprises a filter that prevents the fluorescence excitation wavelength from irradiating the pixel array.

31. The endoscopic system of claim 30 , wherein the filter is located on an optical element of a lens assembly such that the filter prevents the fluorescence excitation wavelength from irradiating the pixel array and allows a fluorescence relaxation wavelength of the reagent to irradiate the pixel array.

32. The endoscopic system of claim 1 , wherein the pixel array detects a fluorescence relaxation wavelength and outputs fluorescence imaging data, wherein the fluorescence imaging data comprises information for identifying one or more critical structures in a human body.

33. The endoscopic system of claim 32 , wherein the one or more critical structures in the human body comprises one of a nerve, a ureter, a blood vessel, an artery, a blood flow, or a tumor.

34. The endoscopic system of claim 32 , wherein the one or more critical structures in the human body are cancer cells, and wherein the fluorescence relaxation wavelength is released by a reagent comprising a fluorophore that fluoresces when exposed to electromagnetic radiation having a wavelength between 770 nm and 815 nm.

35. The endoscopic system of claim 1 , wherein the controller calculates the pulse cycle to comprise a fluorescence excitation emission that causes a reagent to fluoresce, and wherein the fluorescence excitation emission comprises electromagnetic radiation within a waveband from 770 nm to 815 nm.

36. The endoscopic system of claim 1 , further comprising a filter that prevents electromagnetic radiation within a waveband from 770 nm to 815 nm from irradiating the pixel array of the image sensor.

37. The endoscopic system of claim 1 , wherein the controller calculates the pulse cycle such that the plurality of pulses of electromagnetic radiation emitted by the light engine comprises a fluorescence excitation emission that causes a reagent to fluoresce, and wherein the fluorescence excitation emission comprises only electromagnetic radiation within a waveband from about 770 nm to about 815 nm.

38. The endoscopic system of claim 1 , further comprising a filter that prevents electromagnetic radiation within a waveband from about 770 nm to about 790 nm from irradiating the pixel array of the image sensor.

39. The endoscopic system of claim 1 , further comprising a filter that prevents electromagnetic radiation within a waveband from about 795 nm to about 815 nm from irradiating the pixel array of the image sensor.

40. The endoscopic system of claim 37 , further comprising:

a first filter that prevents electromagnetic radiation between 770 nm and 790 nm from irradiating the pixel array of the image sensor; and

a second filter that prevents electromagnetic radiation between 795 nm and 815 nm from irradiating the pixel array of the image sensor.

41. The endoscopic system of claim 37 , further comprising one or more filters that allow electromagnetic radiation between 790 nm and 800 nm and above 815 nm to pass through the one or more filters to the image sensor.

42. The endoscopic system of claim 1 , wherein the image sensor comprises a first image sensor and a second image sensor, and wherein the dimensional information is calculated based on data output by the first image sensor and the second image sensor to produce a three-dimensional image.

43. The endoscopic system of claim 42 , further comprising a filter that prevents the first image sensor from being irradiated by electromagnetic radiation having a wavelength less than 815 nm.

44. The endoscopic system of claim 42 , further comprising a filter that prevents the second image sensor from being irradiated by electromagnetic radiation outside a waveband from 785 nm to 800 nm.

45. The endoscopic system of claim 42 , further comprising:

one or more filters that prevent the first image sensor from being irradiated by electromagnetic radiation between 770 nm and 815 nm; and

one or more filters that prevent the second image sensor from being irradiated by electromagnetic radiation between 760 nm and 785 nm and electromagnetic radiation between 800 nm and 850 nm.

46. The endoscopic system of claim 1 , further comprising a first filter that filters electromagnetic radiation between 770 nm and 815 nm.

47. The endoscopic system of claim 46 , further comprising a second filter that filters electromagnetic radiation between 760 nm and 785 nm and electromagnetic radiation between 800 nm and 850 nm.

48. The endoscopic system of claim 1 , wherein the two or more sources of electromagnetic radiation pulsed according to the pulse cycle comprises:

a visible source that pulses electromagnetic radiation within a visible waveband, wherein pulses by the visible source correspond with the image sensor outputting the color imaging data; and

one or more of the first spectral source or the second spectral source, wherein pulses by the one or more of the first spectral source or the second spectral source corresponding with the image sensor outputting fluorescence imaging data for identifying the target tissue with the algorithm.

49. The endoscopic system of claim 1 , wherein the two or more sources of electromagnetic radiation pulsed according to the pulse cycle comprises:

a visible source that pulses electromagnetic radiation within a visible waveband, wherein pulses by the visible source correspond with the image sensor outputting the color imaging data; and

a laser mapping source that pulses electromagnetic radiation diffracted into a laser map pattern, wherein pulses by the laser mapping source correspond with the image sensor outputting data for calculating the dimensional information.

50. The endoscopic system of claim 1 , wherein the controller is further configured to:

calculate an operational cycle for the image sensor that corresponds in time with the pulse cycle of the light engine; and

instruct the image sensor to accumulate electromagnetic radiation and read out a plurality of exposure frames according to the operational cycle.

51. The endoscopic system of claim 50 , wherein the controller is further configured to:

receive an exposure frame output by the image sensor according to the operational cycle; and

assign a datatype to the exposure frame based on when the pixel array detected the electromagnetic radiation for the exposure frame and further based on the pulse cycle of the light engine.

52. The endoscopic system of claim 51 , wherein the controller is configured to assign a fluorescence datatype to the exposure frame in response to determining the pixel array detected the electromagnetic radiation when the light engine pulsed an excitation source of electromagnetic radiation.

53. The endoscopic system of claim 51 , wherein the controller is configured to assign a multispectral datatype to the exposure frame in response to determining the pixel array detected the electromagnetic radiation when the light engine pulsed one or more of the first spectral source or the second spectral source.

54. The endoscopic system of claim 51 , wherein the controller is configured to assign a laser mapping datatype to the exposure frame in response to determining the pixel array detected the electromagnetic radiation when the light engine pulsed electromagnetic radiation diffracted into a laser map pattern.

55. The endoscopic system of claim 51 , wherein the controller is configured to assign a color datatype to the exposure frame in response to determining the pixel array detected the electromagnetic radiation when the light engine pulsed white light.

56. The endoscopic system of claim 51 , wherein the pixel array comprises a color filter array, and wherein the image sensor outputs a plurality of exposure frames comprising a plurality of datatypes depending on which of the sources of electromagnetic radiation the light engine cycled on when the pixel array detected the electromagnetic radiation.

57. The endoscopic system of claim 56 , wherein the plurality of datatypes comprises:

a color datatype detected by the pixel array when the light engine cycled on a white light source;

a fluorescence datatype detected by the pixel array when the light engine cycled on an excitation source of electromagnetic radiation selected to fluoresce a tissue or reagent;

a multispectral datatype detected by the pixel array when the light engine cycled on one or more of the first spectral source or the second spectral source; and

a laser mapping datatype detected by the pixel array when the light engine cycled on electromagnetic radiation diffracted into a laser map pattern.