Granted Patent B2

US 12,740,739 · App. 18/893,673 · Granted Sep 22, 2026

Inventors: David W. Roberts (Lyme, NH); Keith D. Paulsen (Hanover, NH); Alexander Hartov (Enfield, NH); Songbai Ji (Hanover, NH); Xiaoyao Fan (Lebanon, NH)

Assignee: The Trustees of Dartmouth College

A61B5/377A61B5/0042A61B5/0077G06T3/14G06T7/33H04N13/239H04N13/257A61B5/055A61B5/369A61B2034/104A61B2034/105A61B2034/107A61B2090/367G06T2207/10012G06T2207/10064G06T2207/10072G06T2207/30016

Abstract

Claims (42)

1 . A system, comprising:

a stereo image capture device configured to capture image data of a surgical field;

a processor; and

memory storing an image enhancer module and a registration module, each as non-transitory computer readable instructions that when executed by the processor cause the system to:

receive a stereo image stream and a stereo hyperspectral image stream of the surgical field based on the image data,

generate, using the hyperspectral image stream, a three-dimensional (3D) hyperspectral model of the surgical field,

generate, using the stereo image stream, a 3D surface model of the surgical field,

co-register annotation data to one or both of the 3D surface model and the 3D hyperspectral model to yield an enhanced-data 3D model,

analyze intraoperative visual information based on one or both of the stereo image stream and the stereo hyperspectral image stream with a tissue classifier to yield a tissue classification data, and

output an enhanced image stream based on the stereo image stream enhanced with data from one or more of the 3D hyperspectral model, the 3D surface model, the enhanced-data 3D model, and the tissue classification data.

2 . The system of claim 1 , the stereo image capture device including first and second cameras for capturing the stereo image stream and independent first and second hyperspectral cameras for capturing the stereo hyperspectral image stream.

3 . The system of claim 1 , further comprising an image processing module, configured as further non-transitory computer readable instructions that, when executed by the processor, cause the system to recombine the stereo image stream from the hyperspectral image stream.

4 . The system of claim 1 , the image enhancer module further causing the system to generate a 3D fluorescence model from the hyperspectral data stream identifying depth in tissue of fluorophores in the surgical field.

5 . The system of claim 4 , the depth in tissue of fluorophores being identified based on intensity of observed fluorescent emissions in the hyperspectral image stream at two or more wavelengths.

6 . The system of claim 4 , the image enhancer module further causing the system to:

generate a plurality of model references to a patient-centered coordinate system including at least two of a hyperspectral-model reference of the 3D hyperspectral model, a surface-model reference of the 3D surface model, an enhanced-data-model reference of the enhanced-data 3D model, and a fluorescence-model reference of the 3D fluorescence model, align at least two of the 3D hyperspectral model, the 3D surface model, the enhanced-data 3D model, the fluorescence-model using the plurality of model references, and map the tissue classification data;

wherein output an enhanced data stream includes output the aligned at least two of the 3D hyperspectral model, the 3D surface model, the enhanced-data 3D model, the fluorescence-model, and the mapped tissue classification data.

7 . The system of claim 1 , the tissue classification data including tissue physiological properties.

8 . The system of claim 1 , wherein the annotating data includes one or more of previously-captured hyperspectral stereo image streams, previously-captured fluorescent image streams, or determined 3D models from previously-captured hyperspectral stereo image streams or previously-captured fluorescent image streams.

9 . The system of claim 1 , wherein the annotating data includes previously-recorded data including one or more of (a) preoperative, intraoperative and postoperative radiological studies selected from the group including MRI, fMRI, CT, SPECT, and PET, (b) preoperative, intraoperative and postoperative physiological studies selected from the group including EEG, evoked potentials, and magnetoencephalography (MEG).

10 . The system of claim 1 , further comprising a display configured to display the enhanced image stream, or the memory storing the enhanced image stream.

11 . The system of claim 1 , the image enhancer module further causing the system to output the tissue classification data.

12 . A method, comprising:

receiving a stereo image stream and a stereo hyperspectral image stream of a surgical field;

generating, using the hyperspectral image stream, a three-dimensional (3D) hyperspectral model of the surgical field;

generating, using the stereo image stream, a 3D surface model of the surgical field;

co-registering annotation data to one or both of the 3D surface model and the 3D hyperspectral model to yield an enhanced-data 3D model;

analyzing intraoperative visual information based on one or both of the stereo image stream and the stereo hyperspectral image stream with a tissue classifier to yield a tissue classification data;

outputting an enhanced image stream based on the stereo image stream enhanced with data from one or more of the 3D hyperspectral model, the 3D surface model, the enhanced-data 3D model, and the tissue classification data.

13 . The method of claim 12 , wherein the stereo image stream and the stereo hyperspectral image stream are generated independently.

14 . The method of claim 12 , further comprising recombining the stereo image stream from the hyperspectral image stream.

15 . The method of claim 12 , further comprising generating a 3D fluorescence model from the hyperspectral data stream identifying depth in tissue of fluorophores in the surgical field.

16 . The method of claim 15 , the depth in tissue of fluorophores being identified based on intensity of observed fluorescent emissions in the hyperspectral image stream at two or more wavelengths.

17 . The method of claim 15 , further comprising:

generating a plurality of model references to a patient-centered coordinate system including at least two of a hyperspectral-model reference of the 3D hyperspectral model, a surface-model reference of the 3D surface model, an enhanced-data-model reference of the enhanced-data 3D model, and a fluorescence-model reference of the 3D fluorescence model;

aligning at least two of the 3D hyperspectral model, the 3D surface model, the enhanced-data 3D model, the fluorescence-model using the plurality of model references; and

mapping the tissue classification data;

wherein outputting an enhanced data stream includes outputting the aligned at least two of the 3D hyperspectral model, the 3D surface model, the enhanced-data 3D model, the fluorescence-model, and the mapped tissue classification data.

18 . The method of claim 12 , the tissue classification data including tissue physiological properties.

19 . The method of claim 12 , wherein the annotating data includes one or more of previously-captured hyperspectral stereo image streams, previously-captured fluorescent image streams, or determined 3D models from previously-captured hyperspectral stereo image streams or previously-captured fluorescent image streams.

20 . The method of claim 12 , wherein the annotating data includes previously-recorded data including one or more of (a) preoperative, intraoperative and postoperative radiological studies selected from the group including MRI, fMRI, CT, SPECT, and PET, (b) preoperative, intraoperative and postoperative physiological studies selected from the group including EEG, evoked potentials, and magnetoencephalography (MEG).

21 . The method of claim 12 , wherein the outputting includes one or both of displaying the enhanced data stream on a display or storing the enhanced data stream in memory.

Assignments (1)

Continuity (18)

References Cited (198)