System and method of otoscopy image analysis to diagnose ear pathology
Disclosed herein are systems and methods to detect a wide range of eardrum abnormalities by using high-resolution otoscope images and report the condition of the eardrum as “normal” or “abnormal.”
1. A method of classifying tympanic membrane pathologies, comprising:
performing preprocessing on a plurality of images of a tympanic membrane to produce one or more composite images, wherein regions of interest are selected from each of the plurality of images or a subset of the plurality of images and the selected regions of interest are used to form the one or more composite images;
extracting computer vision features (CVF) and clinically motivated eardrum features (CMEF) from the one or more composite images; and
classifying pathologies of the tympanic membrane using the CVF and CMEF information, wherein classifying pathologies of the tympanic membrane using the CVF and CMEF information comprises classifying pathologies of the tympanic membrane using the CVF and CMEF information fused by a two-layered stacked generalization algorithm that focuses on complementary features instead of strong features.
2. The method of claim 1 , wherein the plurality of images are captured using an image capture device.
3. The method of claim 2 , wherein the image capture device comprises a video image capture device and each of the plurality of images are frames captured from a video of the tympanic membrane.
4. The method of claim 1 , wherein preprocessing comprises detecting obstructions in and/or illumination of each of the images that comprise the plurality of images.
5. The method of claim 4 , wherein the detected obstructions and/or illumination are used in selecting the regions of interest.
6. The method of claim 5 , wherein the regions of interest selected from each of the plurality of images or the subset of the plurality of images are free from the obstructions such that the one or more produced composite images do not include the obstructions.
7. The method of claim 4 , wherein the obstructions comprise one or more of blurring, wax, glare, and hair in the video frames.
8. The method of claim 1 , wherein the two-layered stacked generalization algorithm comprises using a Fuzzy Stacked Generalization (FSG) classifier.
9. The method of claim 1 , wherein the preprocessing on the plurality of images of a tympanic membrane to produce one or more composite images includes one or more of reducing sensor based problems, selecting the regions of interest in the video frames, detecting glare effects, and creating a copy of the video frames with reduced glare effects.
10. The method of claim 1 , wherein extracting the CVF from the one or more composite images comprises using one or more of visual MPEG-7 descriptors, Histogram of Gradient, and Grid Color Moment features to extract color, texture and shape information from the one or more composite images.
11. The method of claim 1 , wherein the CMEF identify clues for abnormalities and normality of the tympanic membrane from the one or more composite images.
12. The method of claim 1 , wherein CMEF comprises a location of a cone of light, a visibility of malleus, a protrusion of membrane, an existence of tympanostomy tube, or an existence of wax.
13. The method of claim 1 , wherein classifying pathologies of the tympanic membrane using the using the CVF and CMEF information comprises using deep learning for the automated identification of abnormalities.
14. The method of claim 13 , wherein the deep learning comprises deep learning networks including Inception V3 or ResNet.
15. The method of claim 13 , wherein the abnormalities include one or more of acute otitis media (AOM), middle ear effusion (non-infected fluid), cholesteatoma (a common destructive skin cyst in the ear), eardrum perforation, and eardrum retraction vs normal.
16. The method of claim 1 , wherein classifying pathologies of the tympanic membrane using the using the CVF and CMEF information comprises using content-based image retrieval (CBIR) to compare the using the CVF and CMEF information to a library of images to identify abnormalities.
17. The method of claim 16 , wherein the abnormalities include one or more of acute otitis media (AOM), middle ear effusion (non-infected fluid), cholesteatoma (a common destructive skin cyst in the ear), eardrum perforation, and eardrum retraction vs normal.
18. A method of classifying tympanic membrane pathologies, comprising:
performing preprocessing on a plurality of images of a tympanic membrane to produce one or more composite images, wherein regions of interest are selected from each of the plurality of images or a subset of the plurality of images and the selected regions of interest are used to form the one or more composite images;
extracting computer vision features (CVF) and clinically motivated eardrum features (CMEF) from the one or more composite images; and
classifying pathologies of the tympanic membrane using the CVF and CMEF information, wherein an image entropy is calculated and the images with highest entropy values are selected from the plurality of images to produce the one or more composite images.
19. A system for classifying tympanic membrane pathologies, comprising:
a memory; and
a processor in communication with the memory, wherein the memory contains computer-executable instructions that when executed by the processor cause the processor to:
perform preprocessing on a plurality of images of a tympanic membrane to produce one or more composite images, wherein regions of interest are selected from each of the plurality of images or a subset of the plurality of images and the selected regions of interest are used to form the one or more composite images;
extract computer vision features (CVF) and clinically motivated eardrum features (CMEF) from the one or more composite images; and
classify pathologies of the tympanic membrane using the CVF and CMEF information, wherein classifying pathologies of the tympanic membrane using the CVF and CMEF information comprises classifying pathologies of the tympanic membrane using the CVF and CMEF information fused by a two-layered stacked generalization algorithm that focuses on complementary features instead of strong features.
20. A computer program product with computer-executable instructions embodied on a non-transitory computer readable medium, wherein the computer-executable instructions when executed by a computer perform a process of classifying tympanic membrane pathologies comprising:
performing preprocessing on a plurality of images of a tympanic membrane to produce one or more composite images, wherein regions of interest are selected from each of the plurality of images or a subset of the plurality of images and the selected regions of interest are used to form the one or more composite images;
extracting computer vision features (CVF) and clinically motivated eardrum features (CMEF) from the one or more composite images; and
classifying pathologies of the tympanic membrane using the CVF and CMEF information, wherein classifying pathologies of the tympanic membrane using the CVF and CMEF information comprises classifying pathologies of the tympanic membrane using the CVF and CMEF information fused by a two-layered stacked generalization algorithm that focuses on complementary features instead of strong features.
21. A system for classifying tympanic membrane pathologies, comprising:
a memory; and
a processor in communication with the memory, wherein the memory contains computer-executable instructions that hen executed by the processor cause the processor to:
perform preprocessing on a plurality of images of a tympanic membrane to produce one or more composite images, wherein regions of interest are selected from each of the plurality of images or a subset of the plurality of images and the selected regions of interest are used to form the one or more composite images;
extract computer vision features (CVF) and clinically motivated eardrum features (CMEF) from the one or more composite images; and
classify pathologies of the tympanic membrane using the CVF and CMEF information, wherein an image entropy is calculated and the images with highest entropy values are selected from the plurality of images to produce the one or more composite images.
22. A computer program product with computer-executable instructions embodied on a non-transitory computer readable medium, wherein the computer-executable instructions when executed by a computer perform a process of classifying tympanic membrane pathologies comprising:
performing preprocessing on a plurality of images of a tympanic membrane to produce one or more composite images, wherein regions of interest are selected from each of the plurality of images or a subset of the plurality of images and the selected regions of interest are used to form the one or more composite images;
extracting computer vision features (CVF) and clinically motivated eardrum features (CMEF) from the one or more composite images; and
classifying pathologies of the tympanic membrane using the CVF and CMEF information, wherein an image entropy is calculated and the images with highest entropy values are selected from the plurality of images to produce the one or more composite images.