Machine learning enabled system for skin abnormality interventions
Systems and methods for use in classifying and measuring images of skin abnormalities. The systems and methods can be used in an AR enabled system that may be used to assist in skin surgeries and skin abnormality triaging and diagnosis. The system uses a convolutional neural network to classify a skin abnormality in an initial image. The CNN may also be used in determining the boundaries of the skin abnormality. A fiducial marker may be present in the initial image and this marker may be used in automatically measuring the size of the skin abnormality. An adjusted image is generated based on the measured abnormality and this adjusted image can be used as an overlay in an AR enabled system for use in assisting surgical procedures.
1 . A system for use with digital images, the system comprising:
a digital imaging device on a mobile computing device configured to produce an initial digital image of a basal cell carcinoma (BCC) or a squamous cell carcinoma (SCC) of a patient;
a processor configured to receive the initial digital image and to receive at least one adjusted image;
a data communication path configured to send the initial digital image to at least two processing units, including:
a first processing unit comprising at least one convolutional neural network (CNN) configured to classify the initial digital image and to generate a first adjusted image by creating a composite Boundary-Attention Map (BAM) whose mean difference from a boundary map is less than 0.05; and
a second processing unit configured to measure three-dimensional (3D) features in the initial digital image using a 3D fiducial marker and to generate a second adjusted image; and
an overlay module configured to overlay at least one of the first adjusted image and the second adjusted image over a working image of the same carcinoma on the mobile computing device as an augmented-reality (AR) overlay.
2 . The system according to claim 1 , wherein the 3D fiducial marker has a diameter less than one inch and a height less than 0.2 inch.
3 . The system according to claim 1 , wherein the overlay module presents gridlines and measured dimensions derived from the 3D fiducial marker.
4 . The system according to claim 1 , wherein the first processing unit's CNN comprises a stem block with variable skipped connections, a building module, a final module, and a dense layer.
5 . The system according to claim 4 , wherein attention pixels in the initial digital image are captured by the final module and combined with boundary maps extracted from the stem block to create the BAM.
6 . The system according to claim 1 , wherein the AR overlay is displayed in real-time or near-real-time on the mobile device while a clinician marks excision margins.
7 . The system according to claim 1 ,
wherein the at least two processing units are remote from the mobile device.
8 . A method for processing an initial digital image of a skin cancer comprising a basal cell carcinoma (BCC) or a squamous cell carcinoma (SCC), the method comprising:
by way of a data processor, receiving said initial digital image;
sending the initial digital image to a first processing unit comprising a convolutional neural network (CNN);
using the CNN to classify at least one element in the initial digital image and to generate a first adjusted image by forming a composite BAM whose mean difference from a boundary map is less than 0.05;
using a second processing unit to measure 3D features of the skin cancer in the initial digital image using a 3D fiducial marker, and to generate a second adjusted image;
by way of the data processor, receiving the first adjusted image and the second adjusted image; and
overlaying at least one of the first adjusted image and the second adjusted image over a working image of the skin cancer as an AR overlay on a mobile device.
9 . The method according to claim 8 , further comprising capturing contour data and attention pixels from said initial digital image.
10 . The method according to claim 8 , further comprising determining a 3D relative size of the skin cancer using the 3D fiducial marker.
11 . A non-transitory computer readable medium storing instructions that, when executed, implements a convolutional neural network (CNN) comprising:
a stem block with variable skipped connections;
a building module;
a final module; and
a dense layer;
the instructions causing the CNN to:
extract boundary maps from stem-block features;
capture attention pixels from the final module; and
form a composite BAM whose mean difference from a boundary map is less than 0.05 for use in generating an adjusted image of a skin abnormality.
12 . The non-transitory computer readable medium according to claim 11 , wherein said CNN is trained on a dataset comprising images taken using mobile devices.
13 . The non-transitory computer readable medium according to claim 11 , wherein the adjusted image is output for AR overlay on a mobile device during treatment planning or surgery.