Smartphone-based digital pupillometer
In some embodiments, techniques for using machine learning to enable visible light pupilometry are provided. In some embodiments, a smartphone may be used to create a visible light video recording of a pupillary light reflex (PLR). A machine learning model may be used to detect a size of a pupil in the video recording over time, and the size over time may be presented to a clinician. In some embodiments, a system that includes a smartphone and a box that holds the smartphone in a predetermined relationship to a subject's face is provided. In some embodiments, a sequential convolutional neural network architecture is used. In some embodiments, a fully convolutional neural network architecture is used.
1. A computer-implemented method of generating and using a machine learning model to measure pupillary response for diagnosis of brain injury, the method comprising:
collecting, using one or more mobile computing devices, video recordings of eyes responding to light stimuli, wherein the video recordings each include a left eye of a subject and a right eye of the subject;
splitting, by at least one computing device, each video recording horizontally into a left eye video recording and a right eye video recording;
horizontally flipping, by at least one computing device, the left eye video recording or the right eye video recording to create training data;
receiving, by at least one computing device, tagging information indicating a location and a size of pupils in frames of each left eye video recording and right eye video recording of the training data;
using, by at least one computing device, the training data and the tagging information to train a machine learning model to recognize pupil location and size in video frames; and
storing the machine learning model on a mobile computing device.
2. The method of claim 1 , further comprising:
collecting, using the mobile computing device, experimental data comprising a video recording of at least one eye of a subject responding to a light stimulus;
using, by the mobile computing device, the machine learning model to determine changes over time in a size of a pupil of the at least one eye; and
presenting, by the mobile computing device, a plot of the changes over time in the size of the pupil of the at least one eye.
3. The method of claim 1 , wherein the machine learning model includes at least one convolutional neural network (CNN).
4. The method of claim 3 , wherein the machine learning model includes a first CNN configured to recognize a pupil location and a second CNN configured to determine a pupil diameter.
5. The method of claim 3 , wherein the at least one CNN includes 5 convolutional layers and three fully connected layers.
6. The method of claim 3 , wherein convolutional layers of the at least one CNN include a rectified linear (ReLU) activation function followed by 2×2 mean-pooling layers.
7. The method of claim 2 , further comprising preprocessing at least one of the training data and the experimental data.
8. The method of claim 7 , wherein preprocessing includes one or more of:
cropping the video recording;
down-sampling the video recording;
converting the video recording to a hue, saturation, and lightness (HSL) color space;
applying contrast-limited adaptive histogram equalization (CLAHE) to a lightness channel of the video recording; and
converting the video recording to grayscale.
9. A non-transitory computer-readable medium having computer-executable instructions stored thereon which, in response to execution by one or more processors of one or more computing devices, cause the one or more computing devices to perform actions comprising:
collecting, using one or more mobile computing devices, video recordings of eyes responding to light stimuli, wherein the video recordings each include a left eye of a subject and a right eye of the subject;
splitting, by at least one computing device, each video recording horizontally into a left eye video recording and a right eye video recording;
horizontally flipping, by at least one computing device, the left eye video recording or the right eye video recording to create training data;
receiving, by at least one computing device, tagging information indicating a location and a size of pupils in frames of each left eye video recording and right eye video recording of the training data;
using, by at least one computing device, the training data and the tagging information to train a machine learning model to recognize pupil location and size in video frames; and
storing the machine learning model on a mobile computing device.
10. The computer-readable medium of claim 9 , wherein the actions further comprise:
collecting, using the mobile computing device, experimental data comprising a video recording of at least one eye of a subject responding to a light stimulus;
using, by the mobile computing device, the machine learning model to determine changes over time in a size of a pupil of the at least one eye; and
presenting, by the mobile computing device, a plot of the changes over time in the size of the pupil of the at least one eye.
11. The computer-readable medium of claim 9 , wherein the machine learning model includes at least one convolutional neural network (CNN), including at least a first CNN configured to recognize a pupil location and a second CNN configured to determine a pupil diameter.
12. The computer-readable medium of claim 11 , wherein the at least one CNN includes 5 convolutional layers and three fully connected layers.
13. The computer-readable medium of claim 11 , wherein convolutional layers of the at least one CNN include a rectified linear (ReLU) activation function followed by 2×2 mean-pooling layers.
14. The computer-readable medium of claim 10 , wherein the actions further comprise preprocessing at least one of the training data and the experimental data;
wherein preprocessing includes one or more of:
cropping the video recording;
down-sampling the video recording;
converting the video recording to a hue, saturation, and lightness (HSL) color space;
applying contrast-limited adaptive histogram equalization (CLAHE) to a lightness channel of the video recording; and
converting the video recording to grayscale.
15. A computing system comprising:
one or more mobile computing devices configured to collect video recordings of eyes responding to light stimuli, wherein the video recordings each include a left eye of a subject and a right eye of the subject;
at least one computing device configured to split each video recording horizontally into a left eye video recording and a right eye video recording;
at least one computing device configured to flip the left eye video recording or the right eye video recording to create training data;
at least one computing device configured to receive tagging information indicating a location and a size of pupils in frames of each left eye video recording and right eye video recording of the training data;
at least one computing device configured to use the training data and the tagging information to train a machine learning model to recognize pupil location and size in video frames; and
a mobile computing device configured to store the machine learning model.
16. The system claim 15 , wherein the mobile computing device configured to store the machine learning model is further configured to:
collect experimental data comprising a video recording of at least one eye of a subject responding to a light stimulus;
use the machine learning model to determine changes over time in a size of a pupil of the at least one eye; and
present a plot of the changes over time in the size of the pupil of the at least one eye.
17. The system of claim 15 , wherein the machine learning model includes at least one convolutional neural network (CNN), including at least a first CNN configured to recognize a pupil location and a second CNN configured to determine a pupil diameter.
18. The system of claim 17 , wherein the at least one CNN includes 5 convolutional layers and three fully connected layers.
19. The system of claim 17 , wherein convolutional layers of the at least one CNN include a rectified linear (ReLU) activation function followed by 2×2 mean-pooling layers.
20. The system of claim 18 , wherein the system further comprises at least one computing device configured to preprocess at least one of the training data and the experimental data;
wherein preprocessing includes one or more of:
cropping the video recording;
down-sampling the video recording;
converting the video recording to a hue, saturation, and lightness (HSL) color space;
applying contrast-limited adaptive histogram equalization (CLAHE) to a lightness channel of the video recording; and
converting the video recording to grayscale.