Video-based automated detection of generalized tonic-clonic seizures using deep learning
View Patent ↗Systems and methods of the present disclosure determine whether the patient experiences a grand tonic clonic seizures using video recordings or sensor data or both. The systems and methods receive data from a device that continuously records video and/or sensor data, continuously analyzes the data with a processing unit utilizing machine learning to classify segments of the data as seizure or no seizure, and to classify seizure types. An alarm is produced using an output unit when an epileptic data segment is detected. The processing unit thus provides continuous and in real-time monitoring of an epilepsy patient in the home or hospital setting, e.g. while the patient is sleeping in bed. In case a seizure is detected, an alarm may inform caregivers or clinicians to help them intervene and limit the complications of this seizure for the patient.
1 . A method comprising:
receiving, by at least one processor, real-time video data comprising a continuous video feed of a patient location;
extracting, by the at least one processor, an epoch of video data comprising a video segment from the continuous video feed from a patient monitoring period preceding a current time;
inputting, by the at least one processor, the video segment of the epoch of video data into an encoding neural network to convert the video segment into a sequence of video encodings representative of the epoch of video data, the sequence of video encodings forming time-series data representative of the epoch of video data;
inputting, by the at least one processor, each video encoding of the sequence of video encodings into a seizure recognition machine learning model configured to output a seizure classification of the video segment, comprising either a seizure video segment classification or no-seizure video segment classification, the seizure recognition machine learning model comprising at least one layer of long short term memory (LSTM) having learned model parameters; and
causing to produce, in real time, by the at least one processor, a seizure indication at a computing device associated with a caregiver for each video segment identified as a seizure video segment to alert the caregiver of a seizure.
2 . The method as recited in claim 1 , further comprising encoding, by the at least one processor, the epoch of video data into the time-series data using the encoding neural network comprising a pre-trained encoding convolutional neural network having at least one top layer removed.
3 . The method as recited in claim 2 , wherein the pre-trained encoding model comprises MobileNet.
4 . The method as recited in claim 1 , further comprising downsampling, by the at least one processor, the video segment by downsample one of:
i) frame rate,
ii) color data,
iii) resolution, or
iv) combinations thereof.
5 . The method as recited in claim 1 , wherein the seizure recognition machine learning model comprises a recurrent neural network.
6 . The method as recited in claim 1 , wherein the seizure recognition machine learning model comprises a long short-term memory network.
7 . The method as recited in claim 1 , further comprising:
determining, by the at least one processor, a seizure probability value using the seizure recognition machine learning model based on the time-series data; and
determining, by the at least one processor, the seizure classification of the video segment based on a comparison of the seizure probability value to a classification threshold.
8 . The method as recited in claim 7 , wherein the seizure classification comprises the no seizure video segment classification where the seizure probably value is less than the classification threshold.
9 . The method as recited in claim 1 , wherein the patient monitoring period comprises 5 seconds associated with the epoch of video data comprising 5 seconds of video data.
10 . The method as recited in claim 1 , further comprising utilizing, by the at least one processor, the seizure recognition machine learning model for each subsequent epoch of video data received in the continuous video feed.
11 . A system comprising:
at least one processor configured to receive instructions stored in a non-transitory memory that cause the at least one processor to perform steps to:
receive real-time video data comprising a continuous video feed of a patient location;
extract an epoch of video data comprising a video segment from the continuous video feed from a patient monitoring period preceding a current time;
input the video segment of the epoch of video data into an encoding neural network to convert the video segment into sequence of video encodings representative of the epoch of video data, the sequence of video encodings forming time-series data representative of the epoch of video data;
input each video encoding of the sequence of video encodings into a seizure recognition machine learning model configured to output a seizure classification of the video segment, comprising either a seizure video segment classification or no-seizure video segment classification, the seizure recognition machine learning model comprising at least one layer of long short term memory (LSTM) having learned model parameters; and
cause to produce, in real time, a seizure indication at a computing device associated with a caregiver for each video segment identified as a seizure video segment to alert the caregiver of a seizure.
12 . The system as recited in claim 11 , wherein the at least one processor is further configured to receive instructions causing the at least one processor to perform steps to encode the epoch of video data into the time-series data using the encoding neural network comprising a pre-trained encoding convolutional neural network having at least one top layer removed.
13 . The system as recited in claim 12 , wherein the pre-trained encoding model comprises MobileNet.
14 . The system as recited in claim 11 , wherein the at least one processor is further configured to receive instructions causing the at least one processor to perform steps to downsample the video segment by downsample one of:
i) frame rate,
ii) color data,
iii) resolution, or
iv) combinations thereof.
15 . The system as recited in claim 11 , wherein the seizure recognition machine learning model comprises a recurrent neural network.
16 . The system as recited in claim 11 , wherein the seizure recognition machine learning model comprises a long short-term memory network.
17 . The system as recited in claim 11 , wherein the at least one processor is further configured to receive instructions causing the at least one processor to perform steps to:
determine a seizure probability value using the seizure recognition machine learning model based on the time-series data; and
determine the seizure classification of the video segment based on a comparison of the seizure probability value to a classification threshold.
18 . The system as recited in claim 17 , wherein the seizure classification comprises the no seizure video segment classification where the seizure probably value is less than the classification threshold.
19 . The system as recited in claim 11 , wherein the patient monitoring period comprises 5 seconds associated with the epoch of video data comprising 5 seconds of video data.
20 . The system as recited in claim 11 , wherein the at least one processor is further configured to receive instructions causing the at least one processor to perform steps to utilize the seizure recognition machine learning model for each subsequent epoch of video data received in the continuous video feed.