Methods and system for cardiac arrhythmia prediction using transformer-based neural networks
Methods and systems are provided for predicting cardiac arrhythmias based on multi-modal patient monitoring data via deep learning. In an example, a method may include predicting an imminent onset of a cardiac arrhythmia in a patient, before the cardiac arrhythmia occurs, by analyzing patient monitoring data via a multi-arm deep learning model, outputting an arrhythmia event in response to the prediction, and outputting a report indicating features of the patient monitoring data contributing to the prediction. In this way, the multi-arm deep learning model may predict cardiac arrhythmias before their onset.
1 . A method, comprising:
predicting an imminent onset of a cardiac arrhythmia in a patient, before the cardiac arrhythmia occurs, by analyzing patient monitoring data via a multi-arm deep learning model comprising a transformer network having an encoder portion and a decoder portion, a first multilayer perceptron, and a two-dimensional convolutional neural network (CNN), wherein analyzing the patient monitoring data via the multi-arm deep learning model comprises:
extracting local features of the patient monitoring data, the local features including beat-level features of the patient monitoring data and waveforms of the patient monitoring data;
extracting contextual features of the patient monitoring data, the contextual features including multi-beat segment-level features of the patient monitoring data;
generating two-dimensional (2D) spectrograms comprising a frequency domain transform of multi-beat segments of the patient monitoring data;
combining the beat-level features of the patient monitoring data and the waveforms of the patient monitoring data to produce a temporally varying feature set;
inputting the temporally varying feature set into the transformer network, wherein the temporally varying feature set is encoded by the encoder portion, and then decoded by the decoder portion of the transformer network;
inputting the multi-beat segment-level features into the first multilayer perceptron; inputting the two-dimensional spectrograms into the 2D CNN;
inputting an output of each of the transformer network, the first multilayer perceptron, and the 2D CNN into a second multilayer perceptron; and receiving an arrhythmia prediction score from the second multilayer perceptron, wherein predicting the imminent onset of the cardiac arrhythmia in the patient is in response to the arrhythmia prediction score being greater than a threshold score;
outputting an arrhythmia event in response to the prediction; and outputting a report indicating features of the patient monitoring data contributing to the prediction.
2 . The method of claim 1 , wherein the patient monitoring data comprises electrocardiogram (ECG) data acquired from the patient, and the report indicates beats of the ECG data contributing to the prediction.
3 . The method of claim 2 , wherein the report is generated via an explainability algorithm that receives attention matrices of the beats of the ECG data generated by the transformer network of the multi-arm deep learning model.
4 . The method of claim 2 , wherein the extracted local features comprise:
a PR interval measured from a beginning of a P wave of the ECG data to a beginning of a ORS complex of the ECG data;
an RR interval measured from a first R wave to a second R wave of a subsequent beat of the ECG data; and
a QT interval measured from the beginning of the ORS complex to an end of a T wave of the ECG data.
5 . The method of claim 2 , wherein the extracted local and/or contextual features further include a noise level and a signal-to-noise ratio.
6 . The method of claim 1 , wherein the local features and the contextual features are extracted by a feature extraction algorithm prior to being input into the multi-arm deep learning model.
7 . The method of claim 1 , wherein the extracted local and/or contextual features further include arrhythmia annotations.
8 . The method of claim 1 , wherein the cardiac arrhythmia is one of ventricular fibrillation, atrial fibrillation, and ventricular tachycardia.
9 . A method, comprising:
acquiring electrocardiogram (ECG) data from a patient;
extracting local and contextual features from the ECG data as the ECG data is acquired, the local features including beat-level features of the ECG data and waveforms of the ECG data, the contextual features including features of multi-beat segments of the ECG data;
inputting the beat-level features of the ECG data and the waveforms of the ECG data into a transformer network, wherein the beat-level features of the ECG data and the waveforms of the ECG data are encoded by an encoder portion, and then decoded by a decoder portion of the transformer network;
inputting the multi-beat segment features into a first multilayer perceptron;
inputting two-dimensional (2D) spectrograms comprising a frequency domain transform of the multi-beat segments into a 2D convolutional neural network (CNN);
inputting an output of each of the transformer network, the first multilayer perceptron, and the 2D CNN into a second multilayer perceptron;
predicting whether or not an imminent ventricular fibrillation is expected to occur in the patient based on an output of the second multilayer perceptron, wherein the second multilayer perceptron outputs a ventricular fibrillation probability score and wherein predicting whether or not the imminent ventricular fibrillation is expected to occur in the patient based on the output of the second multilayer perceptron comprises:
predicting the imminent ventricular fibrillation is expected to occur in the patient in response to the ventricular fibrillation probability score being greater than a threshold score; and
predicting the imminent ventricular fibrillation is not expected to occur in the patient in response to the ventricular fibrillation probability score being less than or equal to the threshold score; and
outputting an arrhythmia event and an explainability report indicating portions of the ECG data contributing to the prediction in response to the imminent ventricular fibrillation being expected to occur in the patient.
10 . The method of claim 9 , wherein the ECG data is single-lead ECG data.
11 . The method of claim 9 , wherein the explainability report is generated using an explainability algorithm that identifies beats of the ECG data contributing to the predication using an attention matrix received from transformer network.
12 . A system, comprising:
a display device; and
a computing device operably coupled to the display device and storing instructions executable to:
receive patient monitoring data acquired from a patient, wherein the patient monitoring data comprises at least electrocardiogram (ECG) data;
predict whether an imminent cardiac arrhythmia is expected to occur in the patient by analyzing the patient monitoring data via a multi-arm deep learning model;
output an arrhythmia event to the display device in response to the imminent cardiac arrhythmia being expected to occur;
identify portions of the patient monitoring data leading to the prediction in response to the imminent cardiac arrhythmia being expected to occur, wherein the identified portions include beats of the ECG data having higher than a threshold attention scores output by a transformer network of the multi-arm deep learning model, wherein the identified portions of the patient monitoring data are encoded by an encoder portion, and then decoded by a decoder portion of the transformer network; and
output an explainability report of the identified portions to the display device;
wherein further instructions are stored in the computing device executable to:
extract local and contextual features from the ECG data as the ECG data is acquired, the local features including beat-level features of the ECG data and waveforms of the ECG data, the contextual features including features of multi-beat segments of the ECG data;
input the beat-level features of the ECG data and the waveforms of the ECG data into the transformer network;
input the multi-beat segment features into a first multilayer perceptron;
input two-dimensional (2D) spectrograms comprising a frequency domain transform of the multi-beat segments into a 2D convolutional neural network (CNN);
input an output of each of the transformer network, the first multilayer perceptron, and the 2D CNN into a second multilayer perceptron; and
receive a prediction of the imminent cardiac arrhythmia as an output of the second multilayer perceptron.
13 . The system of claim 12 , wherein the patient monitoring data further comprises patient vital sign data, and wherein local and contextual features of the ECG data and the patient vital sign data are extracted by an algorithm in real-time as the patient monitoring data is acquired, and the extracted local and contextual features are input into the multi-arm deep learning model.
14 . The system of claim 13 , wherein the multi-arm deep learning model comprises three parallel subnetworks that each receive different subsets of the extracted local and contextual features and a fourth subnetwork in series with the three parallel subnetworks.