Waveform Analysis And Detection Using Machine Learning Transformer Models

1 . A computerized method of analyzing a waveform using a machine learning transformer model, the method comprising:

obtaining labeled waveform training data and unlabeled waveform training data;

supplying the unlabeled waveform training data to a transformer model to pre-train the transformer model by masking a portion of an input to the transformer model;

supplying the labeled waveform training data to the transformer model without masking a portion of the input to the transformer model to fine-tune the transformer model, wherein each waveform in the labeled waveform training data includes at least one label identifying a feature of the waveform; and

supplying a target waveform to the transformer model to classify at least one feature of the target waveform, wherein the at least one classified feature corresponds to the least one label of the labeled waveform training data.

2 . The method of claim 1 , further comprising:

obtaining categorical risk factor data;

obtaining numerical risk factor data;

embedding categorical risk factor data and concatenating the embedded categorical risk factor data with the numerical risk factor data to form a concatenated feature vector; and

supplying the concatenated feature vector to the transformer model to increase an accuracy of the at least one classified feature.

3 . The method of claim 2 , wherein:

the unlabeled waveform training data, the labeled waveform training data, and the target waveform each comprise an electrocardiogram (ECG) waveform recorded from a patient;

the categorical risk factor data includes a sex of the patient; and

the numerical risk factor data includes at least one of an age of the patient, a height of the patient, and a weight of the patient.

4 . The method of claim 2 , wherein:

the categorical risk factor data includes multiple groups of categorical values;

each group is encoded using one-hot encoding; and

embedding the categorical risk factor data includes combining each of the encoded groups into a combined encoded vector and then feeding the combined encoded vector to a neural network to output an embedded categorical risk factor vector.

5 . The method of claim 1 , wherein:

the unlabeled waveform training data, the labeled waveform training data, and the target waveform each comprise an electrocardiogram (ECG) waveform recorded from a patient;

the at least one label of each waveform in the labeled waveform training data includes at least one of a detected heart arrhythmia, a P wave and a T wave; and

the at least one classified feature includes the at least one of a detected heart arrhythmia, a P wave and a T wave.

6 . The method of claim 1 , wherein the transformer model comprises a Bidirectional Encoder Representations from Transformers (BERT) model.

7 . The method of claim 1 , wherein supplying the unlabeled waveform training data to pre-train the transformer model and supplying the labeled waveform training data to fine-tune the transformer model each include periodically relaxing a learning rate of the transformer model by reducing the learning rate during a specified number of epochs and then resetting the learning rate to an original value before running a next specified number of epochs.

8 . The method of claim 1 , wherein:

the unlabeled waveform training data includes daily seismograph waveforms;

the labeled waveform training data includes detected earthquake event seismograph waveforms; and

the at least one classified feature includes a detected earthquake event.

9 . The method of claim 1 , wherein the labeled waveform training data, the unlabeled waveform training data, and the target waveform each include at least one of an automobile traffic pattern waveform, a human traffic pattern waveform, an electroencephalogram (EEG) waveform, a network data flow waveform, a solar activity waveform, and a weather waveform.

10 . The method of claim 1 , wherein:

the transformer model is located on a processing server;

the target waveform is stored on a local device separate from the processing server; and

the method further includes compressing the target waveform and transmitting the target waveform to the processing server for input to the transformer model.

11 . A computer system comprising:

memory hardware configured to store unlabeled waveform training data, labeled waveform training data, a target waveform, a transformer model, and computer-executable instructions; and

processor hardware configured to execute the instructions, wherein the instructions include:

obtaining labeled waveform training data and unlabeled waveform training data;

supplying the unlabeled waveform training data to the transformer model to pre-train the transformer model by masking a portion of an input to the transformer model;

supplying the labeled waveform training data to the transformer model without masking a portion of the input to the transformer model to fine-tune the transformer model, each waveform in the labeled waveform training data including at least one label identifying a feature of the waveform; and

supplying a target waveform to the transformer model to classify at least one feature of the target waveform, wherein the at least one classified feature corresponds to the least one label of the labeled waveform training data.

12 . The computer system of claim 11 , wherein the instructions include:

obtaining categorical risk factor data;

obtaining numerical risk factor data;

embedding categorical risk factor data and concatenating the embedded categorical risk factor data with the numerical risk factor data to form a concatenated feature vector; and

supplying the concatenated feature vector to the transformer model to increase an accuracy of the at least one classified feature.

13 . The computer system of claim 12 , wherein:

the unlabeled waveform training data, the labeled waveform training data, and the target waveform each comprise an electrocardiogram (ECG) waveform recorded from a patient;

the categorical risk factor data includes a sex of the patient; and

the numerical risk factor data includes at least one of an age of the patient, a height of the patient, and a weight of the patient.

14 . The computer system of claim 12 , wherein:

the categorical risk factor data includes multiple groups of categorical values;

each group is encoded using one-hot encoding; and

embedding the categorical risk factor data includes combining each of the encoded groups into a combined encoded vector and then feeding the combined encoded vector to a neural network to output an embedded categorical risk factor vector.

15 . The computer system of claim 11 , wherein:

the unlabeled waveform training data, the labeled waveform training data, and the target waveform each comprise an electrocardiogram (ECG) waveform recorded from a patient;

the at least one label of each waveform in the labeled waveform training data includes at least one of a detected heart arrhythmia, a P wave and a T wave; and

the at least one classified feature includes the at least one of a detected heart arrhythmia, a P wave and a T wave.

16 . A non-transitory computer-readable medium storing processor-executable instructions, the instructions comprising:

obtaining labeled waveform training data and unlabeled waveform training data;

supplying the unlabeled waveform training data to a transformer model to pre-train the transformer model by masking a portion of an input to the transformer model;

supplying the labeled waveform training data to the transformer model without masking a portion of the input to the transformer model to fine-tune the transformer model, each waveform in the labeled waveform training data including at least one label identifying a feature of the waveform; and

supplying a target waveform to the transformer model to classify at least one feature of the target waveform, wherein the at least one classified feature corresponds to the least one label of the labeled waveform training data.

17 . The non-transitory computer-readable medium of claim 16 , wherein supplying the unlabeled waveform training data to pre-train the transformer model and supplying the labeled waveform training data to fine-tune the transformer model each include periodically relaxing a learning rate of the transformer model by reducing the learning rate during a specified number of epochs and then resetting the learning rate to an original value before running a next specified number of epochs.

18 . The non-transitory computer-readable medium of claim 16 , wherein:

the unlabeled waveform training data includes daily seismograph waveforms;

the labeled waveform training data includes detected earthquake event seismograph waveforms; and

the at least one classified feature includes a detected earthquake event.

19 . The non-transitory computer-readable medium of claim 16 , wherein the labeled waveform training data, the unlabeled waveform training data, and the target waveform each include at least one of an automobile traffic pattern waveform, a human traffic pattern waveform, an electroencephalogram (EEG) waveform, a network data flow waveform, a solar activity waveform, and a weather waveform.

20 . The non-transitory computer-readable medium of claim 16 , wherein:

the transformer model is located on a processing server;

the target waveform is stored on a local device separate from the processing server; and

the instructions further include compressing the target waveform and transmitting the target waveform to the processing server for input to the transformer model.