IP Library › Granted Patent US 12,430,573
Granted Patent B2
US 12,430,573 · App. 17/326,098 · Granted Sep 30, 2025

Subject-aware contrastive learning for biosignals

Inventors: Joseph Y. Cheng (Santa Clara, CA); Erdrin Azemi (San Mateo, CA); Hanlin Goh (Sunnyvale, CA); Kaan E. Dogrusoz (Santa Clara, CA); Cuneyt O. Tuzel (Cupertino, CA)
Assignee: Apple Inc.
G06N5/04G06N20/00
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,430,573
App. No.
17/326,098
Granted
Sep 30, 2025
Kind
B2
Abstract

The present application relates to apparatus, systems, and methods to perform subject-aware self-supervised learning of a machine-learning model for classification of data, such as classification of biosignals.

Claims (53)

1. A computer-implemented method comprising:

accessing a training data set that includes multiple training elements, wherein each training element of the multiple training elements includes a signal, and wherein the multiple training elements are associated with two or more subjects;

training a machine-learning model using the training data set, wherein the training includes:

identifying subject information associated with the two or more subjects from the training data set;

determining contrastive loss for the training data set in view of the identified subject information;

training a first encoder of the machine-learning model to move representations of training elements, of the multiple training elements, having similar characteristics closer together and move representations of training elements, of the multiple training elements, having dissimilar characteristics farther apart based on the contrastive loss; and

training a second encoder of the machine-learning model to move representations of training elements associated with a same subject closer together and move the representations of training elements associated with different subjects farther apart based on the contrastive loss, the second encoder having momentum that causes second encoder to be trained slower than the first encoder during the training; and

using the trained machine-learning model to process an input signal.

2. The computer-implemented method of claim 1 , further comprising removing the identified subject information from the training data set, wherein determining the contrastive loss for the training data set in view of the identified subject information includes determining the contrastive loss with the identified subject information removed from the training data set.

3. The computer-implemented method of claim 1 , wherein training the machine-learning model includes:

training the first encoder with an adversarial identifier disabled, the adversarial identifier to remove the identified subject information;

fixing the first encoder as trained with the adversarial identifier disabled;

training the adversarial identifier with the first encoder fixed as trained with the adversarial identifier disabled;

fixing the adversarial identifier as trained with the first encoder fixed; and

retraining the first encoder with the adversarial identifier fixed as trained with the first encoder fixed.

4. The computer-implemented method of claim 1 , wherein the machine-learning model includes a first transformation element coupled between an input of the machine-learning model and an input of the first encoder, and a second transformation element coupled between the input of the machine-learning model and an input of the second encoder, and wherein the method further comprises:

applying, by the first transformation element, a first transformation to the training data set received at the input of the machine-learning model; and

applying, by the second transformation element, a second transformation to the training data set, the second transformation being different than the first transformation, and the second transformation includes a perturbation.

5. The computer-implemented method of claim 4 , wherein the perturbation includes temporal cutout, temporal delay, noise, bandstop filtering, signal mixing, spatial rotation, spatial shift, sensor dropout, or sensor cutout.

6. The computer-implemented method of claim 1 , wherein the signal of each training element includes a biosignal.

7. A computer-implemented method comprising:

accessing a training data set that includes multiple training elements, wherein each training element of the multiple training elements includes a signal, and wherein the multiple training elements are associated with a single subject and each training element of the multiple training elements are associated with different times;

training a machine-learning model using the training data set including:

determining, by a fully connected layer of the machine-learning model coupled to an output of a first encoder of the machine-learning model and an output of a second encoder of the machine-learning model, contrastive loss for the training data set based on a difference between signals on the output of the first encoder and signals on the output of the second encoder, the first encoder and the second encoder coupled to an input of the machine-learning model; and

training the first encoder of the machine-learning model to move representations of training elements, of the multiple training elements, having similar characteristics closer together and representations of training elements, of the multiple training elements, having dissimilar characteristics farther apart based on the contrastive loss; and

using the trained machine-learning model to process an input signal.

8. The computer-implemented method of claim 7 , wherein determining the contrastive loss includes including subject information related to the single subject in a negative component for determining the contrastive loss.

9. The computer-implemented method of claim 7 , wherein the machine-learning model further includes a transformation element coupled between the input of the machine-learning model and the second encoder, wherein the method further includes applying, by the transformation element, perturbations to signals received from the input of the machine-learning model.

10. The computer-implemented method of claim 9 , wherein the signal of each of the multiple training elements is an electroencephalogram (EEG) signal, and wherein the perturbations include spatial rotations, spatial shifts, sensor dropouts, or sensor cutouts.

11. A computer-program product tangibly embodied in one or more non-transitory machine-readable storage media, including instructions to cause a machine-learning model to:

access a training data set that includes multiple training elements, wherein each training element of the multiple training elements includes a signal, and wherein the multiple training elements are associated with one or more subjects;

identify subject information associated with the one or more subjects from the training data set;

determine contrastive loss for the training data set in view of the identified subject information;

apply a first transformation to the multiple training elements to produce first transformed multiple elements;

train a first encoder of the machine-learning model to move representations of training elements, of the first transformed multiple training elements, having similar characteristics closer together and representations of training elements, of the first transformed multiple training elements, having dissimilar characteristics farther apart;

apply a second transformation to the multiple training elements to produce second transformed multiple elements, the second transformed multiple elements to be supplied to a second encoder, wherein the second transformation is different than the first transformation; and

use the machine-learning model to process an input signal.

12. The computer-program product of claim 11 , wherein the instructions are further to:

cause the machine-learning model to train an adversarial identifier using cross entropy loss with the first encoder fixed with a certain encoding algorithm; and

remove, by the adversarial identifier, the identified subject information from the training data set, wherein to determine the contrastive loss for the training data set in view of the identified subject information includes to determine the contrastive loss with the identified subject information removed from the training data set.

13. The computer-program product of claim 12 , wherein to train the first encoder includes to train the first encoder with the adversarial identifier fixed with a certain identification model.

14. The computer-program product of claim 11 , wherein the second transformation includes temporal cutout, temporal delay, bandstop filtering, signal mixing, spatial rotation, spatial shift, sensor dropout, or sensor cutout of the multiple training elements.

15. The computer-program product of claim 11 , wherein the instructions are further to cause the machine-learning model to predict the one or more subjects from the multiple training elements of the training data set.

16. A system, comprising:

one or more data processors; and

a non-transitory computer-readable storage medium containing instructions which, when executed on the one or more data processors, cause the one or more data processors to:

access a training data set that includes multiple training elements, wherein each training element of the multiple training elements includes a signal, and wherein the multiple training elements are associated with one or more subjects;

identify subject information associated with the one or more subjects from the training data set;

determine contrastive loss for the training data set in view of the identified subject information;

train a first encoder of a machine-learning model to move representations of training elements, of the multiple training elements, having similar characteristics closer together and representations of training elements, of the multiple training elements, having dissimilar characteristics further apart based on the contrastive loss;

train a second encoder with the training data set, the second encoder implementing momentum that causes the second encoder to be trained slower than the first encoder; and

use the machine-learning model to process an input signal.

17. The system of claim 16 , wherein instructions, when executed on the one or more data processors, further cause the one or more data processors to remove the identified subject information from the training data set, wherein to determine the contrastive loss for the training data set in view of the identified subject information includes to determine the contrastive loss with the identified subject information removed from the training data set.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 28, 2021
From: CHENG, JOSEPH Y.; AZEMI, ERDRIN; GOH, HANLIN; DOGRUSOZ, KAAN E.; TUZEL, CUNEYT O.
To: APPLE INC.
Reel/Frame 056389/0430 →
Continuity (2)
Provisional Application 63029945 · May 26, 2020
Related Publication 20210374570A1 · Dec 2, 2021
References Cited (61)
US 10839013B1 · Werris · 2020 [cited by examiner]
US 11769110B1 · Bourke · 2023 [cited by examiner]
US 20180218256A1 · Raviv · 2018 [cited by examiner]
US 20210049628A1 · Baird · 2021 [cited by examiner]
US 20210233615A1 · Banavar · 2021 [cited by examiner]
US 20210295091A1 · Li · 2021 [cited by examiner]
US 20210334645A1 · Pardeshi · 2021 [cited by examiner]
US 20210358177A1 · Park · 2021 [cited by examiner]
US 20220139070A1 · Okamoto · 2022 [cited by examiner]
US 20220198356A1 · Gabale · 2022 [cited by examiner]
Hadsell et al., “Dimensionality Reduction by Learning an Invariant Mapping.” Nov. 2005, 8 pages. [cited by applicant]
Chen et al., A Simple Framework for Contrastive Learning of Visual Representations. arXiv:2002.05709 [cs, stat], Feb. 2020, 17 pages. [cited by applicant]
Cui et al., Class-Balanced Loss Based on Effective Number of Samples. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, Jun. 2019, pp. 9260-9269. [cited by applicant]
P. de Chazal et al., Automatic Classification of Heartbeats Using ECG Morphology and Heartbeat Interval Features. IEEE Transactions on Biomedical Engineering, vol. 51, No. 7, Jul. 2004, pp. 1196-1206. [cited by applicant]
DeVries et al., Improved Regularization of Convolutional Neural Networks with Cutout. arXiv:1708.04552 [cs], Nov. 2017, 8 pages. [cited by applicant]
Doquire et al. Feature Selection for Interpatient Supervised Heart Beat Classification. Computational Intelligence and Neuroscience, 2011, pp. 1-9. [cited by applicant]
Garcia et al., Inter-Patient ECG Heartbeat Classification with Temporal VCG Optimized by PSO. Scientific Reports, vol. 7, No. 1, Dec. 2017, 11 pages. [cited by applicant]
He et al., Identity Mappings in Deep Residual Networks. In European Conference on Computer Vision, Jul. 2016, pp. 630-645. [cited by applicant]
Huang et al., A new hierarchical method for inter-patient heartbeat classification using random projections and RR intervals. BioMedical Engineering OnLine, vol. 13, No. 1, 2014, p. 90. [cited by applicant]
Ioffe et al., Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. International Conference on Machine Learning, Mar. 2015. 9 pages. [cited by applicant]
Lin et al. Heartbeat Classification Using Normalized RR Intervals and Morphological Features. Mathematical Problems in Engineering, 2014, pp. 1-11. [cited by applicant]
Luo et al., Patient-Specific Deep Architectural Model for ECG Classification. Journal of Healthcare Engineering, 2017, pp. 1-13. [cited by applicant]
Niu et al., Inter-Patient ECG Classification with Symbolic Representations and Multi-Perspective Convolutional Neural Networks. IEEE Journal of Biomedical and Health Informatics, 2019, pp. 1321-1332. [cited by applicant]
Xu et al., Towards End-to-End ECG Classification With Raw Signal Extraction and Deep Neural Networks. IEEE Journal of Biomedical and Health Informatics, vol. 23, No. 4, Jul. 2019, pp. 1574-1584. [cited by applicant]
Yun et al., CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features. In International Conference on Computer Vision, Aug. 2019, pp. 6023-6032. [cited by applicant]
Zhang et al., “mixup: Beyond Empirical Risk Minimization. In International Conference on Learning Representations,” Apr. 2018, 13 pages. [cited by applicant]
Acharya et al., “Automated EEG analysis of epilepsy: A review,” Knowledge-Based Systems, vol. 45, Jun. 2013. pp. 147-165. [cited by applicant]
De Aguiar Neto et al., “Depression biomarkers using non-invasive EEG: A review,” Neuroscience & Biobehavioral Reviews, vol. 105, Oct. 2019, pp. 83-93. [cited by applicant]
J. Healey et al., “Detecting Stress During Real-World Driving Tasks Using Physiological Sensors,” IEEE Transactions on Intelligent Transportation Systems, vol. 6, Jun. 2005, pp. 156-166. [cited by applicant]
Misra et al., “Shuffle and Learn: Unsupervised Learning using Temporal Order Verification,” European Conference on Computer Vision, Jul. 2016, 21 pages. [cited by applicant]
Wei et al., “Learning and Using the Arrow of Time,” in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, (Salt Lake City, UT), Jun. 2018, pp. 8052-8060. [cited by applicant]
Oord et al., “Representation Learning with Contrastive Predictive Coding,” arXiv:1807.03748 [cs, stat], Jan. 2019, 13 pages. [cited by applicant]
Doersch et al., “Multi-task Self-Supervised Visual Learning,” 2017 IEEE International Conference on Computer Vision (ICCV), Oct. 2017, pp. 2070-2079. [cited by applicant]
Banville et al., “Self-supervised representation learning from electroencephalography signals,” 2019 IEEE 29th International Workshop on Machine Learning for Signal Processing (MLSP), Nov. 2019, 6 pages. [cited by applicant]
Sarkar et al., “Self-supervised Learning for ECG-based Emotion Recognition,” ICASSP 2020—2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), May 2020, pp. 3217-3221. [cited by applicant]
Wu et al., “Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, May 2018, pp. 3733-3742. [cited by applicant]
Sermanet et al., “Time-Contrastive Networks: Self-Supervised Learning from Video,” arXiv:1704.06888 [cs], Mar. 2018, 15 pages. [cited by applicant]
He et al., “Momentum Contrast for Unsupervised Visual Representation Learning,” arXiv:1911.05722 [cs], Nov. 2019, 11 pages. [cited by applicant]
Hyvärinen et al., “Unsupervised Feature Extraction by Time-Contrastive Learning and Nonlinear ICA,” in Advances in Neural Information Processing Systems 29, 2016, 11 pages. [cited by applicant]
Tzeng et al., “Adversarial Discriminative Domain Adaptation,” 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, Jul. 2017, pp. 2962-2971. [cited by applicant]
Farshchian, et al., “Adversarial Domain Adaptation for Stable Brain-Machine Interfaces,” International Conference on Learning Representations, 2019, 14 pages. [cited by applicant]
Tzeng et al., “Simultaneous Deep Transfer Across Domains and Tasks,” in The IEEE International Conference on Computer Vision, 2015, pp. 4068-4076. [cited by applicant]
Xie et al., “Controllable Invariance through Adversarial Feature Learning,” in Advances in Neural Information Processing Systems, 2017, 11 pages. [cited by applicant]
Ozdenizci et al., “Learning Invariant Representations From EEG via Adversarial Inference,” IEEE Access, vol. 8, 2020, pp. 27074-27085. [cited by applicant]
Krell et al., “Rotational data augmentation for electroencephalographic data,” in 2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), (Seogwipo), IEEE, Jul. 2017, pp… [cited by applicant]
Gutmann et al., “Noise-contrastive estimation: A new estimation principle for unnormalized statistical models,” in International Conference on Artificial Intelligence and Statistics, (Sardinia, Italy), 2010, pp. 297-304. [cited by applicant]
Goodfellow et al., “Generative Adversarial Nets,” in Advances in Neural Information Processing Systems 27, 2014, pp. 2672-2680. [cited by applicant]
Paszke et al., “Pytorch: An imperative style, high-performance deep learning library,” in Advances in Neural Information Processing Systems 32, Curran Associates, Inc., 2019, pp. 8026-8037. [cited by applicant]
Kingma, et al., “Adam: A Method for Stochastic Optimization,” International Conference for Learning Representations, Jan. 2015, 9 pages. [cited by applicant]
Goldberger et al., “PhysioBank, PhysioToolkit, and PhysioNet: Components of a New Research Resource for Complex Physiologic Signals,” Circulation, vol. 101, Jun. 2000, 6 pages. [cited by applicant]
Schalk et al., “BCI2000: A General-Purpose Brain-Computer Interface (BCI) System,” IEEE Transactions on Biomedical Engineering, vol. 51, Jun. 2004, pp. 1034-1043. [cited by applicant]
Kim et al., “Motor Imagery Classification Using Mu and Beta Rhythms of EEG with Strong Uncorrelating Transform Based Complex Common Spatial Patterns,” Computational Intelligence and Neuroscience, vol. 2016, 2016, pp. 1-… [cited by applicant]
Dose et al., “A Deep Learning MI—EEG Classification Model for BCIs,” in 2018 26th European Signal Processing Conference (EUSIPCO), (Rome), IEEE, Sep. 2018, pp. 1676-1679. [cited by applicant]
Moody et al., “The impact of the MIT-BIH Arrhythmia Database,” IEEE Engineering in Medicine and Biology Magazine, vol. 20, Jun. 2001, pp. 45-50. [cited by applicant]
Dash et al., “Automatic Real Time Detection of Atrial Fibrillation,” Annals of Biomedical Engineering, vol. 37, Sep. 2009, pp. 1701-1709. [cited by applicant]
Bruun et al., “Automatic Atrial Fibrillation detection: A novel approach using discrete wavelet transform and heart rate variability,” in 2017 39th Annual International Conference of the IEEE Engineering in Medicine and… [cited by applicant]
Wu et al., “A Deep Learning Method to Detect Atrial Fibrillation Based on Continuous Wavelet Transform,” in 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), (Berl… [cited by applicant]
PhysioNet, MIT-BIH Arrhythmia Database, Located at https://physionet.org/content/mitdb/1.0.0/, 2005, 62 pages. [cited by applicant]
Tian et al., Contrastive multiview coding, arXiv:1906.05849v5, Dec. 18, 2020, 16 pages. [cited by applicant]
Tian et al., What makes for good views for contrastive learning?, arXiv:2005.10243v3, Dec. 18, 2020, 24 pages. [cited by applicant]
Xiao et al., What Should Not Be Contrastive in Contrastive Learning, arXiv:2008.05659v2, Mar. 18, 2021, 13 pages. [cited by applicant]