IP Library › Granted Patent US 12,541,964
Granted Patent B2
US 12,541,964 · App. 18/125,341 · Granted Feb 3, 2026

Adaptable automated interpretation of rapid diagnostic tests using self-supervised learning and few-shot learning

Inventors: Samuel K. Sia (New York, NY); Shih-Fu Chang (New York, NY); Siddarth Arumugam (New York, NY); Uzay Macar (New York, NY); Jiawei Ma (New York, NY); Guangxing Han (New York, NY); David Colburn (New York, NY)
Assignee: The Trustees of Columbia University in the City of New York
G06V10/82G06V10/7715
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,541,964
App. No.
18/125,341
Granted
Feb 3, 2026
Kind
B2
Abstract

A framework for a few-shot learning method is disclosed. In a first part, self-supervision and classification supervision are used to train a feature extractor. An example self-supervision method comprises running grayscale images through an edge filter, normalizing the filtered images, setting the normalized images to ground truth, generating feature-extracted images, using a decoder to reconstruct images from the feature-extracted images, determining a loss between the reconstructed images and the ground truth images, and using the loss to update parameters of the feature extractor. In a second part, a few-shot adaptation process is performed to adapt the model to a novel rapid test kit.

Claims (62)

1 . A system for automated image interpretation of rapid test kits, wherein the system comprises:

a neural network;

wherein the neural network comprises a feature extractor;

wherein the feature extractor has been pre-trained using self-supervised learning with a set of images of first rapid test kits;

wherein the neural network has been adapted to a second rapid test kit that is a different type of test kit than the first rapid test kits;

wherein the self-supervised learning by which the feature extractor has been pre-trained comprises:

inputting the set of images of first rapid test kits;

converting the images in the inputted set of images to grayscale images;

running the grayscale images through an edge filter, normalizing the filtered images, and setting the normalized images to ground truth;

generating feature-extracted images from the images in the inputted set of images;

using a decoder to reconstruct images from the feature-extracted images;

determining a first calculated loss between the reconstructed images and the ground truth images; and

using the first calculated loss to update parameters of the feature extractor; and

wherein in addition to the set of images of first rapid test kits, the feature extractor has also been pre-trained with a synthetic dataset mixed with the set of images of first rapid test kits, wherein the synthetic dataset comprises partially-formed indicia.

2 . The system for automated image interpretation of rapid test kits as in claim 1 , wherein the feature extractor also has been pre-trained by:

using a classifier to classify the feature-extracted images;

determining a second calculated loss between the classified images and labels for the images in the inputted set of images; and

using the second calculated loss to update parameters of the feature extractor.

3 . The system for automated image interpretation of rapid test kits as in claim 1 , wherein the neural network further comprises a classifier.

4 . The system for automated image interpretation of rapid test kits as in claim 3 , wherein the neural network has been adapted to the second rapid test kit using a few-shot adaptation process comprising supervised contrastive learning.

5 . The system for automated image interpretation of rapid test kits as in claim 3 , wherein the neural network has been adapted to the second rapid test kit using a few-shot adaptation process comprising a meta-learning technique, wherein the meta-learning comprises:

(i) a meta-training step comprising:

separating a set of images of the first rapid test kits into a support set and a query set;

using the support set of the first rapid test kits to update a classifier;

using the query set of the first rapid test kits to calculate a loss on the query set samples; and

using the loss to update the initialization of the classifier; and

(ii) a meta-testing step comprising training the classifier with the updated initialization values on a set of images of the second rapid test kits to get an updated network adapted to the second rapid test kit, wherein the number of images in the set of images of the second rapid test kits is smaller than the number of images in the set of images of the first rapid test kits.

6 . The system for automated image interpretation of rapid test kits as in claim 1 , wherein the system comprises software that is implemented on a user's smart phone, wherein the software uses a gyroscope of the user's smart phone to get a test kit image into a correct orientation.

7 . The system for automated image interpretation of rapid test kits as in claim 6 , wherein the system comprises a deep neural network for instance segmentation to detect kit and membrane boundaries.

8 . The system for automated image interpretation of rapid test kits as in claim 6 , wherein the system is adapted to extract individual zones from each kit using kit dimensions provided by a kit manufacturer.

9 . A system for automated image interpretation of rapid test kits, wherein the system comprises:

a neural network;

wherein the neural network comprises a feature extractor;

wherein the feature extractor has been pre-trained using self-supervised learning with a set of images of first rapid test kits;

wherein the neural network has been adapted to a second rapid test kit that is a different type of test kit than the first rapid test kits;

wherein the neural network further comprises a classifier; and

wherein the number of images in the set of images of the second rapid test kits is at most 20.

10 . A system for automated image interpretation of rapid test kits, wherein the system comprises:

a neural network;

wherein the neural network comprises a feature extractor;

wherein the feature extractor has been pre-trained using self-supervised learning with a set of images of first rapid test kits;

wherein the neural network has been adapted to a second rapid test kit that is a different type of test kit than the first rapid test kits;

wherein the neural network further comprises a classifier; and

wherein the ratio of the number of images in the set of images of the first rapid test kits to the number of images in the set of images of the second rapid test kits is 5:1 or greater.

11 . A method for automated image interpretation of rapid test kits, wherein the method comprises:

providing a neural network comprising a feature extractor;

pre-training the feature extractor using self-supervised learning with a set of images of first rapid test kits; and

adapting the neural network to a second rapid test kit that is a different type of test kit than the first rapid test kits;

wherein the pre-training of the feature extractor using self-supervised learning with the set of images of first rapid test kits comprises:

inputting the set of images of first rapid test kits;

converting the images in the inputted set of images to grayscale images;

running the grayscale images through an edge filter, normalizing the filtered images, and setting the normalized images to ground truth;

generating feature-extracted images from the images in the inputted set of images;

using a decoder to reconstruct images from the feature-extracted images;

determining a first calculated loss between the reconstructed images and the ground truth images; and

using the first calculated loss to update parameters of the feature extractor;

wherein the method further comprises generating a synthetic dataset comprising partially-formed indicia, mixing the synthetic dataset with the set of images of first rapid test kits, and using the synthetic dataset mixed with the set of images of first rapid test kits to pre-train the feature extractor.

12 . The method for automated image interpretation of rapid test kits as in claim 5 , further comprising pre-training the feature extractor by:

using a classifier to classify the feature-extracted images;

determining a second calculated loss between the classified images and labels for the images in the inputted set of images; and

using the second calculated loss to update parameters of the feature extractor.

13 . The method for automated image interpretation of rapid test kits as in claim 11 , wherein the neural network further comprises a classifier.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 23, 2023
From: SIA, SAMUEL K.; MA, JIAWEI; CHANG, SHIH-FU; HAN, GUANGXING; ARUMUGAM, SIDDARTH; COLBURN, DAVID; MACAR, UZAY
To: THE TRUSTEES OF COLUMBIA UNIVERSITY IN THE CITY OF NEW YORK
Reel/Frame 063076/0926 →
Continuity (4)
Continuation PCTUS2021053708 · Oct 6, 2021
Provisional Application 63208852 · Jun 9, 2021
Provisional Application 63089675 · Oct 9, 2020
Related Publication 20230274538A1 · Aug 31, 2023
References Cited (103)
US 7421125B1 · Rees · 2008 [cited by applicant]
US 10340032B2 · Ramsey et al. · 2019 [cited by applicant]
US 11783563B2 · Yaghoubi · 2023 [cited by examiner]
US 20150055134A1 · Papautsky · 2015 [cited by examiner]
US 20150111201A1 · Ozcan · 2015 [cited by examiner]
US 20160080548A1 · Erickson · 2016 [cited by examiner]
US 20160274104A1 · Aminoff · 2016 [cited by examiner]
US 20160373647A1 · Morate et al. · 2016 [cited by applicant]
US 20180293350A1 · Dimov · 2018 [cited by examiner]
US 20190080453A1 · Song et al. · 2019 [cited by applicant]
US 20190252073A1 · Hsu · 2019 [cited by examiner]
US 20200019823A1 · Wang · 2020 [cited by applicant]
US 20200051241A1 · Clark et al. · 2020 [cited by applicant]
US 20200085382A1 · Taerum et al. · 2020 [cited by applicant]
US 20200210767A1 · Do et al. · 2020 [cited by applicant]
US 20200226462A1 · Maddison et al. · 2020 [cited by applicant]
US 20200234025A1 · Cohen et al. · 2020 [cited by applicant]
US 20200278297A1 · Karlovac et al. · 2020 [cited by applicant]
US 20200311914A1 · Zaharchuk et al. · 2020 [cited by applicant]
US 20210287766A1 · Juhila · 2021 [cited by examiner]
CN 111260677A · 2020 [cited by applicant]
CN 111353548A · 2020 [cited by applicant]
WO WO9953288A2 · 1999 [cited by examiner]
WO 2019122047A1 · 2019 [cited by applicant]
WO 2020242993A1 · 2020 [cited by applicant]
WO 2021202866A1 · 2021 [cited by applicant]
WO 2022006416A1 · 2022 [cited by applicant]
WO 2022086940A1 · 2022 [cited by applicant]
WO 2022086945A1 · 2022 [cited by applicant]
Li et al., “Few-Shot Image Classification via Contrastive Self-Supervised Learning,” arXiv:2008.09942v1 [cs.CV] Aug. 23, 2020 (Year: 2020). [cited by examiner]
Yuan et al., “Current and Perspective Diagnostic Techniques for COVID-19,” ACS Infect. Dis. 2020, 6, 1998-2016 (Year: 2020). [cited by examiner]
Carrio et al., “Automated Low-Cost Smartphone-Based Lateral Flow Saliva Test Reader for Drugs-of-Abuse Detection, ” Sensors 2015, 15, 29569-29593; doi:10.3390/s151129569 (Year: 2015). [cited by examiner]
Pratondo, et al., “Integrating machine learning with region-based active contour models in medical image segmentation,” Journal of Visual Communication and Image Representation, Feb. 2017, vol. 43, pp. 1-9, Elsevier. [cited by applicant]
Raghu et al., “Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAML,” arXiv:1909.09157 [cs.LG], ICLR (2020). [cited by applicant]
Ren et al., “Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks,” IEEE Trans Pattern Anal Mach Intell, vol. 39, pp. 1137-1149, 2017. [cited by applicant]
Ristaino et al., “Detection of Phytophthora infestans by Loop-Mediated Isothermal Amplification, Real-Time LAMP, and Droplet Digital PCR,” Plant Disease, pp. 1-9, Nov. 2019. [cited by applicant]
Ruppert et al., “A smartphone readout system for gold nanoparticle-based lateral flow assays: application to monitoring of digoxigenin,” Microchimica Acta, vol. 186, p. 119, Jan. 2019. [cited by applicant]
Sajid et al., “Designs, formats and applications of lateral flow assay: A literature review,” Journal of Saudi Chemical Society, vol. 19, pp. 689-705, 2015. [cited by applicant]
Shah et al., “Mobile Phone Ratiometric Imaging Enables Highly Sensitive Fluorescence Lateral Flow Immunoassays without External Optical Filters,” Anal. Chem., vol. 90, No. 11, pp. 6967-6974, Jun. 2018. [cited by applicant]
Shaw et al., “Improving Image Classifiers With Supervised Contrastive Learning.” Weights & Biases, https://wandb.ai/authors/scl/reports/Improving-Image-Classifiers-With-Supervised-Contrastive-Learning--VmlldzoxMzQwNzE; … [cited by applicant]
Snell, “Prototypical Networks for Few-shot Learning,” Advances in Neural Information Processing Systems, pp. 4077-4087, Jun. 2017. [cited by applicant]
Solmaz et al., “Quantifying colorimetric tests using a smartphone app based on machine learning classifiers,” Sensors and Actuators B: Chemical, Feb. 2018, vol. 255, Part 2, pp. 1967-1973, Elsevier. [cited by applicant]
Sun et al., “Meta-Transfer Learning for Few-Shot Learning,” Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 403-412, 2019. [cited by applicant]
Unknown, “FDA In Vitro Diagnostics EUA's—Antigen Diagnostic Tests for SARS-CoV-2,” 2021, https://www.fda.gov/medical-devices/coronavirus-disease-2019-covid-19-emergency-use-authorizations-medical-devices/in-vitro-diagno… [cited by applicant]
Unknown, “Rapid Lateral Flow Test Strips: Considerations for Product Development,” Millipore, 2013. [cited by applicant]
Wan et al., “Variational autoencoder based synthetic data generation for imbalanced learning,” IEEE Symposium Series on Computational Intelligence (SSCI), pp. 1-7, 2017, Abstract,. [cited by applicant]
Wang et al., “Generalizing from a Few Examples: A Survey on Few-Shot Learning,” ACM Computer Surveys, vol. 1, No. 1, Article 1, pp. 1:1-1:34, Mar. 2020. [cited by applicant]
Wang et al., “Learning to Learn: Model Regression Networks for Easy Small Sample Learning, ” European Conference on Computer Vision, pp. 616-634, 2016. [cited by applicant]
Wang et al., “Meta-Learning to Detect Rare Objects,” Proceedings of the IEEE International Conference on Computer Vision, pp. 9925-9934, 2019. [cited by applicant]
Whitelaw et al., “Applications of digital technology in COVID-19 pandemic planning and response,” Lancet Digital Health, vol. 2, pp. e435-e440, Jun. 2020. [cited by applicant]
Wood et al., “Taking connected mobile-health diagnostics of infectious diseases to the field,” Nature, vol. 566, No. 7745, pp. 467-474, Feb. 2019. [cited by applicant]
Wu et al., “Towards One Shot Learning by Imitation for Humanoid Robots,” Proceedings—IEEE International Conference on Robotics and Automation, pp. 2889-2894, May 2010. [cited by applicant]
Yetisen et al., “Paper-based microfluidic point-of-care diagnostic devices,” Lab on a Chip, vol. 13, pp. 2210-2251, 2013. [cited by applicant]
Abingdon Health, “Abingdon Health announces £1M Innovate UK funding for the development of a fully quantitative lateral flow Smartphone Reader,” 2 pages, Sep. 17, 2020. [cited by applicant]
Abingdon Health, “New flexible lateral flow Smartphone reader App developed,” 2 pages, Dec. 20, 2018. [cited by applicant]
Abingdon Health, “AppDx” Smartphone Reader brochure, from https://www.abingdonhealth.com/contract-services/lateral-flow-reader/appdx/ (undated). [cited by applicant]
Andryukov, “Six decades of lateral flow immunoassay: from determining metabolic markers to diagnosing COVID-19,” Aims Microbiology, vol. 6, No. 3, pp. 280-304, Aug. 2020. [cited by applicant]
Aveyard et al., “One step visual detection of PCR products with gold nanoparticles and a nucleic acid lateral flow (NALF) device,” Chemical Communications, vol. 41, No. 41, pp. 4251-4253, Dec. 2007. [cited by applicant]
Bahadir et al., “Lateral flow assays: Principles, designs and labels,” TrAC Trends in Analytical Chemistry, vol. 82, pp. 286-306, Sep. 2016. [cited by applicant]
Balan, et al., “SMARTtest: A Smartphone App to Facilitate HIV and Syphilis Self-and Partner-Testing, Interpretation of Results, and Linkage to Care,” AIDS and Behavior, 2019, vol. 24, pp. 1560-1573, https://doi.org/10.1… [cited by applicant]
Ballard et al., “Deep learning-enabled point-of-care sensing using multiplexed paper-based sensors,” NPJ Digital Medicine, vol. 3, No. 66, pp. 1-7, 2020. [cited by applicant]
Budd et al., “Digital technologies in the public-health response to COVID-19,” Nature Medicine, vol. 26, pp. 1183-1192, Aug. 2020. [cited by applicant]
Butt et al., “Deep learning system to screen coronavirus disease 2019 pneumonia.” Applied Intelligence, 2020, pp. 1-7. [cited by applicant]
Carrio, et al., “Automated Low-Cost Smartphone-Based Lateral Flow Saliva Test Reader for Drugs-of-Abuse Detection,” Sensors (Basel), 2015, vol. 15(11), pp. 29569-29593. [cited by applicant]
Chhikara, “The inverse Gaussian distribution theory: methodology, and applications,” vol. 95, CRC Press, 1988 (Description and Table of Contents). [cited by applicant]
Colubri et al., “Transforming Clinical Data into Actionable Prognosis Models: Machine-Learning Framework and Field-Deployable App to Predict Outcome of Ebola Patients, ” PLOS Neglected Tropical Diseases, 2016, vol. 10(3… [cited by applicant]
Contreras-Naranjo et al., “Mobile Phone Based Microscopy, Sensing, and Diagnostics,” IEEE Journal of Selected Topics in Quantum Electronics, vol. 22, pp. 1-14, 2015. [cited by applicant]
De Bruijne, “Machine learning approaches in medical image analysis: From detection to diagnosis,” Medical Image Analysis, Oct. 2016, vol. 33, pp. 94-97, 20th anniversary of the Medical Image Analysis journal (MedIA), El… [cited by applicant]
Dell et al., “Mobile tools for point-of-care diagnostics in the developing world,” Proceedings of the 3rd ACM Symposium on Computing for Development, Article No. 9, pp. 1-10, Jan. 2013, Abstract. [cited by applicant]
Dell et al., “Towards a point-of-care diagnostic system: automated analysis of immunoassay test data on a cell phone,” NSDR'11 Proceedings of the 5th ACM Workshop on Networked Systems for developing regions, pp. 3-8, Ju… [cited by applicant]
Deng et al., “ImageNet: A large-scale hierarchical image database,” IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009, Abstract. [cited by applicant]
Draz et al., “Virus detection using nanoparticles and deep neural network-enabled smartphone system,” Science Advances, vol. 6, No. 51, p. eabd5354, Dec. 2020. [cited by applicant]
Finn, et al., “Model-agnostic meta-learning for fast adaptation of deep networks,” arXiv preprint arXiv:1703.03400, 2017, (13 pages). [cited by applicant]
Folks et al., “The Inverse Gaussian Distribution and its Statistical Application: a Review,” J.R. Statist. Soc. B, vol. 40, No. 3, pp. 263-289, 1978. [cited by applicant]
Fuhad et al., “Deep Learning Based Automatic Malaria Parasite Detection from Blood Smear and Its Smartphone Based Application,” Diagnostics (Basel), 2020, vol. 10(5), Article 329, pp. 1-22. [cited by applicant]
Ghani et al., “Expanding the role of diagnostic and prognostic tools for infectious diseases in resource-poor settings,” Nature, vol. 528, pp. S50-S52, Dec. 2015. [cited by applicant]
Girschick, “Fast R-CNN,” IEEE International Conference on Computer Vision (ICCV), pp. 1440-1448, 2015. [cited by applicant]
He et al., “Mask R-CNN,” arXiv preprint arXiv:1703.06870, pp. 2961-2969, 2017. [cited by applicant]
He et al., “Mask R-CNN,” arXiv preprint arXiv: 1703.06870v3, pp. 1-12, 2018. [cited by applicant]
He, et al., “Mask R-CNN,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 42, Issue 2, pp. 386-397, Feb. 2020, Abstract. [cited by applicant]
Huang et al., “Gold nanoparticles: Optical properties and implementations in cancer diagnosis and photothermal therapy,” Journal of Advanced Research, vol. 1, Issue 1, pp. 13-28, Jan. 2010. [cited by applicant]
Ibitoye et al., “Home Testing Past, Present and Future: Lessons Learned and Implications for HIV Home Tests (A Review),” AIDS Behav., vol. 18, No. 5, pp. 933-949, May 2014. [cited by applicant]
International Search Report and Written Opinion mailed Mar. 1, 2022 for International Patent Application No. PCT/US2021/053708. [cited by applicant]
Joung et al., “Point-of-Care Serodiagnostic Test for Early-Stage Lyme Disease Using a Multiplexed Paper-Based Immunoassay and Machine Learning,” ACS Nano., vol. 14, No. 1, pp. 229-240, Jan. 2020, Abstract. [cited by applicant]
Kang et al., “Few-shot Object Detection via Feature Reweighting,” Proceedings of the IEEE International Conference on Computer Vision, pp. 8420-8429, 2019. [cited by applicant]
Ker et al., “Deep Learning Applications in Medical Image Analysis,” IEEE Access, 2018, vol. 6, pp. 9375-9389. [cited by applicant]
Khosla et al., “Supervised Contrastive Learning,” 34th Conference on Neural Information Processing Systems, 2020. [cited by applicant]
Kim et al., “Colorimetric analysis of saliva-alcohol test strips by smartphone-based instruments using machine-learning algorithms,” Applied Optics, 2017, vol. 56(1), pp. 84-92, OSA Publishing, Abstract. [cited by applicant]
Kingma et al., “Auto-Encoding Variational Bayes,” CoRR, abs/1312.6114, 2013. [cited by applicant]
Kiran et al., “Automatic Rectification of Perspective Distortion From a Single Image Using Plane Homography,” International Journal on Computational Sciences & Applications, vol. 3, No. 5, pp. 47-58, Oct. 2013. [cited by applicant]
Kittler, “On the accuracy of the Sobel edge detector,” Image and Vision Computing, vol. 1, Issue 1, pp. 37-42, Feb. 1983. [cited by applicant]
Koczula et al., “Lateral flow assays,” Essays in Biochemistry, vol. 60, pp. 111-120, Jun. 2016. [cited by applicant]
Kong et al., “Highly Stable and Sensitive Nucleic Acid Amplification and Cell-Phone-Based Readout,” ACS Nano, vol. 11, No. 3, pp. 2934-2943, Feb. 2017, Abstract. [cited by applicant]
Kose, et al., “Utilizing Machine Learning for Image Quality Assessment for Reflectance Confocal Microscopy,” Journal of Investigative Dermatology, Jun. 2020, vol. 140(6), pp. 1214-1222, Elsevier. [cited by applicant]
Lake et al., “Human-level concept learning through probabilistic program induction,” Science, vol. 350, Issue 6266, pp. 1332-1338, Dec. 2015. [cited by applicant]
McAulay et al., “Retrospective clinical evaluation of 4 lateral flow assays for the detection of SARS-CoV-2 IgG,” Diagnostic Microbiology and Infectious Disease, vol. 98, p. 115161, Aug. 2020. [cited by applicant]
Mei et al., “Artificial intelligence-enabled rapid diagnosis of patients with COVID-19,” Natural Medicine, 2020, vol. 26, pp. 1224-1228, Nature. [cited by applicant]
Minaee et al., “Image Segmentation Using Deep Learning: A Survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, Issue 7, Jul. 2022, Abstract. [cited by applicant]
Murray et al., “Digital public health and COVID-19,” Lancet Public Health, vol. 9, pp. e469-e470, Sep. 2020. [cited by applicant]
Parolo et al., “Tutorial: design and fabrication of nanoparticle-based lateral-flow immunoassays,” Nature Protocols, vol. 15, Suppl. 1, pp. 3788-3816, Oct. 2020. [cited by applicant]
Pathania, et al., “Point-of-care cervical cancer screening using deep learning-based microholography,” Theranostics, 2019, vol. 9(26), pp. 8438-8447, Ivyspring International Publishing. [cited by applicant]
Posthuma-Trumpie et al., “Lateral flow (immuno) assay: its strengths, weaknesses, opportunities and threats. A literature survey,” Anal Bioanal Chem, vol. 393, 569-582 (2009). [cited by applicant]
Potluri et al., “An inexpensive smartphone-based device for point-of-care ovulation testing,” Lab Chip., vol. 19, No. 1, pp. 59-67, Dec. 2018. [cited by applicant]