Granted Patent B2

US 12,536,773 · App. 18/343,825 · Granted Jan 27, 2026

Inventors: Laura D. Hamilton (Chicago, IL); Vinit Garg (Fremont, CA); Ayush Tomar (Morgan Hill, CA); Fazle Shahnawaz Muhibul Karim (Chicago, IL); Chenwei Liu (Rockville, MD)

Assignee: Optum, Inc.

G06V10/764G06V10/774G16H10/60G16H70/40

Abstract

Claims (69)

1 . A method for identifying pills performed by one or more processors, the method comprising:

receiving a pill identification request including one or more images of a pill and an identifier of a user associated with the pill;

generating one or more image embeddings based on the one or more images;

retrieving, using the identifier of the user, claims data for the user;

encoding the claims data to generate a claims embedding;

identifying the pill based on the one or more image embeddings and the claims embedding; and

generating and transmitting a response to the pill identification request based on the identifying.

2 . The method of claim 1 , wherein the one or more image embeddings are generated using a first machine learning system including a plurality of fine-tuned models, and generating the one or more image embeddings comprises:

augmenting the one or more images to generate one or more augmented images;

generating, using the plurality of fine-tuned models, one or more sets of a plurality of image embeddings for the one or more augmented images; and

averaging the plurality of image embeddings within each of the one or more sets to generate the one or more image embeddings.

3 . The method of claim 2 , wherein augmenting each of the one or more images comprises:

sequentially augmenting each of the one or more images a predefined number of times based on a predefined sequence of augmentation types.

4 . The method of claim 1 , wherein the pill is identified using a second machine learning system including a trained model, and identifying the pill comprises:

concatenating the one or more image embeddings and the claims embedding to generate a concatenated embedding;

generating, using the trained model, a probability distribution over a plurality of pill identifiers indicating a likelihood the pill corresponds to each of the plurality of pill identifiers based on the concatenated embedding; and

identifying the pill based on a pill identifier from the plurality of pill identifiers that the pill has a highest likelihood of corresponding to based on the probability distribution.

5 . The method of claim 4 , further comprising:

determining the likelihood of the pill corresponding to the pill identifier is above a predefined threshold.

6 . The method of claim 4 , wherein the pill identifier is a code that indicates a formula, a dosage, and a form of the pill.

7 . The method of claim 6 , generating the response comprises:

including one or more of the formula, the dosage, or the form of the pill in the response.

8 . The method of claim 1 , wherein the claims data includes one or more diagnoses, one or more procedures, or one or more pharmaceutical claims associated with the user.

9 . The method of claim 1 , wherein encoding the claims data comprises:

using at least one of multi-hot encoding, a latent dimension of an autoencoder, or a dimension reduction technique to encode the claims data.

10 . The method of claim 1 , wherein the one or more image embeddings are generated using a first machine learning system including a plurality of fine-tuned models, and the plurality of fine-tuned models are generated by:

receiving a pre-trained source model;

generating a target model based on the pre-trained source model;

receiving, as first training data, a plurality of samples and corresponding labels, wherein each sample of the plurality of samples includes one or more training images of a pill and a respective corresponding label includes a pill identifier for the pill;

dividing the first training data into a plurality of folds to generate a plurality of first training data subsets; and

generating, from the target model, each of the plurality of fine-tuned models using one of the plurality of first training data subsets.

11 . The method of claim 1 , wherein the pill is identified using a second machine learning system including a trained model, and the trained model is generated by:

receiving, as second training data, a plurality of samples and corresponding labels, wherein each sample of the plurality of samples includes one or more training images of a pill and training claims data for a user having a pill identifier corresponding to the pill, and a respective corresponding label includes the pill identifier for the pill; and

training a model based on at least a portion of the plurality of samples and corresponding labels to output a probability distribution over a plurality of pill identifiers indicating a likelihood a pill included in an image corresponds to each of the plurality of pill identifiers.

12 . The method of claim 1 , wherein:

the one or more images of the pill include a first image of a front side of the pill and a second image of a back side of the pill, and

the one or more image embeddings include a first image embedding generated based on the first image and a second image embedding generated based on the second image.

13 . A system for pill identification, the system comprising:

one or more processors; and

at least one memory storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including:

receiving a pill identification request including one or more images of a pill and an identifier of a user associated with the pill;

generating one or more image embeddings based on the one or more images;

retrieving, using the identifier of the user, claims data for the user;

encoding the claims data to generate a claims embedding;

identifying the pill based on the one or more image embeddings and the claims embedding; and

generating and transmitting a response to the pill identification request based on the identifying.

14 . The system of claim 13 , wherein the one or more image embeddings are generated using a first machine learning system including a plurality of fine-tuned models, and generating the one or more image embeddings includes:

augmenting the one or more images to generate one or more augmented images;

generating, using the plurality of fine-tuned models, one or more sets of a plurality of image embeddings for the one or more augmented images; and

averaging the plurality of image embeddings within each of the one or more sets to generate the one or more image embeddings.

15 . The system of claim 13 , wherein the pill is identified using a second machine learning system including a trained model, and identifying the pill comprises:

concatenating the one or more image embeddings and the claims embedding to generate a concatenated embedding;

generating, using the trained model, a probability distribution over a plurality of pill identifiers indicating a likelihood the pill corresponds to each of the plurality of pill identifiers based on the concatenated embedding; and

identifying the pill based on a pill identifier from the plurality of pill identifiers that the pill has a highest likelihood of corresponding to based on the probability distribution.

16 . The system of claim 15 , wherein the pill identifier is a code that indicates a formula, a dosage, and a form of the pill, and generating the response includes:

including one or more of the formula, the dosage, or the form of the pill in the response.

17 . The system of claim 13 , wherein the claims data includes one or more diagnoses, one or more procedures, or one or more pharmaceutical claims associated with the user.

18 . The system of claim 13 , wherein encoding the claims data includes:

using at least one of multi-hot encoding, a latent dimension of an autoencoder, or a dimension reduction technique to encode the claims data.

19 . The system of claim 13 , wherein:

the one or more images of the pill include a first image of a front side of the pill and a second image of a back side of the pill, and

the one or more image embeddings include a first image embedding generated based on the first image and a second image embedding generated based on the second image.

20 . A non-transitory computer readable medium for identifying pills, the non-transitory computer readable medium storing instructions which, when executed by one or more processors, cause the one or more processors to perform operations comprising:

receiving a pill identification request including one or more images of a pill and an identifier of a user associated with the pill;

generating one or more image embeddings based on the one or more images;

retrieving, using the identifier of the user, claims data for the user;

encoding the claims data to generate a claims embedding;

identifying the pill based on the one or more image embeddings and the claims embedding; and

generating and transmitting a response to the pill identification request based on the identifying.

Assignments (1)

Continuity (1)

References Cited (9)