IP Library Granted Patent US 11,907,332
Granted Patent B2
US 11,907,332 · App. 17/575,533 · Granted Feb 20, 2024

Systems and methods for machine learning models for performance measurement

Inventors: Nathaniel Freese (San Francisco, CA); Meera Rao (San Francisco, CA); Rick Wolf (San Francisco, CA); Peyton Rose (San Francisco, CA); Stephen Martin (San Francisco, CA); Sameer Soi (San Francisco, CA); Zachary Taylor (San Francisco, CA); Ye Wang (San Francisco, CA)
Assignee: Included Health, Inc.
G06F18/2113G06F16/2443G06F16/2465G06F18/10G06F18/2155G06N20/00G06Q30/0282
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,907,332
App. No.
17/575,533
Granted
Feb 20, 2024
Kind
B2
Abstract

Methods, systems, and computer-readable media for generating a statistically covaried machine learning model for performance measurement of service providers. The method receives a configuration file that includes one or more parameters associated with a plurality of individuals and parses it to generate and executing the database query on input data to generate sets of tabulated data of individuals of the plurality of individuals. The method next determines one or more measures of service providers listed in the configuration file using two or more tabulated data of individuals from the sets of tabulated data of individuals. The method finally generates a covaried machine learning model by training a machine learning model by statistically covarying measures and using them as training data.

Claims (67)

1. A non-transitory computer readable medium including instructions that are executable by one or more processors to cause a system to perform a method for determining matching sets of individuals using performance measurements of service providers associated with sets of individuals, the method comprising:

receiving a configuration file that includes one or more parameters associated with a plurality of individuals;

parsing the configuration file to generate a database query;

executing the database query on input data to generate sets of tabulated data of individuals of the plurality of individuals in the input data, wherein the sets of tabulated data of individuals satisfy values of the one or more parameters associated with the plurality of individuals;

determining a first set of measures of a first set of service providers listed in the configuration file using a first set of tabulated data of a first set of individuals from the sets of tabulated data of individuals, wherein the sets of tabulated data of individuals includes data representing the service providers; and

determining a second set of individuals matching the first set of individuals by determining a second set of tabulated data of the second set of individuals includes matching cohort of individuals to the first set of individuals, wherein the first set of measures of the first set of service providers using the first set of tabulated data of the first set of individuals matches the first set of measures of a second set of service providers using the second set of tabulated data of the second set of individuals.

2. The non-transitory computer readable medium of claim 1 further comprises:

determining a second set of measures of the second set of service providers associated with the second set of individuals using the second set of measures of the first set of service providers associated with the first set of individuals, wherein the second set of measures of the second set of service providers associated with the first set of individuals is calculated using values of the one or more parameters associated with the first set of individuals, and the one or more parameters have missing values for the second set of individuals.

3. The non-transitory computer readable medium of claim 2 further comprises:

the second set of measures of the second set of service providers associated with the second set of individuals is predicted using a machine learning model using as input the second set of measures of the first set of service providers of the first set of individuals.

4. The non-transitory computer readable medium of claim 3 , wherein a machine learning model using as input the second set of measures of the first set of service providers of the first set of individuals further comprises:

generating a covaried machine learning model using a machine learning platform, wherein the covaried machine learning model is generated by training a machine learning model on the machine learning platform using the first set of measures of the first set of service providers of the first set of individuals as training data, wherein the first set of measures of the first set of service providers of the first set of individuals are statistically covaried before providing them as input to the machine learning model, and further providing as input for training the first set of measures of the second set of service providers of the second set of individuals

executing the covaried machine learning model using as input the second set of measures of the first set of service providers of the first set of individuals.

5. The non-transitory computer readable medium of claim 4 further comprises:

extracting features from the first set of tabulated data of the first set of individuals;

covarying features to generate covaried features;

training the machine learning model using covaried features as input;

predicting adjusted measures, wherein the adjusted measures are predicted by executing the covaried machine learning model; and

storing the predicted adjusted measures in a data storage.

6. The non-transitory computer readable medium of claim 4 further comprises:

generating one or more metrics indicating performance of the covaried machine learning model, wherein the one or more metrics are generated by executing the covaried machine learning model.

7. The non-transitory computer readable medium of claim 6 further comprises:

supplying the covaried machine learning model to a search engine for the service providers; and

adjusting results of the search engine for the service providers based on the one or more metrics indicating performance of the covaried machine learning model.

8. The non-transitory computer readable medium of claim 4 , wherein parsing the configuration file to generate a database query further comprises:

retrieving a stored database query from a data storage, wherein a measure in the configuration file matches a measure associated with the stored database query.

9. The non-transitory computer readable medium of claim 8 , further comprises:

pre-determining the first set of measures based on stored database queries in the data storage;

covarying the pre-determined first set of measures; and

storing the covaried pre-determined first set of measures in the data storage.

10. The non-transitory computer readable medium of claim 9 further comprises:

retrieving the stored covaried pre-determined first set of measures in the data storage; and

providing the covaried pre-determined first set of measures as input to the machine learning model.

11. The non-transitory computer readable medium of claim 1 , wherein parsing the configuration file to generate a database query, further comprises:

accessing one or more query templates stored in a data storage, wherein the one or more query templates map to the one or more parameters;

populating the one or more query templates using values, wherein the values are obtained from the configuration file;

generating a job to be added to a queue of a job server; and

submitting the job to a query server to generate the database query.

12. The non-transitory computer readable medium of claim 1 , wherein one or more parameters associated with a plurality of individuals further comprises:

a first set of parameters to define a population of individuals; and

a second set of parameters to determine individuals of the defined population of individuals to be included in the first set of tabulated data of the first set of individuals of the sets of tabulated data of individuals.

13. The non-transitory computer readable medium of claim 1 , wherein determining a first set of measures of a first set of service providers listed in the configuration file using the first set of tabulated data of the first set of individuals further comprises:

parsing the configuration file to generate a second database query;

executing the second database query on the input data to generate a third set of tabulated data of the first set of individuals; and

determining a measure of the first set of measures using the first set of tabulated data of the first set of individuals, wherein the first set of tabulated data of the first set of individuals comprises of the third set of tabulated data of the first set of individuals.

14. The non-transitory computer readable medium of claim 13 , wherein executing the second database query on the input data further comprises:

providing the sets of tabulated data of individuals as the input data for the second database query.

15. The non-transitory computer readable medium of claim 1 , wherein determining first set of measures of a first set of service providers listed in the configuration file using a first set of tabulated data of a first set of individuals further comprises:

generating a table of the first set of measures of the first set of service providers and pointers to the first set of individuals.

16. The non-transitory computer readable medium of claim 1 , wherein the values of the one or more parameters associated with the plurality of individuals comprises numbers, strings, or boolean values.

17. The non-transitory computer readable medium of claim 1 , wherein the sets of tabulated data of individuals satisfy values of the one or more parameters associated with the plurality of individuals further comprises:

aggregating the data of individuals that satisfy values of the parameters associated with the plurality of individuals.

18. The non-transitory computer readable medium of claim 1 , wherein the one or more parameters included in the configuration file are pairs of keys and values.

19. A method performed by a system for determining matching set of individuals using performance measurements of service providers associated with sets of individuals, the method comprising:

receiving a configuration file that includes one or more parameters associated with a plurality of individuals;

parsing the configuration file to generate a database query;

executing the database query on input data to generate sets of tabulated data of individuals of the plurality of individuals in the input data, wherein the sets of tabulated data of individuals satisfy values of the one or more parameters associated with the plurality of individuals;

determining a first set of measures of a first set of service providers listed in the configuration file using a first set of tabulated data of a first set of individuals from the sets of tabulated data of individuals, wherein the sets of tabulated data of individuals includes data representing the service providers; and

determining a second set of individuals matching the first set of individuals by determining a second set of tabulated data of the second set of individuals includes matching cohort of individuals to the first set of individuals, wherein the first set of measures of the first set of service providers using the first set of tabulated data of the first set of individuals matches the first set of measures of a second set of service providers using the second set of tabulated data of the second set of individuals.

20. A matching cohort identification system comprising:

one or more memory devices storing processor-executable instructions; and

one or more processors configured to execute instructions to cause the machine learning model generation system to perform:

receiving a configuration file that includes one or more parameters associated with a plurality of individuals;

parsing the configuration file to generate a database query;

executing the database query on input data to generate sets of tabulated data of individuals of the plurality of individuals in the input data, wherein the sets of tabulated data of individuals satisfy values of the one or more parameters associated with the plurality of individuals;

determining a first set of measures of a first set of service providers listed in the configuration file using a first set of tabulated data of a first set of individuals from the sets of tabulated data of individuals, wherein the sets of tabulated data of individuals includes data representing the service providers; and

determining a second set of individuals matching the first set of individuals by determining a second set of tabulated data of the second set of individuals includes matching cohort of individuals to the first set of individuals, wherein the first set of measures of the first set of service providers using the first set of tabulated data of the first set of individuals matches the first set of measures of a second set of service providers using the second set of tabulated data of the second set of individuals.

Assignments (3)
CHANGE OF NAME Recorded Jul 1, 2022
From: GRAND ROUNDS, INC.
To: INCLUDED HEALTH, INC.
Reel/Frame 060425/0892 →
CORRECTIVE ASSIGNMENT TO CORRECT THE ASSISGNEE ADDRESS PREVIOUSLY RECORDED AT REEL: 058652 FRAME: 0845. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT . Recorded Jan 20, 2022
From: FREESE, NATHANIEL; RAO, MEERA; WOLF, RICK; ROSE, PEYTON; MARTIN, STEPHEN; SOI, SAMEER; TAYLOR, ZACHARY; WANG, YE
To: GRAND ROUNDS, INC.
Reel/Frame 058788/0677 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 13, 2022
From: FREESE, NATHANIEL; RAO, MEERA; WOLF, RICK; ROSE, PEYTON; MARTIN, STEVE; SOI, SAMEER; TAYLOR, ZACHARY; WANG, YE
To: GRAND ROUNDS, INC.
Reel/Frame 058652/0845 →
Continuity (2)
Continuation 17209769 · Mar 23, 2021
Related Publication 20220309286A1 · Sep 29, 2022