Granted Patent B2

US 12,387,131 · App. 15/994,964 · Granted Aug 12, 2025

Inventors: Santosh Chandwani (Redmond, WA); Ameet Vijay Joshi (Redmond, WA); Amit Martu Kamat (Sammamish, WA); Raveendmathan Loganathan (Sammamish, WA); Veera Venkata Stya Sridhar Maddipati (Issaquah, WA)

Assignee: Microsoft Technology Licensing, LLC

G06N20/00G06F16/288G06F18/214G06F18/217G06F18/24G06N7/01

Abstract

Claims (96)

1. A computerized system comprising:

a processor; and

computer storage memory having computer-executable instructions stored thereon which, when executed by the processor, implement a method comprising:

receiving a corpus that includes a plurality of records, a first record of the plurality of records encoding a value for a data element (DE) of a plurality of DEs encoded in the plurality of records;

determining, based on a structure associated with the plurality of records, a first relationship of a plurality of relationships records of the plurality of records, the first relationship of the plurality of relationships being indicative of a level of statistical confidence of the first relationship between the first record and a second record;

receiving a user selection that indicates the DE is a DE of interest (DEOI) of the plurality of DEs;

receiving an additional user selection that indicates a first value and a second value associated with the DEOI, where the first value and the second value include representative values of the DEOI useable to determine classification labels during training of a predictive model (PM);

determining a relevant set of records of the plurality of records based on the plurality of relationships and the DEOI, each record included in the relevant set of records including the DEOI or is related to another record of the relevant set of records that includes the DEOI via the plurality of relationships;

training the PM based on the set of relevant records, the DEOI, the first value of the DEOI, and the second value of the DEOI, the trained PM is trained, based on input records that do not include the first value and the second value of the DEOI and based on the additional user selection that indicates the first value and the second value of the DEOI, to predict at least one value for the DEOI of an incomplete record that does not include the first and second value of the DEOI; and

providing the trained PM to a user.

2. The system of claim 1 , wherein the PM is a classifier machine learning (ML) model, the first value of the DEOI is classified as a first outcome associated with the DEOI, the second value of the DEOI is classified as a second outcome associated with the DEOI, and training the ML model includes supervised training based on a plurality of values for the DEOI included in the relevant set of records, the first outcome associated with the DEOI, and the second outcome associated with the DEOI.

3. The system of claim 1 , wherein the method further comprises:

generating a graph that includes a plurality of nodes and a plurality of edges, each node of the plurality of nodes corresponding to a particular record of the plurality of records and each edge of the plurality of edges corresponds to a particular relationship of the plurality of relationships between records corresponding to connected nodes in the graph;

generating a connected subgraph of the graph based on the DEOI, each node of the subgraph corresponding to a record of the plurality of records that includes the DEOI or is directly or indirectly connected to another node of the plurality of nodes that corresponds to another record of the plurality of records that includes the DEOI via the plurality of edges; and

traversing the subgraph to determine the relevant set of records, each record included in the relevant set of records corresponding to a node included in the subgraph.

4. The system of claim 1 , wherein the method further comprises:

determining, based on the set of relevant records, a set of potentially predictive DEs (PPDEs), each PPDE in the set of PPDEs is included in at least a portion of the records of the relevant set; and

pruning the set of PPDEs based on a DE density requirement.

5. The system of claim 1 , wherein the method further comprises:

determining, based on the relevant set of records, a set of potentially predictive DEs (PPDEs), each PPDE in the set of PPDEs is included in at least a portion of the records of the relevant set;

determining a significance of each PPDE within the set of PPDEs, with respect to the DEOI;

pruning the set of PPDEs based on the significance of each PPDE included the set of PPDEs and a correlation threshold; and

generating a set of significantly predictive DEs (SPDEs) based on the pruned set of PPDEs.

6. The system of claim 1 , wherein the method further comprises:

training a plurality of PMs, each PM included in the plurality of PMs corresponding to a separate value for a hyper-parameter of a PM type;

validating each PM of the plurality of PMs;

determining an accuracy metric for each PM of the plurality of PMs, the accuracy metric for a particular PM of the plurality of PMs being based on validating the particular PM; and

selecting an optimal PM of the plurality of PMs based on the accuracy metric for each PM of the plurality of PMs.

7. The system of claim 1 , wherein the method further comprises:

featurizing the relevant set of records;

generating a training data set based on a portion of the featurized relevant set of records;

generating a validation data set based on another portion of the featurized relevant data set;

training the PM based on the training data set;

validating the trained PM based on the validation data set;

determining a key performance indicator (KPI) of the trained PM based on validating the trained PM; and

providing the KPI to the user.

8. A method comprising:

receiving a corpus that includes a plurality of records, a first record of the plurality of records encoding a value for a data element (DE) of a plurality of DEs encoded in the plurality of records;

determining, based on a structure associated with the plurality of records, a first relationship of a plurality of relationships between the first record and at least one other record of the plurality of records, the first relationship of the plurality of relationships being indicative of a level of statistical confidence of the first relationship;

receiving, through a user interface displayed on a user device, a user selection that indicates a DE of interest (DEOI) of the plurality of DEs;

receiving, through the user interface, an additional user selection that indicates a first value and a second value of the DEOI, where the first value and the second value include representative values of the DEOI useable to determine classification labels during training of a predictive model (PM);

determining a relevant set of records of the plurality of records based on the plurality of relationships and the DEOI, each record included in the relevant set of records including the DEOI or is related to another record of the relevant set of records that includes the DEOI via the plurality of relationships;

training the PM based on the set of relevant records, the DEOI, the first value of the DEOI, and the second value of the DEOI, the trained PM is trained, based on input records that do not include the first value and the second value of the DEOI and based on the additional user selection that indicates the first value and the second value of the DEOI, to predict at least one value for the DEOI of an incomplete record that does not include the first and second value of the DEOI; and

providing the trained PM to a user.

9. The method of claim 8 , wherein the PM is a classifier machine learning (ML) model, the first value of the DEOI is classified as a first outcome associated with the DEOI, the second value of the DEOI is classified as a second outcome associated with the DEOI, and training the ML model includes supervised training based on a plurality of values for the DEOI included in the relevant set of records, the first outcome associated with the DEOI, and the second outcome associated with the DEOI.

10. The method of claim 8 , further comprising:

generating a graph that includes a plurality of nodes and a plurality of edges, each node of the plurality of nodes corresponding to a particular record of the plurality of records and each edge of the plurality of edges corresponds to a particular relationship of the plurality of relationships between records corresponding to connected nodes in the graph;

generating a connected subgraph of the graph based on the DEOI, each node of the subgraph corresponding to a record of the plurality of records that includes the DEOI or is directly or indirectly connected to another node of the plurality of nodes that corresponds to another record of the plurality of records that includes the DEOI via the plurality of edges; and

traversing the subgraph to determine the relevant set of records, each record included in the relevant set of records corresponding to a node included in the subgraph.

11. The method of claim 8 , further comprising:

determining, based on the relevant set of records, a set of potentially predictive DEs (PPDEs), each PPDE in the set of PPDEs is included in at least a portion of the records of the relevant set; and

pruning the set of PPDEs based on a DE density requirement.

12. The method of claim 8 , further comprising:

determining, based on the relevant set of records, a set of potentially predictive DEs (PPDEs), each PPDE in the set of PPDEs is included in at least a portion of the records of the relevant set;

determining a significance of each PPDE within the set of PPDEs, with respect to the DEOI;

pruning the set of PPDEs based on the significance of each PPDE included in the set of PPDEs and a correlation threshold; and

generating a set of significantly predictive DEs (SPDEs) based on the pruned set of PPDEs.

13. The method of claim 8 , further comprising:

training a plurality of PMs, each PM included in the plurality of PMs corresponding to a separate value for a hyper-parameter of a PM type;

validating each PM of the plurality of PMs;

determining an accuracy metric for each PM of the plurality of PMs, the accuracy metric for a particular PM of the plurality of PMs being based on validating the particular PM; and

selecting an optimal PM of the plurality of PMs based on the accuracy metric for each PM of the plurality of PMs.

14. The method of claim 8 , further comprising:

featurizing the relevant set of records;

generating a training data set based on a portion of the featurized relevant set of records;

generating a validation data set based on another portion of the featurized relevant data set;

training the PM based on the training data set;

validated the trained PM based on the validation data set;

determining a key performance indicator (KPI) of the trained PM based on validating the trained PM; and

providing the KPI to the user.

15. A non-transitory computer-readable media having instructions stored thereon, wherein the instructions, when executed by a processor of a computing device, cause the computing device to perform actions including:

receiving a corpus that includes a plurality of records, a first record of the plurality of records encoding a value for a data elements (DE) of a plurality of DEs encoded in the plurality of records;

determining, based on a structure of the plurality of records, a first relationship between individual the first record and a second record of the plurality of records, the first relationship indicative of a level of statistical confidence of the first relationship;

receiving, through a user interface displayed on a user device, a user selection that indicates a DE of interest (DEOI) of the plurality of DEs;

receiving, through the user interface displayed on a user device, an additional user selection that indicates a first value and a second value of the DEOI, where the first value and the second value include representative values if the DEOI useable to determine classification labels during training of a predictive model (PM);

determining a relevant set of records of the plurality of records based on the plurality of relationships and the DEOI, each record included in the relevant set of records including the DEOI or is related to another record of the relevant set of records that includes the DEOI via the plurality of relationships;

training the PM based on the set of relevant records, the DEOI, the first value of the DEOI, and the second value of the DEOI, the trained PM is trained, based on input records that do not include the first value and the second value of the DEOI and based on the additional user selection that indicates the first value and the second value of the DEOI, to predict at least one value for the DEOI of an incomplete record that does not include the first and second value of the DEOI; and

providing the trained PM to a user.

16. The computer-readable media of claim 15 , wherein the PM is a classifier machine learning (ML) model, the first value of the DEOI is classified as a first outcome associated with the DEOI, the second value of the DEOI is classified as a second outcome associated with the DEOI, and training the ML model includes supervised training based on a plurality of values for the DEOI included in the relevant set of records, the first outcome associated with the DEOI, and the second outcome associated with the DEOI.

17. The computer-readable media of claim 15 , the actions further comprising:

generating a graph that includes a plurality of nodes and a plurality of edges, each node of the plurality of nodes corresponding to a particular record of the plurality of records and each edge of the plurality of edges corresponds to a particular relationship of the plurality of relationships between records corresponding to connected nodes in the graph;

generating a connected subgraph of the graph based on the DEOI, each node of the subgraph corresponding to a record of the plurality of records that includes the DEOI or is directly or indirectly connected to another node of the plurality of nodes that corresponds to another record of the plurality of records that includes the DEOI via the plurality of edges; and

traversing the subgraph to determine the relevant set of records, each record included in the relevant set of records corresponding to a node included in the subgraph.

18. The computer-readable media of claim 15 , the actions further comprising:

determining, based on the relevant set of records, a set of potentially predictive DEs (PPDEs), each PPDE in the set of PPDEs is included in at least a portion of the records of the relevant set; and

pruning the set of PPDEs based on a DE density requirement.

19. The computer-readable media of claim 15 , the actions further comprising:

determining, based on the relevant set of records, a set of potentially predictive DEs (PPDEs), each PPDE in the set of PPDEs is included in at least a portion of the records of the relevant set;

determining a significance of each PPDE within the set of PPDEs, with respect to the DEOI;

pruning the set of PPDEs based on the significance of each PPDE included in the set of PPDEs and a correlation threshold; and

generating a set of significantly predictive DEs (SPDEs) based on the pruned set of PPDEs.

20. The computer-readable media of claim 15 , the actions further comprising:

training a plurality of PMs, each PM included in the plurality of PMs corresponding to a separate value for a hyper-parameter of a PM type;

validating each PM of the plurality of PMs;

determining an accuracy metric for each PM of the plurality of PMs, the accuracy metric for a particular PM of the plurality of PMs being based on validating the particular PM; and

selecting an optimal PM of the plurality of PMs based on the accuracy metric for each PM of the plurality of PMs.

Assignments (1)

Continuity (1)

References Cited (18)