Molecular evidence platform for auditable, continuous optimization of variant interpretation in genetic and genomic testing and analysis
Disclosed herein are system, method, and computer program product embodiments for optimizing the determination of a phenotypic impact of a molecular variant identified in molecular tests, samples, or reports of subjects by way of regularly incorporating, updating, monitoring, validating, selecting, and auditing the best-performing evidence models for the interpretation of molecular variants across a plurality of evidence classes.
1. A computer implemented method for predicting a phenotypic impact of a molecular variant of interest, the method comprising:
obtaining an evidence model comprising evidence data, wherein the evidence data comprises objects, algorithms, and/or functions that yield predictions of phenotypic impacts of molecular variants for a target entity;
determining validation performance data for the evidence model based on production data, wherein the production data represents a first plurality of molecular variants with associated phenotypic impacts derived from clinical data and/or population data, and wherein the validation performance data corresponds to a uniform set of performance metrics computed using the production data;
determining test performance data for the evidence model based on the evidence data and test data, wherein the test data comprises a second plurality of molecular variants with associated phenotypic impacts derived from clinical data and/or population data, wherein the second plurality of molecular variants are disjoint from the first plurality of molecular variants, and wherein the test performance data corresponds to the uniform set of performance metrics computed using the test data; and
generating a ranking for the target entity based on the validation performance data and/or the test performance data, the ranking comprising the evidence model and one or more additional evidence models.
2. The method of claim 1 , wherein the target entity comprises a functional element, molecule or molecular variant, and a phenotype of interest.
3. The method of claim 1 , further comprising providing a predicted phenotypic impact of a molecular variant of interest for the target entity using a best-performing evidence model for the target entity based on the ranking in response to a query for the predicted phenotypic impact of the molecular variant of interest for the target entity.
4. The method of claim 1 , wherein the production data is received from a clinical knowledgebase.
5. The method of claim 1 , wherein the determining the validation performance data comprises:
calculating a phenotype impact score for one or more molecular variants of the target entity in the production data using the evidence model and a model validation technique; and
generating the validation performance data based on the phenotype impact scores using the uniform set of performance metrics.
6. The method of claim 1 , wherein the determining the test performance data comprises:
calculating a phenotype impact score for one or more molecular variants of the target entity in the test data using the evidence model and a model validation technique; and
generating the test performance data based on the phenotype impact scores using the uniform set of performance metrics.
7. The method of claim 1 , further comprising:
generating the evidence model based on the production data, wherein generating the evidence model comprises using a machine learning technique, a functional assay, a biophysical simulation, or a combination thereof.
8. The method of claim 7 , wherein the machine learning technique is unsupervised, supervised, or semi-supervised.
9. The method of claim 1 , wherein the uniform set of performance metrics comprises one or more diagnostic metrics, classification metrics, or regression accuracy metrics.
10. The method of claim 9 , wherein the diagnostic metrics comprises one or more of: raw accuracy, balanced accuracy, true positive rate, true negative rate, positive predictive value, negative predictive value, true positive, true negative, false positive, false negative, and coverage.
11. A system for predicting a phenotypic impact of a molecular variant of interest, the system comprising:
at least one computer hardware processor; and
at least one non-transitory computer readable storage medium storing processor executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to:
obtain an evidence model comprising evidence data, wherein the evidence data comprises objects, algorithms, and/or functions that yield predictions of phenotypic impacts of molecular variants for a target entity;
determine validation performance data for the evidence model based on production data, wherein the production data represents a first plurality of molecular variants with associated phenotypic impacts derived from clinical data and/or population data, and wherein the validation performance data corresponds to a uniform set of performance metrics computed using the production data;
determine test performance data for the evidence model based on the evidence data and test data, wherein the test data comprises a second plurality of molecular variants with associated phenotypic impacts derived from clinical data and/or population data, wherein the second plurality of molecular variants are disjoint from the first plurality of molecular variants, and wherein the test performance data corresponds to the uniform set of performance metrics computed using the test data; and
generate a ranking for the target entity based on the validation performance data and/or the test performance data, the ranking comprising the evidence model and one or more additional evidence models.
12. The system of claim 11 , wherein the at least one non-transitory computer readable storage medium stores processor executable instructions that cause the at least one computer hardware processor to provide a predicted phenotypic impact of a molecular variant of interest for the target entity using a best-performing evidence model for the target entity based on the ranking in response to a query for the predicted phenotypic impact of the molecular variant of interest for the target entity.
13. The system of claim 11 , wherein the target entity comprises a functional element, molecule or molecular variant, and a phenotype of interest.
14. The system of claim 11 , wherein the at least one non-transitory computer readable storage medium stores processor executable instructions that cause the at least one computer hardware processor to determine the validation performance data by calculating a phenotype impact score for one or more molecular variants of the target entity in the production data using the evidence model and a model validation technique; and, generating the validation performance data based on the phenotype impact scores using the uniform set of performance metrics.
15. The system of claim 11 , wherein the at least one non-transitory computer readable storage medium stores processor executable instructions that cause the at least one computer hardware processor to determine the test performance data by calculating a phenotype impact score for one or more molecular variants of the target entity in the test data using the evidence model and a model validation technique; and, generating the test performance data based on the phenotype impact scores using the uniform set of performance metrics.
16. At least one non-transitory computer readable storage medium storing processor executable instructions that, when executed by at least one computer hardware processor, cause the at least one computer hardware processor to:
obtain an evidence model comprising evidence data, wherein the evidence data comprises objects, algorithms, and/or functions that yield predictions of phenotypic impacts of molecular variants for a target entity;
determine validation performance data for the evidence model based on production data, wherein the production data represents a first plurality of molecular variants with associated phenotypic impacts derived from clinical data and/or population data, and wherein the validation performance data corresponds to a uniform set of performance metrics computed using the production data;
determine test performance data for the evidence model based on the evidence data and test data, wherein the test data comprises a second plurality of molecular variants with associated phenotypic impacts derived from clinical data and/or population data, wherein the second plurality of molecular variants are disjoint from the first plurality of molecular variants, and wherein the test performance data corresponds to the uniform set of performance metrics computed using the test data; and
generate a ranking for the target entity based on the validation performance data and/or the test performance data, the ranking comprising the evidence model and one or more additional evidence models.
17. The at least one non-transitory computer readable storage medium storing processor executable instructions of claim 16 , wherein the at least one non-transitory computer readable storage medium stores processor executable instructions that cause the at least one computer hardware processor to provide a predicted phenotypic impact of a molecular variant of interest for the target entity using a best-performing evidence model for the target entity based on the ranking in response to a query for the predicted phenotypic impact of the molecular variant of interest for the target entity.
18. The at least one non-transitory computer readable storage medium storing processor executable instructions of claim 16 , wherein the target entity comprises a functional element, molecule or molecular variant, and a phenotype of interest.
19. The at least one non-transitory computer readable storage medium storing processor executable instructions of claim 16 , wherein the at least one non-transitory computer readable storage medium stores processor executable instructions that cause the at least one computer hardware processor to determine the validation performance data by calculating a phenotype impact score for one or more molecular variants of the target entity in the production data using the evidence model and a model validation technique; and, generating the validation performance data based on the phenotype impact scores using the uniform set of performance metrics.
20. The at least one non-transitory computer readable storage medium storing processor executable instructions of claim 16 , wherein the at least one non-transitory computer readable storage medium stores processor executable instructions that cause the at least one computer hardware processor to determine the test performance data by calculating a phenotype impact score for one or more molecular variants of the target entity in the test data using the evidence model and a model validation technique; and, generating the test performance data based on the phenotype impact scores using the uniform set of performance metrics.