IP Library › Granted Patent US 12,282,527
Granted Patent B2
US 12,282,527 · App. 17/008,747 · Granted Apr 22, 2025

Determining system performance without ground truth

Inventors: Dinesh C. Verma (New Castle, NY); Seraphin Bernard Calo (Cortlandt Manor, NY)
Assignee: International Business Machines Corporation
G06F18/2185G06F18/2148G06N20/00
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,282,527
App. No.
17/008,747
Granted
Apr 22, 2025
Kind
B2
Abstract

Techniques for determining system performance without ground truth include receiving a trained model and one or more generator models, the trained model having been trained on training data. The trained model is used on testing data to produce labeled testing data, and the labeled testing data are used to train a proxy model. The one or more generator models are used to produce synthetic training data that are representative of the training data. The proxy model is used on the synthetic training data to produce predictions, and performance of the trained model is determined based on the predictions by the proxy model.

Claims (48)

1. A computer-implemented method comprising:

requesting, by an edge node of a plurality of edge nodes, both a trained model and one or more generator models from a core node coupled to the plurality of edge nodes;

receiving, by the edge node, both the trained model and the one or more generator models from the core node coupled to the plurality of edge nodes, the trained model having been trained on training data at the core node, the one or more generator models being configured to produce synthetic training data representative of the training data and having been created at the core node, wherein the edge node receives both the trained model and the one or more generator models from the core node in response to the requesting the trained model and the one or more generator models;

executing the edge node in a real-world environment to capture testing data, wherein the executing the edge node in the real-world environment to capture the testing data comprises employing components to capture the testing data under operating conditions;

inputting, by the edge node, the testing data to the trained model to produce labeled testing data, wherein the edge node comprises a proxy model distinct from the trained model and the one or more generator models, wherein the trained model, the one or more generator models, and the proxy model are machine learning models;

training, by the edge node, the proxy model with the labeled testing data, the proxy model having a machine learning architecture corresponding to the trained model; and

inputting, by the edge node, the synthetic training data to the proxy model to produce predictions related to a performance of the trained model.

2. The computer-implemented method of claim 1 , wherein the performance of the trained model is based on comparing the predictions of the proxy model to synthetic labels in the synthetic training data.

3. The computer-implemented method of claim 1 , wherein the performance of the trained model is based on using a success rate of a comparison between the predictions of the proxy model and synthetic labels in the synthetic training data as a proxy for the performance of the trained model.

4. The computer-implemented method of claim 1 , wherein the predictions of the proxy model comprise predicted labels for the synthetic training data, further comprising:

determining a success rate of the proxy model by comparing the predicted labels to synthetic labels in the synthetic training data; and

attributing the success rate of the proxy model to the trained model.

5. The computer-implemented method of claim 1 , wherein the one or more generator models are configured to produce synthetic labels for the synthetic training data representative of original labels in the training data.

6. The computer-implemented method of claim 1 , wherein the training data are remote from the edge node having the processor.

7. The computer-implemented method of claim 1 , wherein the testing data are captured at the edge node, the edge node being remote from the training data, the training data being inaccessible and not received by the processor at the edge node.

8. The computer-implemented method of claim 1 , wherein the one or more generator models comprise multiple generator models configured to produce multiple synthetic training data each being representative of the training data, further comprising:

using an intersection of the multiple synthetic training data as the synthetic training data.

9. A system comprising:

a memory having computer readable instructions; and

one or more processors of an edge node for executing the computer readable instructions, the computer readable instructions controlling the one or more processors to perform operations comprising:

requesting, by edge node of a plurality of edge nodes, both a trained model and one or more generator models from a core node coupled to the plurality of edge nodes;

receiving, by the edge node, both the trained model and the one or more generator models from the core node coupled to the plurality of edge nodes, the trained model having been trained on training data at the core node, the one or more generator models being configured to produce synthetic training data that are representative of the training data and having been created at the core node, wherein the edge node receives both the trained model and the one or more generator models from the core node in response to the requesting the trained model and the one or more generator models;

executing the edge node in a real-world environment to capture testing data, wherein the executing the edge node in the real-world environment to capture the testing data comprises employing components to capture the testing data under operating conditions;

inputting, by the edge node, testing data to the trained model to produce labeled testing data, wherein the edge node comprises a proxy model distinct from the trained model and the one or more generator models, wherein the trained model, the one or more generator models, and the proxy model are machine learning models;

training, by the edge node, the proxy model with the labeled testing data, the proxy model having a machine learning architecture corresponding to the trained model; and

inputting, by the edge node, the synthetic training data to the proxy model to produce predictions related to a performance of the trained model.

10. The system of claim 9 , wherein the performance of the trained model is based on comparing the predictions of the proxy model to synthetic labels in the synthetic training data.

11. The system of claim 9 , wherein the performance of the trained model is based on using a success rate of a comparison between the predictions of the proxy model and synthetic labels in the synthetic training data as a proxy for the performance of the trained model.

12. The system of claim 9 , wherein the predictions of the proxy model comprise predicted labels for the synthetic training data, further comprising:

determining a success rate of the proxy model by comparing the predicted labels to synthetic labels in the synthetic training data; and

attributing the success rate of the proxy model to the trained model.

13. The system of claim 9 , wherein the one or more generator models are configured to produce synthetic labels for the synthetic training data representative of original labels in the training data.

14. The system of claim 9 , wherein the training data are remote from the edge node.

15. The system of claim 9 , wherein the testing data is are captured at the edge node, the edge node being remote from the training data, the training data being inaccessible and not received by the one or more processors at the edge node.

16. The system of claim 9 , wherein the one or more generator models comprise multiple generator models configured to produce multiple synthetic training data each being representative of the training data, further comprising:

using an intersection of the multiple synthetic training data as the synthetic training data.

17. A computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor of an edge node to cause the edge node to perform operations comprising:

requesting, by the edge node, both a trained model and one or more generator models from a core node coupled to the plurality of edge nodes;

receiving, by the edge node, both the trained model and the one or more generator models from the core node coupled to the plurality of edge nodes, the trained model having been trained on training data at the core node, the one or more generator models being configured to produce synthetic training data that are representative of the training data and having been created at the core node, wherein the edge node receives both the trained model and the one or more generator models from the core node in response to the requesting the trained model and the one or more generator models;

executing the edge node in a real-world environment to capture testing data, wherein the executing the edge node in the real-world environment to capture the testing data comprises employing components to capture the testing data under operating conditions;

inputting, by the edge node, testing data to the trained model to produce labeled testing data, wherein the edge node comprises a proxy model distinct from the trained model and the one or more generator models, wherein the trained model, the one or more generator models, and the proxy model are machine learning models;

training, by the edge node, the proxy model with the labeled testing data, the proxy model having a machine learning architecture corresponding to the trained model; and

inputting, by the edge node, the synthetic training data to the proxy model to produce predictions related to a performance of the trained model.

18. The computer program product of claim 17 , wherein the performance of the trained model is based on comparing the predictions of the proxy model to synthetic labels in the synthetic training data.

19. The computer program product of claim 17 , wherein the performance of the trained model is based on using a success rate of a comparison between the predictions of the proxy model and synthetic labels in the synthetic training data as a proxy for the performance of the trained model.

20. The computer program product of claim 17 , wherein the predictions of the proxy model comprise predicted labels for the synthetic training data, further comprising:

determining a success rate of the proxy model by comparing the predicted labels to synthetic labels in the synthetic training data; and

attributing the success rate of the proxy model to the trained model.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 1, 2020
From: VERMA, DINESH C.; CALO, SERAPHIN BERNARD
To: INTERNATIONAL BUSINESS MACHINES CORPORATION
Reel/Frame 053654/0962 →
Continuity (1)
Related Publication 20220067450A1 · Mar 3, 2022
References Cited (37)
US 10289959B2 · Kakhandiki et al. · 2019 [cited by applicant]
US 10379842B2 · Malladi et al. · 2019 [cited by applicant]
US 10671938B2 · Hammond et al. · 2020 [cited by applicant]
US 10726359B1 · Drouin et al. · 2020 [cited by applicant]
US 20180330238A1 · Luciw et al. · 2018 [cited by applicant]
US 20190138908A1 · Bernat et al. · 2019 [cited by applicant]
US 20200042888A1 · Yu et al. · 2020 [cited by applicant]
US 20200081445A1 · Stetson et al. · 2020 [cited by applicant]
US 20200112609A1 · Hardman, III et al. · 2020 [cited by applicant]
US 20200151558A1 · Ren et al. · 2020 [cited by applicant]
US 20200167652A1 · Huang et al. · 2020 [cited by applicant]
US 20200219014A1 · Verma et al. · 2020 [cited by applicant]
US 20210397972A1 · Walters · 2021 [cited by examiner]
US 20220012595A1 · David · 2022 [cited by examiner]
US 20220013105A1 · Sharma · 2022 [cited by examiner]
US 20220067570A1 · Kong · 2022 [cited by examiner]
US 20230169356A1 · Banerjee · 2023 [cited by examiner]
Papernot et al., “Practical Black-Box Attacks against Machine Learning,” in Proc. 2017 ACM in Asia Conf. Computer and Comms. Security 506-19 (2017). (Year: 2017). [cited by examiner]
A. Baraldi, L. Bruzzone and P. Blonda, “Quality assessment of classification and cluster maps without ground truth knowledge,” in IEEE Transactions on Geoscience and Remote Sensing, vol. 43, No. 4, pp. 857-873, Apr. 200… [cited by applicant]
Balaji Lakshminarayanan, Yee Whye Teh (2013). Inferring ground truth from multi-annotator ordinal data: a probabilistic approach, arXiv preprint arXiv:1305.0015, 2013, 19 pages. [cited by applicant]
Bhaskaruni, D. et al., “Estimating Prediction Qualities without Ground Truth: A Revisit of the Reverse Testing Framework,” Aug. 20-24, 2018, 2018 24th Interntional Conference on Pattern Recognition (ICPR), Beijing, Chin… [cited by applicant]
Carlotto, Mark J. (2009). Effect of errors in ground truth on classification accuracy, International Journal of Remote Sensing, vol. 30, No. 18, 2009, pp. 4831-4849(19). [cited by applicant]
Dutagaci, H., Cheung, C. P., & Godil, A. (2012). Evaluation of 3D interest point detection techniques via human-generated ground truth. The Visual Computer, 28(9), 901-917. [cited by applicant]
Fedorchuk, M. et al., “Statistic Metrics for Evaluation of Binary Classifiers without Ground-Truth,” 2017 IEEE First Ukraine Conference on Electrical and Computer Engineering (UKRCON), downloaded Aug. 17, 2020, 6 pages. [cited by applicant]
Havens, K.A., et al., “Estimtaion of the Probability of Error Without Ground Truth nd Known a Priori Probabilities,” IEEE Transactions on Geoscience Electronics, Jul. 1977, vol. 15, No. 3, pp. 147-152, 6 pages. [cited by applicant]
P. Du, Z. Sun, H. Chen, J. Cho and S. Xu, “Statistical Estimation of Malware Detection Metrics in the Absence of Ground Truth,” in IEEE Transactions on Information Forensics and Security, vol. 13, No. 12, pp. 2965-2980,… [cited by applicant]
Pratzlich, T., et al., “Triple-Based Analysis of Music Alignments Without the Need of Ground-Truth Annotations,” ICASSP 2016, International Audio Laboratories Erlangen, 5 pages. [cited by applicant]
Richter S.R., Vineet V., Roth S., Koltun V. (2016) Playing for Data: Ground Truth from Computer Games. In: Leibe B., Matas J., Sebe N., Welling M. (eds) Computer Vision—ECCV 2016. ECCV 2016. Lecture Notes in Computer Sc… [cited by applicant]
Taylor, G. et al., “OVVV: Using Virtul Worlds to Design and Evaluate Surveillance Systems,” 2007 IEEE, 8 pages. [cited by applicant]
Viinikka, J., Eggeling, R., & Koivisto, M. (Mar. 2018). Intersection-validation: A method for evaluating structure learning without ground truth. In International Conference on Artificial Intelligence and Statistics (pp… [cited by applicant]
Wei Fan, Ian Davidson (2006), ReverseTesting: An Efficient Framework to Select Amongst Classifiers under Sample Selection Bias, KDD '06: Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery… [cited by applicant]
Yang, F., Du, M., & Hu, X. (2019). Evaluating explanation without ground truth in interpretable machine learning. arXiv preprint arXiv:1907.06831, 9 pages. [cited by applicant]
Deng, S. et al., Edge Intelligence: The Confluence of Edge Computing and Artificial Intelligence, Feb. 10, 2020, IEEE, 13 pages. [cited by applicant]
Wang, X. et al., “Convergence of Edge Computing and Deep Learning: A Comprehensive Survey,” Jan. 28, 2020, To Be Appeared in IEEE Communications Surveys & Tutorials, 36 pages. [cited by applicant]
Wang, X. et al., “In-Edge AI: Intelligentizing Mobile Edge Computing, Caching and Communication by Federated Learning, ” Jul. 19, 2019, IEEE Network Magazine, 10 pages. [cited by applicant]
Xu, D., et al., “Edge Intelligence: Architectures, Challenges, and Applications,” Jun. 12, 2020, IEEE, 53 pages. [cited by applicant]
Zhang, D., et al., “EdgeBatch: Towards AI-empowered Optimal Task Batching in Intelligent Edge Systems,” retrieved Aug. 3, 2020, National Science Foundation under Grant No. CNS-1845639, CNS-1831669, CBET-1637251, Army Re… [cited by applicant]