IP Library › Granted Patent US 12,198,069
Granted Patent B2
US 12,198,069 · App. 18/493,524 · Granted Jan 14, 2025

Detecting and correcting anomalies in computer-based reasoning systems

Inventor: Christopher James Hazard (Raleigh, NC)
Assignee: Howso Incorporated
G06N5/025G06F16/2465
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,198,069
App. No.
18/493,524
Granted
Jan 14, 2025
Kind
B2
Abstract

Techniques are provided herein. The techniques can include obtaining current context data and determining a contextually-determined action based on the obtained context data and a reasoning model. The reasoning model may have been determined based on one or more sets of training data. The techniques may cause performance of the contextually-determined action and, potentially, receiving an indication that performing the contextually-determined action in the current context resulted in an anomaly. The techniques include determining a portion of the reasoning model that caused the determination of the contextually-determined action based on the obtained context data and causing removal of the portion of the model that caused the determination of the contextually-determined action, to produce a corrected reasoning model. Subsequently, second context data is obtained, a second action is determined based on that data and the corrected reasoning model, and the second contextually-determined action can be performed.

Claims (38)

1. A method comprising:

determining, using one or more computing devices, a contextually-determined action for a control system based on obtained context data and a reasoning model, wherein the reasoning model was determined based on one or more sets of training data, wherein the one or more sets of training data include multiple context data and action data pairings, where the control system is a self-driving vehicle, and wherein determining the contextually-determined action for the control system comprises determining, using a premetric, closest context data in the one or more sets of training data that is closest to a current context based on the premetric and determining an action paired with the closest context data as the contextually-determined action for the control system, wherein the premetric is a Minkowski distance measure of order less than one;

determining, using the one or more computing devices, whether performance of the contextually-determined action would result in an indication of an anomaly for the control system;

updating, using the one or more computing devices, a portion of the reasoning model associated with the closest context data in the one or more sets of training data that is closest to the current context based on the premetric in order to produce a corrected reasoning model, wherein updating the portion of the reasoning model comprises changing the action paired with the closest context data;

obtaining, using the one or more computing devices, subsequent contextual data for a second context for the control system;

determining, using the one or more computing devices, a second contextually-determined action for the control system based on the obtained subsequent contextual data and the corrected reasoning model; and

causing performance, using the one or more computing devices, of the second contextually-determined action for the control system, wherein the second contextually-determined action is a contextually-determined vehicle actions to be performed by the self-driving vehicle.

2. The method of claim 1 , wherein the reasoning model is a case-based reasoning model.

3. The method of claim 1 , further comprising determining the portion of the reasoning model that caused the determining of the contextually-determined action that resulted in the indication of the anomaly by determining the previously-identified closest context data in the one or more sets of training data.

4. The method of claim 1 , wherein updating the portion of the reasoning model that caused the determining of the contextually-determined action that resulted in the indication of the anomaly to produce the corrected reasoning model comprises removing an association between the previously-identified closest context data and the action paired with the closest context data.

5. The method of claim 1 , wherein updating the portion of the reasoning model that caused the determining of the contextually-determined action that resulted in the indication of the anomaly to produce the corrected reasoning model comprises removing the closest context data and the paired action.

6. The method of claim 1 , wherein the method additionally comprises determining additional portions of the reasoning model that would cause the performance of the contextually-determined action that resulted in the indication of the anomaly in the current context and updating the portion of the reasoning model that caused the determining of the contextually-determined action that resulted in the indication of the anomaly to produce the corrected reasoning model comprises removing the additional portions of the reasoning model.

7. The method of claim 1 , further comprising determining whether performance of the contextually-determined action would result in the indication of the anomaly for the control system based at least in part on a receipt of an anomaly indication from a system separate from the control system.

8. A system for performing a machine-executed operation involving instructions, wherein said instructions are instructions which, when executed by one or more computing devices, cause performance of certain steps including:

determining, using one or more computing devices, a contextually-determined action for a control system based on obtained context data and a reasoning model, wherein the reasoning model was determined based on one or more sets of training data, wherein the one or more sets of training data include multiple context data and action data pairings, where the control system is a self-driving vehicle, and wherein determining the contextually-determined action for the control system comprises determining, using a premetric, closest context data in the one or more sets of training data that is closest to a current context based on the premetric and determining an action paired with the closest context data as the contextually-determined action for the control system, wherein the premetric is a Minkowski distance measure of order less than one;

determining whether performance of the contextually-determined action would result in an indication of an anomaly for the control system;

updating a portion of the reasoning model associated with the closest context data in the one or more sets of training data that is closest to the current context based on the premetric in order to produce a corrected reasoning model, wherein updating the portion of the reasoning model comprises changing the action paired with the closest context data;

obtaining subsequent contextual data for a second context for the control system;

determining a second contextually-determined action for the control system based on the obtained subsequent contextual data and the corrected reasoning model; and

causing performance of the second contextually-determined action for the control system, wherein the second contextually-determined action is a contextually-determined vehicle actions to be performed by the self-driving vehicle.

9. The system of claim 8 , wherein the performed steps additionally comprise determining the portion of the reasoning model that caused the determining of the contextually-determined action that resulted in the indication of the anomaly by determining the previously-identified closest context data in the one or more sets of training data.

10. The system of claim 8 , wherein updating the portion of the reasoning model that cause the determining of the contextually-determined action that resulted in the indication of the anomaly to produce the corrected reasoning model comprises removing the portion of the reasoning model associated the closest context data, and the action paired with the closest context data.

11. The system of claim 8 , wherein the performed steps additionally comprise determining additional portions of the reasoning model that would cause the performance of the contextually-determined action that resulted in the indication of the anomaly and updating the portion of the reasoning model that caused the determining of the contextually-determined action that resulted in the indication of the anomaly to produce the corrected reasoning model comprises removing the additional portions of the reasoning model.

12. The system of claim 8 , wherein the reasoning model is a case-based reasoning model.

13. The system of claim 8 , wherein the performed steps additionally comprise determining whether performance of the contextually-determined action results in the indication of the anomaly for the control system based at least in part on a receipt of an anomaly indication from a system separate from the control system.

14. One or more non-transitory computer readable media storing instructions which, when executed by one or more computing devices, cause performance of certain steps including:

determining, using the one or more computing devices, a contextually-determined action for a control system based on obtained context data and a reasoning model, wherein the reasoning model was determined based on one or more sets of training data, wherein the one or more sets of training data include multiple context data and action data pairings, where the control system is a self-driving vehicle, and wherein determining the contextually-determined action for the control system comprises determining, using a premetric, closest context data in the one or more sets of training data that is closest to a current context based on the premetric and determining an action paired with the closest context data as the contextually-determined action for the control system, wherein the premetric is a Minkowski distance measure of order less than one;

determining whether performance of the contextually-determined action would result in an indication of an anomaly for the control system;

updating a portion of the reasoning model associated with the closest context data in the one or more sets of training data that is closest to the current context based on the premetric in order to produce a corrected reasoning model, wherein updating the portion of the reasoning model comprises changing the action paired with the closest context data;

obtaining subsequent contextual data for a second context for the control system;

determining a second contextually-determined action for the control system based on the obtained subsequent contextual data and the corrected reasoning model; and

causing performance of the second contextually-determined action for the control system, wherein the second contextually-determined action is a contextually-determined vehicle actions to be performed by the self-driving vehicle.

15. The one or more non-transitory computer readable media of claim 14 , wherein the performed steps additionally comprise determining the portion of the reasoning model that caused the determining of the contextually-determined action that resulted in the indication of the anomaly by determining the previously-identified closest context data in the one or more sets of training data.

16. The one or more non-transitory computer readable media of claim 14 , wherein updating the portion of the reasoning model that cause the determining of the contextually-determined action that resulted in the indication of the anomaly to produce the corrected reasoning model comprises removing the portion of the reasoning model associated the closest context data, and the action paired with the closest context data.

17. The one or more non-transitory computer readable media of claim 14 , wherein the performed steps additionally comprise determining additional portions of the reasoning model that would cause the performance of the contextually-determined action that resulted in the indication of the anomaly and updating the portion of the reasoning model that caused the determining of the contextually-determined action that resulted in the indication of the anomaly to produce the corrected reasoning model comprises removing the additional portions of the reasoning model.

18. The one or more non-transitory computer readable media of claim 14 , wherein the reasoning model is a case-based reasoning model.

19. The one or more non-transitory computer readable media of claim 14 , wherein the performed steps additionally comprise determining whether performance of the contextually-determined action would result in the indication of the anomaly for the control system based at least in part on a receipt of an anomaly indication from a system separate from the control system.

20. The one or more non-transitory computer readable media of claim 14 , wherein the performed steps additionally comprise determining whether performance of the contextually-determined action results in the indication of the anomaly for the control system based at least in part on a receipt of an anomaly indication from a system separate from the control system.

Assignments (3)
CHANGE OF NAME Recorded Dec 29, 2023
From: DIVEPLANE CORPORATION
To: HOWSO INCORPORATED
Reel/Frame 066143/0798 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 30, 2023
From: HAZARDOUS SOFTWARE INC.
To: DIVEPLANE CORPORATION
Reel/Frame 065388/0591 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 25, 2023
From: HAZARD, CHRISTOPHER JAMES
To: HAZARDOUS SOFTWARE INC.
Reel/Frame 065343/0715 →
Continuity (6)
Continuation 18181529 · Mar 9, 2023
Continuation 17346583 · Jun 14, 2021
Continuation 15900398 · Feb 20, 2018
Continuation 15681219 · Aug 18, 2017
Related Publication 20240135206A1 · Apr 25, 2024
Related Publication 20240232660A9 · Jul 11, 2024
References Cited (159)
US 4935877A · Koza · 1990 [cited by applicant]
US 5581664A · Allen · 1996 [cited by examiner]
US 6282527B1 · Gounares et al. · 2001 [cited by applicant]
US 6741972B1 · Girardi et al. · 2004 [cited by applicant]
US 7873587B2 · Baum · 2011 [cited by applicant]
US 9489635B1 · Zhu · 2016 [cited by examiner]
US 9922286B1 · Hazard · 2018 [cited by applicant]
US 10459444B1 · Kentley-Klay · 2019 [cited by applicant]
US 20010049595A1 · Plumer et al. · 2001 [cited by applicant]
US 20040019851A1 · Purvis et al. · 2004 [cited by applicant]
US 20050137992A1 · Polak · 2005 [cited by applicant]
US 20060195204A1 · Bonabeau et al. · 2006 [cited by applicant]
US 20080153098A1 · Rimm et al. · 2008 [cited by applicant]
US 20080307399A1 · Zhou et al. · 2008 [cited by applicant]
US 20090006299A1 · Baum · 2009 [cited by applicant]
US 20090144704A1 · Niggemann et al. · 2009 [cited by applicant]
US 20100106603A1 · Dey · 2010 [cited by examiner]
US 20100287507A1 · Paquette et al. · 2010 [cited by applicant]
US 20110060895A1 · Solomon · 2011 [cited by applicant]
US 20110161264A1 · Cantin · 2011 [cited by applicant]
US 20110225564A1 · Biswas et al. · 2011 [cited by applicant]
US 20130006901A1 · Cantin · 2013 [cited by applicant]
US 20130339365A1 · Balasubramanian et al. · 2013 [cited by applicant]
US 20160055427A1 · Adjaoute · 2016 [cited by examiner]
US 20170010106A1 · Shashua et al. · 2017 [cited by applicant]
US 20170012772A1 · Mueller · 2017 [cited by examiner]
US 20170053211A1 · Heo · 2017 [cited by examiner]
US 20170091645A1 · Matus · 2017 [cited by applicant]
US 20170161640A1 · Shamir · 2017 [cited by applicant]
US 20170236060A1 · Ignatyev · 2017 [cited by examiner]
US 20180072323A1 · Gordon · 2018 [cited by examiner]
US 20180089563A1 · Redding et al. · 2018 [cited by applicant]
US 20180235649A1 · Elkadi · 2018 [cited by applicant]
US 20180336018A1 · Lu et al. · 2018 [cited by applicant]
US 20190147331A1 · Arditi · 2019 [cited by applicant]
WO WO2017057528 · 2017 [cited by applicant]
WO WO2017189859 · 2017 [cited by applicant]
Shinozaki Masahisa et al., “Detector, correction system, detection method and program”, JP 2015-232847 A, Published on Dec. 24, 2015. (Previously supplied). (Year: 2015). [cited by examiner]
Smyth B., “Case-Base Maintenance”, 1998. (Previously supplied). (Year: 1998). [cited by examiner]
Salamo et al., “Adaptive case-based reasoning using retention and forgetting strategies”, Knowledge-Based Systems 24 (2011) 230-247. (Previously supplied). (Year: 2011). [cited by examiner]
Abdi, “Cardinality Optimization Problems”, The University of Birmingham, PhD dissertation, May 2013, 197 pages. [cited by applicant]
Aboulnaga, “Generating Synthetic Complex-structured XML Data”, Proceedings of the Fourth International Workshop on the Web and Databases, WebDB 2001, Santa Barbara, California, USA, May 24-25, 2001, 6 pages. [cited by applicant]
Abramson, “The Expected-Outcome Model of Two-Player Games”, PhD Thesis, Columbia University, New York, New York, 1987, 125 pages. [cited by applicant]
Abuelaish et al., “Analysis and Modelling of Groundwater Salinity Dynamics in the Gaza Strip”, Cuadernos Geograficos, vol. 57, Issue 2, pp. 72-91. [cited by applicant]
Agarwal et al., “Nearest-Neighbor Searching Under Uncertainty II”, ACM Transactions on Algorithms, vol. 13, Issue 1, Article 3, 2016, 25 pages. [cited by applicant]
Aggarwal et al., “On the Surprising Behavior of Distance Metrics in High Dimensional Space”, International Conference on Database Theory, London, United Kingdom, Jan. 4-6, 2001, pp. 420-434. [cited by applicant]
Akaike, “Information Theory and an Extension of the Maximum Likelihood Principle”, Proceedings of the 2nd International Symposium on Information Theory, Sep. 2-8, 1971, Tsahkadsor, Armenia, pp. 267-281. [cited by applicant]
Alhaija, “Augmented Reality Meets Computer Vision: Efficient Data Generation for Urban Driving Scenes”, arXiv:1708.01566vl, Aug. 4, 2017, 12 pages. [cited by applicant]
Alpaydin, “Machine Learning: The New A1”, MIT Press, Cambridge, Massachusetts, 2016, 225 pages. [cited by applicant]
Alpaydin, “Voting Over Multiple Condensed Nearest Neighbor”, Artificial Intelligence Review, vol. 11, 1997, pp. 115-132. [cited by applicant]
Altman, “An Introduction to Kernel and Nearest-Neighbor Nonparametric Regression”, The American Statistician, vol. 46, Issue 3, 1992, pp. 175-185. [cited by applicant]
Anderson, “Synthetic data generation for the internet of things,” 2014 IEEE International Conference on Big Data (Big Data), Washington, DC, 2014, pp. 171-176. [cited by applicant]
Archer et al., “Empirical Characterization of Random Forest Variable Importance Measures”, Computational Statistics & Data Analysis, vol. 52, 2008, pp. 2249-2260. [cited by applicant]
Beyer et al., “When is ‘Nearest Neighbor’ Meaningful?” International Conference on Database Theory, Springer, Jan. 10-12, 1999, Jerusalem, Israel, pp. 217-235. [cited by applicant]
Bull, Haploid-Diploid Evolutionary Algorithms: The Baldwin Effect and Recombination Nature's Way, Artificial Intelligence and Simulation of Behaviour Convention, Apr. 19-21, 2017, Bath, United Kingdom, pp. 91-94. [cited by applicant]
Cano et al., “Evolutionary Stratified Training Set Selection for Extracting Classification Rules with Tradeoff Precision-Interpretability” Data and Knowledge Engineering, vol. 60, 2007, pp. 90-108. [cited by applicant]
Chawla, “SMOTE: Synthetic Minority Over-sampling Technique”, arXiv:1106.1813v1, Jun. 9, 2011, 37 pages. [cited by applicant]
Chen, “DropoutSeer: Visualizing Learning Patterns in Massive Open Online Courses for Dropout Reasoning and Prediction”, 2016 IEEE Conference on Visual Analytics Science and Technology (VAST), Oct. 23-28, 2016, Baltimore… [cited by applicant]
Chomboon et al., An Empirical Study of Distance Metrics for k-Nearest Neighbor Algorithm, 3rd International Conference on Industrial Application Engineering, Kitakyushu, Japan, Mar. 28-31, 2015, pp. 280-285. [cited by applicant]
Colakoglu, “A Generalization of the Minkowski Distance and a New Definition of the Ellipse”, arXiv:1903.09657v1, Mar. 2, 2019, 18 pages. [cited by applicant]
International Search Report and Written Opinion for PCT/US2018/047118 mailed on Dec. 3, 2018, 9 pages. [cited by applicant]
Dernoncourt, “MoocViz: A Large Scale, Open Access, Collaborative, Data Analytics Platform for MOOCs”, NIPS 2013 Education Workshop, Nov. 1, 2013, Lake Tahoe, Utah, USA, 8 pages. [cited by applicant]
Ding, “Generating Synthetic Data for Neural Keyword-to-Question Models”, arXiv:1807.05324vl, Jul. 18, 2018, 12 pages. [cited by applicant]
Dwork et al., “The Algorithmic Foundations of Differential Privacy”, Foundations and Trends in Theoretical Computer Science, vol. 9, Nos. 3-4, 2014, pp. 211-407. [cited by applicant]
Extended European Search Report for Application 22163651, dated Oct. 6, 2022, 11 pages. [cited by applicant]
Efros et al., “Texture Synthesis by Non-Parametric Sampling”, International Conference on Computer Vision, Sep. 20-25, 1999, Corfu, Greece, 6 pages. [cited by applicant]
Fathony, “Discrete Wasserstein Generative Adversarial Networks (DWGAN)”, OpenReview.net, Feb. 18, 2018, 20 pages. [cited by applicant]
Ganegedara et al., “Self Organizing Map Based Region of Interest Labelling for Automated Defect Identification in Large Sewer Pipe Image Collections”, IEEE World Congress on Computational Intelligence, Jun. 10-15, 2012,… [cited by applicant]
Gao et al., “Efficient Estimation of Mutual Information for Strongly Dependent Variables”, 18th International Conference on Artificial Intelligence and Statistics, San Diego, California, May 9-12, 2015, pp. 277-286. [cited by applicant]
Gehr et al., “A12: Safety and Robustness Certification of Neural Networks with Abstract Interpretation”, 39th IEEE Symposium on Security and Privacy, San Francisco, California, May 21-23, 2018, 16 pages. [cited by applicant]
Gemmeke et al., “Using Sparse Representations for Missing Data Imputation in Noise Robust Speech Recognition”, European Signal Processing Conference, Lausanne, Switzerland, Aug. 25-29, 2008, 5 pages. [cited by applicant]
Ghosh, “Inferential Privacy Guarantees for Differentially Private Mechanisms”, arXiv: 1603:01508vl, Mar. 4, 2016, 31 pages. [cited by applicant]
Goodfellow et al., “Deep Learning”, MIT Press, 2016, 800 pages. [cited by applicant]
Google AI Blog, “The What-If Tool: Code-Free Probing of Machine Learning Models”, Sep. 11, 2018, https://pair-code,github.io/what-if-tool, retrieved on Mar. 14, 2019, 5 pages. [cited by applicant]
Gottlieb et al., “Near-Optimal Sample Compression for Nearest Neighbors”, Advances in Neural Information Processing Systems, Montreal, Canada, Dec. 8-13, 2014, 9 pages. [cited by applicant]
Gray, “Quickly Generating Billion-Record Synthetic Databases”, SIGMOD '94: Proceedings of the 1994 ACM SIGMOD international conference on Management of data, May 1994, Minneapolis, Minnesota, USA, 29 pages. [cited by applicant]
Hastie et al., “The Elements of Statistical Learning”, Springer, 2001, 764 pages. [cited by applicant]
Hazard et al., “Natively Interpretable Machine Learning and Artificial Intelligence: Preliminary Results and Future Directions”, arXiv:1901vl, Jan. 2, 2019, 15 pages. [cited by applicant]
Hinneburg et al., “What is the Nearest Neighbor in High Dimensional Spaces?”, 26th International Conference on Very Large Databases, Cairo, Egypt, Sep. 10-14, 2000, pp. 506-515. [cited by applicant]
Hmeidi et al., “Performance of KNN and SVM Classifiers on Full Word Arabic Articles”, Advanced Engineering Informatics, vol. 22, Issue 1, 2008, pp. 106-111. [cited by applicant]
Hoag “A Parallel General-Purpose Synthetic Data Generator”, ACM SIGMOID Record, vol. 6, Issue 1, Mar. 2007, 6 pages. [cited by applicant]
Houle et al., “Can Shared-Neighbor Distances Defeat the Curse of Dimensionality?”, International Conference on Scientific and Statistical Database Management, Heidelberg, Germany, Jun. 31-Jul. 2, 2010, 18 pages. [cited by applicant]
Imbalanced-learn, “SMOTE”, 2016-2017, https://imbalanced-learn.readthedocs.jo/en/stable/generated/imbleam.over_sampling.SMOTE.html, retrieved on Aug. 11, 2020, 6 pages. [cited by applicant]
Indyk et al., “Approximate Nearest Neighbors: Towards Removing the Curse of Dimensionality”, Procedures of the 30th ACM Symposium on Theory of Computing, Dallas, Texas, May 23-26, 1998, pp. 604-613. [cited by applicant]
International Search Report and Written Opinion for PCT/US2019/026502, mailed on Jul. 24, 2019, 16 pages. [cited by applicant]
International Search Report and Written Opinion for PCT/US2019/066321, mailed on Mar. 19, 2020, 17 pages. [cited by applicant]
Internet Archive, “SystemVerilog distribution Constraint—Verification Guide”, Aug. 6, 2018, http://web.archive.org/web/20180806225430/https://www.verificationguide.com/p/systemverilog-distribution-constraint.html, retri… [cited by applicant]
Internet Archive, “SystemVerilog Testbench Automation Tutorial”, Nov. 17, 2016, https://web.archive.org/web/20161117153225/http://www.doulos.com/knowhow/sysverilog/tutorial/constraints/, retrieved on Mar. 3, 2020, 6 pag… [cited by applicant]
Kittler, “Feature Selection and Extraction”, Handbook of Pattern Recognition and Image Processing, Jan. 1986, Chapter 3, pp. 115-132. [cited by applicant]
Kohavi et al., “Wrappers for Feature Subset Selection”, Artificial Intelligence, vol. 97, Issues 1-2, Dec. 1997, pp. 273-323. [cited by applicant]
Kontorovich et al., “Nearest-Neighbor Sample Compression: Efficiency, Consistency, Infinite Dimensions”, Advances in Neural Information Processing Systems, 2017, pp. 1573-1583. [cited by applicant]
Kulkarni et al., “Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation”, arXiv:1604.06057v2, May 31, 2016, 14 pages. [cited by applicant]
Kuramochi et al., “Gene Classification using Expression Profiles: A Feasibility Study”, Technical Report TR 01-029, Department of Computer Science and Engineering, University of Minnesota, Jul. 23, 2001, 18 pages. [cited by applicant]
Kushilevitz et al., “Efficient Search for Approximate Nearest Neighbor in High Dimensional Spaces”; Society for Industrial and Applied Mathematics Journal Computing, vol. 30, No. 2, pp. 457-474. [cited by applicant]
Leinster et al., “Maximizing Diversity in Biology and Beyond”, Entropy, vol. 18, Issue 3, 2016, 23 pages. [cited by applicant]
Liao et al., “Similarity Measures for Retrieval in Case-Based Reasoning Systems”, Applied Artificial Intelligence, vol. 12, Issue 4, 1 998, pp. 267-288. [cited by applicant]
Lin et al., “Why Does Deep and Cheap Learning Work So Well?” Journal of Statistical Physics, vol. 168, 2017, pp. 1223-1247. [cited by applicant]
Lin, “Development of a Synthetic Data Set Generator for Building and Testing Information Discovery Systems”, Proceedings of the Third International Conference on Information Technology: New Generations, Nevada, USA, Apr… [cited by applicant]
Lukaszyk, “A New Concept of Probability Metric and its Applications in Approximation of Scattered Data Sets”, Computational Mechanics, vol. 33, 2004, pp. 299-304. [cited by applicant]
Lukaszyk, “Probability Metric, Examples of Approximation Applications in Experimental Mechanics”, PhD Thesis, Cracow University of Technology, 2003, 149 pages. [cited by applicant]
Mann et al., “On a Test of Whether One or Two Random Variables is Stochastically Larger than the Other”, The Annals of Mathematical Statistics, 1947, pp. 50-60. [cited by applicant]
Martino et al., “A Fast Universal Self-Tuned Sampler within Gibbs Sampling”, Digital Signal Processing, vol. 47, 2015, pp. 68-83. [cited by applicant]
Masahisa et al., “Detector, correction system, detection method and program”, JP 2015-232847 A, published on Dec. 24, 2015. [cited by applicant]
Mohri et al., Foundations of Machine Leaming, 2012, 427 pages—uploaded as Part 1 and Part 2. [cited by applicant]
Montanez, “SDV: An Open Source Library for Synthetic Data Generation”, Massachusetts Institute of Technology, Master's Thesis, Sep. 2018, 105 pages. [cited by applicant]
Negra, “Model of a Synthetic Wind Speed Time Series Generator”, Wind Energy, Wiley Interscience, Sep. 6, 2007, 17 pages. [cited by applicant]
Nguyen et al., “NP-Hardness of 0 Minimization Problems: Revision and Extension to the Non-Negative Setting”, 13th International Conference on Sampling Theory and Applications, Jul. 8-12, 2019, Bordeaux, France, 4 pages. [cited by applicant]
Olson et al., “PMLB: A Large Benchmark Suite for Machine Learning Evaluation and Comparison”, arXiv:1703.00512v1, Mar. 1, 2017, 14 pages. [cited by applicant]
Patki, “The Synthetic Data Vault: Generative Modeling for Relational Databases”, Massachusetts Institute of Technology, Master's Thesis, Jun. 2016, 80 pages. [cited by applicant]
Patki, “The Synthetic Data Vault”, 2016 IEEE International Conference on Data Science and Advanced Analytics (DSAA), Oct. 17-19, 2016, Montreal, QC, pp. 399-410. [cited by applicant]
Pedregosa et al., “Machine Learning in Python”, Journal of Machine Learning Research, vol. 12, 2011, pp. 2825-2830. [cited by applicant]
Pei, “A Synthetic Data Generator for Clustering and Outlier Analysis”, The University of Alberta, 2006, 33 pages. [cited by applicant]
Phan et al., “Adaptive Laplace Mechanism: Differential Privacy Preservation in Deep Learning” 2017 IEEE International Conference on Data Mining, New Orleans, Louisiana, Nov. 18-21, 2017, 10 pages. [cited by applicant]
Poerner et al., “Evaluating Neural Network Explanation Methods Using Hybrid Documents and Morphosyntactic Agreement”, Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Long Papers)… [cited by applicant]
Prakosa “Generation of Synthetic but Visually Realistic Time Series of Cardiac Images Combining a Biophysical Model and Clinical Images” IEEE Transactions on Medical Imaging, vol. 32, No. 1, Jan. 2013, pp. 99-109. [cited by applicant]
Priyardarshini, “WEDAGEN: A synthetic web database generator”, International Workshop of Internet Data Management (IDM'99), Sep. 2, 1999, Florence, IT, 24 pages. [cited by applicant]
Pudjijono, “Accurate Synthetic Generation of Realistic Personal Information” Advances in Knowledge Discovery and Data Mining, Pacific-Asia Conference on Knowledge Discovery and Data Mining, Apr. 27-30, 2009, Bangkok, Th… [cited by applicant]
Raikwal et al., “Performance Evaluation of SVM and K-Nearest Neighbor Algorithm Over Medical Data Set”, International Journal of Computer Applications, vol. 50, No. 14, Jul. 2012, pp. 975-985. [cited by applicant]
Rao et al., “Cumulative Residual Entropy: A New Measure of Information”, IEEE Transactions on Information Theory, vol. 50, Issue 6, 2004, pp. 1220-1228. [cited by applicant]
Reiter, “Using CART to Generate Partially Synthetic Public use Microdata” Journal of Official Statistics, vol. 21, No. 3, 2005, pp. 441-462. [cited by applicant]
Ribeiro et al., “'Why Should I Trust You': Explaining the Predictions of any Classifier”, arXiv:1602.04938v3, Aug. 9, 2016, 10 pages. [cited by applicant]
Rosenberg et al., “Semi-Supervised Self-Training of Object Detection Models”, IEEE Workshop on Applications of Computer Vision, 2005, 9 pages. [cited by applicant]
Salamo et al., “Adaptive case-based reasoning using retention and forgetting strategies”, Knowledge-Based Systems 24 (2011) 230-247. (Year: 2011). [cited by applicant]
Schaul et al., “Universal Value Function Approximators”, International Conference on Machine Learning, Lille, France, Jul. 6-11, 2015, 9 pages. [cited by applicant]
Schlabach et al., “FOX-GA: A Genetic Algorithm for Generating and Analyzing Battlefield Courses of Action”, 1999 MIT. [cited by applicant]
Schreck, “Towards an Automatic Predictive Question Formulation”, Massachusetts Institute of Technology, Master's Thesis, Jun. 2016, 121 pages. [cited by applicant]
Schreck, “What would a data scientist ask? Automatically formulating and solving prediction problems”, 2016 IEEE International Conference on Data Science and Advanced Analytics (DSAA), Oct. 17-19, 2016, Montreal, QC, pp… [cited by applicant]
Schuh et al., “Improving the Performance of High-Dimensional KNN Retrieval Through Localized Dataspace Segmentation and Hybrid Indexing”, Proceedings of the 17th East European Conference, Advances in Databases and Infor… [cited by applicant]
Schuh et al., “Mitigating the Curse of Dimensionality for Exact KNN Retrieval”, Proceedings of the 26th International Florida Artificial Intelligence Research Society Conference, St. Pete Beach, Florida, May 22-24, 2014… [cited by applicant]
Schwarz et al., “Estimating the Dimension of a Model”, The Annals of Statistics, vol. 6, Issue 2, Mar. 1978, pp. 461-464. [cited by applicant]
Silver et al., “Mastering the Game of Go Without Human Knowledge”, Nature, vol. 550, Oct. 19, 2017, pp. 354-359. [cited by applicant]
Skapura, “Building Neural Networks”, 1996, p. 63. [cited by applicant]
Smith, “FeatureHub: Towards collaborative data science”, 2017 IEEE International Conference on Data Science and Advanced Analytics (DSAA), Oct. 19-21, 2017, Tokyo, pp. 590-600. [cited by applicant]
Smyth B., “Case-Base Maintenance”, 1998. (Year: 1998). [cited by applicant]
Stephenson et al., “A Continuous Information Gain Measure to Find the Most Discriminatory Problems for AI Benchmarking”, arxiv.org, arxiv.org/abs/1809.02904v2, retrieved on Aug. 21, 2019, 8 pages. [cited by applicant]
Stoppiglia et al., “Ranking a Random Feature for Variable and Feature Selection” Journal of Machine Learning Research, vol. 3, 2003, pp. 1399-1414. [cited by applicant]
Sun et al., “Fuzzy Modeling Employing Fuzzy Polyploidy Genetic Algorithms”, Journal of Information Science and Engineering, Mar. 2002, vol. 18, No. 2, pp. 163-186. [cited by applicant]
Sun, “Learning Vine Copula Models For Synthetic Data Generation”, arXiv:1812.01226v1, Dec. 4, 2018. 9 pages. [cited by applicant]
Surya et al., “Distance and Similarity Measures Effect on the Performance of K-Nearest Neighbor Classifier”, arXiv:1708.04321vl, Aug. 14, 2017, 50 pages. [cited by applicant]
Tan et al., “Incomplete Multi-View Weak-Label Leaming”, 27th International Joint Conference on Artificial Intelligence, 2018, pp. 2703-2709. [cited by applicant]
Tao et al., “Quality and Efficiency in High Dimensional Nearest Neighbor Search”, Proceedings of the 2009 Acm Sigmod International Conference on Management of Data, Providence, Rhode Island, Jun. 29-Jul. 2, 2009, pp. 56… [cited by applicant]
Tishby et al., “Deep Learning and the Information Bottleneck Principle”, arXiv:1503.02406v1, Mar. 9, 2015, 5 pages. [cited by applicant]
Tockar, “Differential Privacy: The Basics”, Sep. 8, 2014, https://research.neustar.biz/2014/09/08/differential-privacy-the-basics/ retrieved on Apr. 1, 2019, 3 pages. [cited by applicant]
Tomasev et al., “Hubness-aware Shared Neighbor Distances for High-Dimensional k-Nearest Neighbor Classification”, 7th international Conference on Hybrid Artificial Intelligent Systems, Salamanca, Spain, Mar. 28-30, 2012… [cited by applicant]
Tran, “Dist-GAN: An Improved GAN Using Distance Constraints”, arXiv:1803.08887V3, Dec. 15, 2018, 20 pages. [cited by applicant]
Trautmann et al., “On the Distribution of the Desirability Index using Harrington's Desirability Function”, Metrika, vol. 63, Issue 2, Apr. 2006, pp. 207-213. [cited by applicant]
Triguero et al., “Self-Labeled Techniques for Semi-Supervised Learning: Taxonomy, Software and Empirical Study”, Knowledge and Information Systems, vol. 42, Issue 2, 2015, pp. 245-284. [cited by applicant]
Tuomisto, “A Consistent Terminology for Quantifying Species Diversity? Yes, It Does Exist” Oecologia, vol. 164, 2010, pp. 853-860. [cited by applicant]
Vacek et al., “Using Case-Based Reasoning for Autonomous Vehicle Guidance”, International Conference on Intelligent Robots and Systems, San Diego, California, October 29-Nov. 2, 2007, pp. 4271-4276. [cited by applicant]
Verleysen et al., “The Curse of Dimensionality in Data Mining and Time Series Prediction” International Work-Conference on Artificial Neural Networks, Barcelona, Spain, Jun. 8-10, 2005, pp. 758-770. [cited by applicant]
Viera, “Generating Synthetic Sequential Data using GANs”, Medium: Toward AI, Jun. 29, 2020, 31 pages. [cited by applicant]
Wachter et al., “Counterfactual Explanations Without Opening the Black Box: Automated Decisions and the GDPR”, Harvard Journal of Law and Technology, vol. 31, No. 2, Spring 2018, 47 pages. [cited by applicant]
Wang et al., “Falling Rule Lists” 18th International Conference on Artificial Intelligence and Statistics, San Diego, California, May 9-12, 2015, 10 pages. [cited by applicant]
Wei et al., “An Operation-Time Simulation Framework for UAV Swarm Configuration and Mission Planning”, 2013. [cited by applicant]
Weselkowski et al., “TraDE: Training Device Selection via Multi-Objective Optimization”, IEEE 2014. [cited by applicant]
Wu, “The Synthetic Student: A Machine Learning Model to Simulate MOOC Data”, Massachusetts Institute of Technology, Master's Thesis, May 2015, 103 pages. [cited by applicant]
Xiao, “Towards Automatically Linking Data Elements”, Massachusetts Institute of Technology, Master's Thesis, Jan. 2017, 92 pages. [cited by applicant]
Xu et al., “An Algorithm for Remote Sensing Image Classification Based on Artificial Immune B-Cell Network”, The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, vol. XXXVII… [cited by applicant]
Zhao et al., “Semi-Supervised Multi-Label Learning with Incomplete Labels”, 24th International Joint Conference on Artificial Intelligence, Buenos Aires, Argentina, Jul. 25-31, 2015, pp. 4062-4068. [cited by applicant]