Dataset quality for synthetic data generation in computer-based reasoning systems
Techniques for synthetic data generation in computer-based reasoning systems are discussed and include receiving a request for generation of synthetic data based on a set of training data cases. One or more focal training data cases are determined. For undetermined features (either all of them or those that are not subject to conditions), a value for the feature is determined based on the focal cases. In some embodiments, the generated synthetic data may be checked for similarity against the training data, and if similarity conditions are met, it may be modified (e.g., resampled), removed, and/or replaced.
1. A non-transitory computer readable medium storing instructions which, when executed by one or more computing devices, cause the one or more computing devices to perform a process of:
receiving a request for generation of synthetic data based on a set of training data cases that meets a dataset quality threshold;
repeatedly determining new a synthetic data case based on a set of one or more focal training data cases to generate a set of two or more synthetic data cases, wherein each set of one or more focal cases are determined from the set of training data cases;
determining a dataset quality metric for the set of two or more synthetic data cases based on the set of training data cases and the set of two or more synthetic data cases, wherein the dataset quality metric is determined based at least in part on at least one dataset privacy metric, which quantifies the likelihood of identification of private data in the set of training data cases from the set of two or more synthetic data cases;
when the dataset quality metric for particular synthetic data cases in the set of two or more synthetic data cases does not meet a dataset quality threshold, taking corrective action for the particular synthetic data cases in the set of two or more synthetic data cases to produce a new set of two or more synthetic data cases to use as the set of two or more synthetic data cases;
when the dataset quality metric for the set of two or more synthetic data cases meets the dataset quality threshold, causing control of a controllable system using the set of two or more synthetic data cases.
2. The non-transitory computer readable medium of claim 1 , wherein taking corrective action comprises one or more of:
modifying one or more of the particular synthetic data cases to produce a new set of two or more synthetic data cases to use as the set of two or more synthetic data cases;
deleting one or more of the particular synthetic data cases to produce a new set of two or more synthetic data cases to use as the set of two or more synthetic data cases; and
replacing one or more of the particular synthetic data cases to produce a new set of two or more synthetic data cases to use as the set of two or more synthetic data cases.
3. The non-transitory computer readable medium of claim 1 , wherein determining at least one dataset quality metric comprises determining a dataset privacy metric based at least in part on a data element distance comparison metric, and when the dataset privacy metric for particular synthetic data cases in the set of two or more synthetic data cases does not meet the dataset quality threshold, taking corrective action for the particular synthetic data cases in set of two or more synthetic data cases to produce a new set of two or more synthetic data cases to use as the set of two or more synthetic data cases.
4. The non-transitory computer readable medium of claim 1 , wherein determining at least one dataset quality metric comprises determining a dataset privacy metric based at least in part on a minimum distance ratio metric, and when the dataset privacy metric for particular synthetic data cases in the set of two or more synthetic data cases does not meet the dataset quality threshold, taking corrective action for the particular synthetic data cases in set of two or more synthetic data cases to produce a new set of two or more synthetic data cases to use as the set of two or more synthetic data cases.
5. The non-transitory computer readable medium of claim 1 , wherein determining at least one dataset quality metric comprises determining a dataset privacy metric based at least in part on a minimum distance percentile metric, and when the dataset privacy metric for particular synthetic data cases in the set of two or more synthetic data cases does not meet the dataset quality threshold, taking corrective action for the particular synthetic data cases in set of two or more synthetic data cases to produce a new set of two or more synthetic data cases to use as the set of two or more synthetic data cases.
6. The non-transitory computer readable medium of claim 1 , wherein determining at least one dataset quality metric comprises determining a dataset privacy metric based at least in part on a minimum expected distance to actual distance metric, and when the dataset privacy metric for particular synthetic data cases in the set of two or more synthetic data cases does not meet the dataset quality threshold, taking corrective action for the particular synthetic data cases in set of two or more synthetic data cases to produce a new set of two or more synthetic data cases to use as the set of two or more synthetic data cases.
7. The non-transitory computer readable medium of claim 1 , wherein determining at least one dataset quality metric comprises determining a dataset privacy metric based at least in part on a probability-based minimum distance metric, and when the dataset privacy metric for particular synthetic data cases in the set of two or more synthetic data cases does not meet the dataset quality threshold, taking corrective action for the particular synthetic data cases in set of two or more synthetic data cases to produce a new set of two or more synthetic data cases to use as the set of two or more synthetic data cases.
8. A system for executing instructions, wherein said instructions are instructions which, when executed by one or more computing devices, cause performance of a process including:
receiving a request for generation of synthetic data based on a set of training data cases that meets a dataset quality threshold;
repeatedly determining new a synthetic data case based on a set of one or more focal training data cases to generate a set of two or more synthetic data cases, wherein each set of one or more focal cases are determined from the set of training data cases;
determining a dataset quality metric for the set of two or more synthetic data cases based on the set of training data cases and the set of two or more synthetic data cases, wherein the dataset quality metric is determined based at least in part on:
at least one statistical quality metric that compares statistical properties of the set of training data cases and the set of two or more synthetic data cases;
at least one model comparison metric that quantifies machine learning model properties and performance of the set of training data cases and the set of two or more synthetic data cases;
at least one dataset privacy metric, which quantifies the likelihood of identification of private data in the set of training data cases from the set of two or more synthetic data cases;
when the dataset quality metric for particular synthetic data cases in the set of two or more synthetic data cases does not meet a dataset quality threshold, taking corrective action for the particular synthetic data cases in the set of two or more synthetic data cases to produce a new set of two or more synthetic data cases to use as the set of two or more synthetic data cases;
when the dataset quality metric for the set of two or more synthetic data cases meets the dataset quality threshold, causing control of a controllable system using the set of two or more synthetic data cases.
9. The system of claim 8 , wherein taking corrective action comprises one or more of:
modifying one or more of the particular synthetic data cases to produce a new set of two or more synthetic data cases to use as the set of two or more synthetic data cases;
deleting one or more of the particular synthetic data cases to produce a new set of two or more synthetic data cases to use as the set of two or more synthetic data cases; and
replacing one or more of the particular synthetic data cases to produce a new set of two or more synthetic data cases to use as the set of two or more synthetic data cases.
10. The system of claim 8 , wherein determining at least one dataset quality metric comprises determining a dataset privacy metric based at least in part on a data element distance comparison metric, and when the dataset privacy metric for particular synthetic data cases in the set of two or more synthetic data cases does not meet the dataset quality threshold, taking corrective action for the particular synthetic data cases in set of two or more synthetic data cases to produce a new set of two or more synthetic data cases to use as the set of two or more synthetic data cases.
11. The system of claim 8 , wherein determining at least one dataset quality metric comprises determining a dataset privacy metric based at least in part on a minimum distance ratio metric, and when the dataset privacy metric for particular synthetic data cases in the set of two or more synthetic data cases does not meet the dataset quality threshold, taking corrective action for the particular synthetic data cases in set of two or more synthetic data cases to produce a new set of two or more synthetic data cases to use as the set of two or more synthetic data cases.
12. The system of claim 8 , wherein determining at least one dataset quality metric comprises determining a dataset privacy metric based at least in part on a minimum distance percentile metric, and when the dataset privacy metric for particular synthetic data cases in the set of two or more synthetic data cases does not meet the dataset quality threshold, taking corrective action for the particular synthetic data cases in set of two or more synthetic data cases to produce a new set of two or more synthetic data cases to use as the set of two or more synthetic data cases.
13. The system of claim 8 , wherein determining at least one dataset quality metric comprises determining a dataset privacy metric based at least in part on a minimum expected distance to actual distance metric, and when the dataset privacy metric for particular synthetic data cases in the set of two or more synthetic data cases does not meet the dataset quality threshold, taking corrective action for the particular synthetic data cases in set of two or more synthetic data cases to produce a new set of two or more synthetic data cases to use as the set of two or more synthetic data cases.
14. The system of claim 8 , wherein determining at least one dataset quality metric comprises determining a dataset privacy metric based at least in part on a probability-based minimum distance metric, and when the dataset privacy metric for particular synthetic data cases in the set of two or more synthetic data cases does not meet the dataset quality threshold, taking corrective action for the particular synthetic data cases in set of two or more synthetic data cases to produce a new set of two or more synthetic data cases to use as the set of two or more synthetic data cases.
15. A method comprising:
receiving a request for generation of synthetic data based on a set of training data cases that meets a dataset quality threshold;
repeatedly determining new a synthetic data case based on a set of one or more focal training data cases to generate a set of two or more synthetic data cases, wherein each set of one or more focal cases are determined from the set of training data cases;
determining a dataset quality metric for the set of two or more synthetic data cases based on the set of training data cases and the set of two or more synthetic data cases, wherein the dataset quality metric is determined based at least in part on one or more of:
at least one statistical quality metric that compares statistical properties of the set of training data cases and the set of two or more synthetic data cases;
at least one model comparison metric that quantifies machine learning model properties and performance of the set of training data cases and the set of two or more synthetic data cases;
at least one dataset privacy metric, which quantifies the likelihood of identification of private data in the set of training data cases from the set of two or more synthetic data cases;
when the dataset quality metric for particular synthetic data cases in the set of two or more synthetic data cases does not meet a dataset quality threshold, taking corrective action for the particular synthetic data cases in the set of two or more synthetic data cases to produce a new set of two or more synthetic data cases to use as the set of two or more synthetic data cases;
when the dataset quality metric for the set of two or more synthetic data cases meets the dataset quality threshold, causing control of a controllable system using the set of two or more synthetic data cases.
16. The method of claim 15 , wherein taking corrective action comprises one or more of:
modifying one or more of the particular synthetic data cases to produce a new set of two or more synthetic data cases to use as the set of two or more synthetic data cases;
deleting one or more of the particular synthetic data cases to produce a new set of two or more synthetic data cases to use as the set of two or more synthetic data cases; and
replacing one or more of the particular synthetic data cases to produce a new set of two or more synthetic data cases to use as the set of two or more synthetic data cases.
17. The method of claim 15 , wherein determining at least one dataset quality metric comprises determining a dataset privacy metric based at least in part on a data element distance comparison metric, and when the dataset privacy metric for particular synthetic data cases in the set of two or more synthetic data cases does not meet the dataset quality threshold, taking corrective action for the particular synthetic data cases in set of two or more synthetic data cases to produce a new set of two or more synthetic data cases to use as the set of two or more synthetic data cases.
18. The method of claim 15 , wherein determining at least one dataset quality metric comprises determining a dataset privacy metric based at least in part on a minimum distance ratio metric, and when the dataset privacy metric for particular synthetic data cases in the set of two or more synthetic data cases does not meet the dataset quality threshold, taking corrective action for the particular synthetic data cases in set of two or more synthetic data cases to produce a new set of two or more synthetic data cases to use as the set of two or more synthetic data cases.
19. The method of claim 15 , wherein determining at least one dataset quality metric comprises determining a dataset privacy metric based at least in part on a minimum distance percentile metric, and when the dataset privacy metric for particular synthetic data cases in the set of two or more synthetic data cases does not meet the dataset quality threshold, taking corrective action for the particular synthetic data cases in set of two or more synthetic data cases to produce a new set of two or more synthetic data cases to use as the set of two or more synthetic data cases.
20. The method of claim 15 , wherein determining at least one dataset quality metric comprises determining a dataset privacy metric based at least in part on a minimum expected distance to actual distance metric, and when the dataset privacy metric for particular synthetic data cases in the set of two or more synthetic data cases does not meet the dataset quality threshold, taking corrective action for the particular synthetic data cases in set of two or more synthetic data cases to produce a new set of two or more synthetic data cases to use as the set of two or more synthetic data cases.