Identifier contribution allocation in synthetic data generation in computer-based reasoning systems
Techniques for synthetic data generation in computer-based reasoning systems are discussed and include receiving a request for generation of synthetic data based on a set of training data cases. One or more focal training data cases are determined. For undetermined features (either all of them or those that are not subject to conditions), a value for the feature is determined based on the focal cases. In some embodiments, the generated synthetic data may be checked for similarity against the training data, and if similarity conditions are met, it may be modified (e.g., resampled), removed, and/or replaced.
1. A system for executing instructions, wherein said instructions are instructions which, when executed by one or more computing devices, cause performance of a process including:
receiving a request for generation of synthetic data based on a set of training data cases;
generating a set of two or more synthetic data cases by repeatedly determining new a synthetic data case based on a set of one or more focal training data cases, wherein each set of one or more focal cases are determined from the set of training data cases;
determining a dataset quality metric for the set of two or more synthetic data cases based on the set of training data cases and the set of two or more synthetic data cases, wherein the dataset quality metric is determined based at least in part on at least one dataset privacy metric, which quantifies the likelihood of identification of private data in the set of training data cases from the set of two or more synthetic data cases;
when the dataset quality metric for particular synthetic data cases in the set of two or more synthetic data cases does not meet a dataset quality threshold, taking corrective action for the particular synthetic data cases in the set of two or more synthetic data cases to produce a new set of two or more synthetic data cases to use as the set of two or more synthetic data cases, wherein taking corrective action comprises holding out one or more of the particular synthetic data cases to produce the new set of two or more synthetic data cases to use as the set of two or more synthetic data cases;
when the dataset quality metric for the set of two or more synthetic data cases meets the dataset quality threshold, causing control of a controllable system using the set of two or more synthetic data cases.
2. The system of claim 1 , wherein taking corrective action further comprises modifying at least one of the one or more of the particular synthetic data cases to produce the new set of two or more synthetic data cases to use as the set of two or more synthetic data cases.
3. The system of claim 1 , wherein taking corrective action further comprises deleting at least one of the one or more of the particular synthetic data cases to produce the new set of two or more synthetic data cases to use as the set of two or more synthetic data cases.
4. The system of claim 1 , wherein taking corrective action further comprises replacing at least one of the one or more of the particular synthetic data cases to produce the new set of two or more synthetic data cases to use as the set of two or more synthetic data cases.
5. The system of claim 1 , wherein determining at least one dataset quality metric comprises determining a dataset privacy metric based at least in part on a data element distance comparison metric, and when the dataset privacy metric for particular synthetic data cases in the set of two or more synthetic data cases does not meet the dataset quality threshold, taking corrective action for the particular synthetic data cases in the set of two or more synthetic data cases to produce the new set of two or more synthetic data cases to use as the set of two or more synthetic data cases.
6. The system of claim 1 , wherein determining at least one dataset quality metric comprises determining a dataset privacy metric based at least in part on a minimum distance ratio metric, and when the dataset privacy metric for particular synthetic data cases in the set of two or more synthetic data cases does not meet the dataset quality threshold, taking corrective action for the particular synthetic data cases in the set of two or more synthetic data cases to produce the new set of two or more synthetic data cases to use as the set of two or more synthetic data cases.
7. The system of claim 1 , wherein determining at least one dataset quality metric comprises determining a dataset privacy metric based at least in part on a minimum distance percentile metric, and when the dataset privacy metric for particular synthetic data cases in the set of two or more synthetic data cases does not meet the dataset quality threshold, taking corrective action for the particular synthetic data cases in the set of two or more synthetic data cases to produce the new set of two or more synthetic data cases to use as the set of two or more synthetic data cases.
8. The system of claim 1 , wherein determining at least one dataset quality metric comprises determining a dataset privacy metric based at least in part on a minimum expected distance to actual distance metric, and when the dataset privacy metric for particular synthetic data cases in the set of two or more synthetic data cases does not meet the dataset quality threshold, taking corrective action for the particular synthetic data cases in the set of two or more synthetic data cases to produce the new set of two or more synthetic data cases to use as the set of two or more synthetic data cases.
9. The system of claim 1 , wherein determining at least one dataset quality metric comprises determining a dataset privacy metric based at least in part on a probability-based minimum distance metric, and when the dataset privacy metric for particular synthetic data cases in the set of two or more synthetic data cases does not meet the dataset quality threshold, taking corrective action for the particular synthetic data cases in the set of two or more synthetic data cases to produce the new set of two or more synthetic data cases to use as the set of two or more synthetic data cases.
10. A non-transitory computer readable medium storing instructions which, when executed by one or more computing devices, cause the one or more computing devices to perform a process including:
receiving a request for generation of synthetic data based on a set of training data cases;
generating a set of two or more synthetic data cases by repeatedly determining new a synthetic data case based on a set of one or more focal training data cases, wherein each set of one or more focal cases are determined from the set of training data cases;
determining a dataset quality metric for the set of two or more synthetic data cases based on the set of training data cases and the set of two or more synthetic data cases, wherein the dataset quality metric is determined based at least in part on:
at least one statistical quality metric that compares statistical properties of the set of training data cases and the set of two or more synthetic data cases;
at least one model comparison metric that quantifies machine learning model properties and performance of the set of training data cases and the set of two or more synthetic data cases;
at least one dataset privacy metric, which quantifies the likelihood of identification of private data in the set of training data cases from the set of two or more synthetic data cases;
when the dataset quality metric for particular synthetic data cases in the set of two or more synthetic data cases does not meet a dataset quality threshold, taking corrective action for the particular synthetic data cases in the set of two or more synthetic data cases to produce a new set of two or more synthetic data cases to use as the set of two or more synthetic data cases, wherein taking corrective action comprises holding out one or more of the particular synthetic data cases to produce the new set of two or more synthetic data cases to use as the set of two or more synthetic data cases;
when the dataset quality metric for the set of two or more synthetic data cases meets the dataset quality threshold, causing control of a controllable system using the set of two or more synthetic data cases.
11. The method non-transitory computer readable medium of claim 10 , wherein taking corrective action further comprises one or more of:
modifying at least one of the one or more of the particular synthetic data cases to produce the new set of two or more synthetic data cases to use as the set of two or more synthetic data cases;
deleting at least one of the one or more of the particular synthetic data cases to produce the new set of two or more synthetic data cases to use as the set of two or more synthetic data cases; and
replacing at least one of the one or more of the particular synthetic data cases to produce the new set of two or more synthetic data cases to use as the set of two or more synthetic data cases.
12. The method non-transitory computer readable medium of claim 10 , wherein determining at least one dataset quality metric comprises determining a dataset privacy metric based at least in part on a data element distance comparison metric, and when the dataset privacy metric for particular synthetic data cases in the set of two or more synthetic data cases does not meet the dataset quality threshold, taking corrective action for the particular synthetic data cases in the set of two or more synthetic data cases to produce the new set of two or more synthetic data cases to use as the set of two or more synthetic data cases.
13. The non-transitory computer readable medium of claim 10 , wherein determining at least one dataset quality metric comprises determining a dataset privacy metric based at least in part on a minimum expected distance to actual distance metric, and when the dataset privacy metric for particular synthetic data cases in the set of two or more synthetic data cases does not meet the dataset quality threshold, taking corrective action for the particular synthetic data cases in the set of two or more synthetic data cases to produce the new set of two or more synthetic data cases to use as the set of two or more synthetic data cases.
14. The non-transitory computer readable medium of claim 10 , wherein determining at least one dataset quality metric comprises determining a dataset privacy metric based at least in part on a probability-based minimum distance metric, and when the dataset privacy metric for particular synthetic data cases in the set of two or more synthetic data cases does not meet the dataset quality threshold, taking corrective action for the particular synthetic data cases in the set of two or more synthetic data cases to produce the new set of two or more synthetic data cases to use as the set of two or more synthetic data cases.
15. The non-transitory computer readable medium of claim 10 , wherein the statistical quality metric may be determined based on a joint distribution metrics, measuring how well joint distributions of the training data cases and the two or more synthetic data cases match.
16. The non-transitory computer readable medium of claim 10 , wherein the statistical quality metric may be determined based on a marginal distribution metrics, measuring how well marginal distributions of training data cases and the two or more synthetic data cases match.
17. The non-transitory computer readable medium of claim 10 , wherein the statistical quality metric may be determined based on a graph quality metric, measuring quality of data synthesis in the context of graphs.
18. A non-transitory computer readable medium storing instructions which, when executed by one or more computing devices, cause the one or more computing devices to perform a process including:
receiving a request for generation of synthetic data based on a set of training data cases;
generating a set of two or more synthetic data cases by repeatedly determining new a synthetic data case based on a set of one or more focal training data cases, wherein each set of one or more focal cases are determined from the set of training data cases;
determining a dataset quality metric for the set of two or more synthetic data cases based on the set of training data cases and the set of two or more synthetic data cases, wherein the dataset quality metric is determined based at least in part on one or more of:
at least one statistical quality metric that compares statistical properties of the set of training data cases and the set of two or more synthetic data cases;
at least one model comparison metric that quantifies machine learning model properties and performance of the set of training data cases and the set of two or more synthetic data cases;
at least one dataset privacy metric, which quantifies the likelihood of identification of private data in the set of training data cases from the set of two or more synthetic data cases;
when the dataset quality metric for particular synthetic data cases in the set of two or more synthetic data cases does not meet a dataset quality threshold, taking corrective action for the particular synthetic data cases in the set of two or more synthetic data cases to produce a new set of two or more synthetic data cases to use as the set of two or more synthetic data cases, wherein taking corrective action comprises holding out one or more of the particular synthetic data cases to produce the new set of two or more synthetic data cases to use as the set of two or more synthetic data cases;
when the dataset quality metric for the set of two or more synthetic data cases meets the dataset quality threshold, causing control of a controllable system using the set of two or more synthetic data cases.
19. The non-transitory computer readable medium of claim 18 , wherein taking corrective action comprises one or more of:
modifying at least one of the one or more of the particular synthetic data cases to produce the new set of two or more synthetic data cases to use as the set of two or more synthetic data cases;
deleting at least one of the one or more of the particular synthetic data cases to produce the new set of two or more synthetic data cases to use as the set of two or more synthetic data cases; and
replacing at least one of the one or more of the particular synthetic data cases to produce the new set of two or more synthetic data cases to use as the set of two or more synthetic data cases.