Privacy and proprietary-information preserving collaborative multi-party machine learning
Provided is a process that includes sharing information among two or more parties or systems for modeling and decision-making purposes, while limiting the exposure of details either too sensitive to share, or whose sharing is controlled by laws, regulations, or business needs.
1. A tangible, non-transitory, machine-readable medium, storing instructions that when executed by one or more processors effectuate operations comprising:
obtaining, with one or more processors, a first trained machine learning model, wherein:
the first machine learning model is trained on a first training set that includes data the first entity is not permitted to provide to a second entity,
the first trained machine learning model is configured to output tokens, and
the tokens do not reveal more than a threshold amount of information about the data the first entity is not permitted to provide to the second entity;
receiving, with one or more processors, a first set of input features with the first trained machine learning model and, in response, outputting a first token; and
causing, with one or more processors, the first token and a first value associated with the first token to be input into a second trained machine learning model accessible to the second entity, wherein the first value associated with the first token is a token-context value corresponding to the first token.
2. The medium of claim 1 , wherein:
the first value is a member of a set of feature values input into the first model to cause the first model to output the first token.
3. The medium of claim 1 , the operations comprising:
causing the first token and a second value associated with the first token to be input into a third trained machine learning model accessible to a third entity but not the second entity.
4. The medium of claim 3 , wherein:
the first token is based on a value of a secret random variable input into the first model to generate the first token along with the first value and the second value.
5. The medium of claim 4 , wherein:
a first conditional probability distribution of the secret random variable given the first token and the first value is different from a second conditional probability distribution of the secret random variable given the first token and the second value.
6. The medium of claim 1 , wherein:
the token-context value is selected to modulate an amount of information provided to the second entity about the data the first entity is not permitted to provide to the second entity.
7. The medium of claim 1 , wherein the operations comprise:
receiving a second set of input features with the first trained machine learning model and, in response, outputting a second token; and
causing, with one or more processors, the second token and a second value associated with the second token to be input into the second trained machine learning model accessible to the second entity.
8. The medium of claim 7 , wherein:
the first token, the first value, the second token, and the second value are configured to be part of a training set by which the second machine learning model is trained or re-trained.
9. The medium of claim 7 , wherein:
the first token, the first value, the second token, and the second value are configured to be part of input feature sets by which the second machine learning model generates outputs.
10. The medium of claim 1 , wherein:
the operations comprise training the first machine learning model by adjusting parameters of the first machine learning model based on an objective function and a training set.
11. The medium of claim 10 , wherein:
the objective function is based on both an effectiveness of the first machine learning model at predicting label values in the training set and an amount of information conveyed by token values about at least some features in the training set to which the label values correspond.
12. The medium of claim 11 , wherein:
training the first machine learning model tends to adjust parameters of the first machine learning model to both increase accuracy of the first machine at predicting label values in the training set and decrease amounts of information conveyed by token values about secret features in the training set to which the label values correspond relative to results from model parameters subject to earlier adjustments in the training.
13. The medium of claim 10 , wherein:
the first machine learning model is a supervised machine learning model; and
training effectuates a lossy compression of the training set that is encoded in parameters of the first machine learning model.
14. The medium of claim 1 , wherein:
the first machine learning model or the second machine learning model comprise means for machine learning.
15. The medium of claim 1 , wherein:
the tokens correspond to a principle component of an input feature set of the first machine learning model with respect to an objective metric of the first machine learning model.
16. The medium of claim 15 , wherein:
the tokens are configured to discriminate outputs along a dimension corresponding to the principle component for the first training set.
17. The medium of claim 1 , wherein the operations comprise:
inputting the first token and a second token output by the second machine learning model into a rules bank.
18. The medium of claim 1 , wherein:
the first trained machine learning model is configured to accept a second token from an upstream third machine learning model as part of the first set of input features upon which the first token is based.
19. The medium of claim 1 , wherein the operations comprise:
concurrently training the first machine learning model and the second machine learning model by adjusting parameters of the first machine learning model and the second machine learning model based on the first training set, a second training set of the second entity, and a single objective function applied to training both the first machine learning model and the second machine learning model.
20. A method, comprising:
obtaining, with one or more processors, a first trained machine learning model, wherein:
the first machine learning model is trained on a first training set that includes data the first entity is not permitted to provide to a second entity,
the first trained machine learning model is configured to output tokens, and
the tokens do not reveal more than a threshold amount of information about the data the first entity is not permitted to provide to the second entity;
receiving, with one or more processors, a first set of input features with the first trained machine learning model and, in response, outputting a first token; and
causing, with one or more processors, the first token and a first value associated with the first token to be input into a second trained machine learning model accessible to the second entity, wherein the first value associated with the first token is a token-context value corresponding to the first token.