Distributed privacy-preserving computing on protected data
The present disclosure relates to techniques for developing artificial intelligence algorithms by distributing analytics to multiple sources of privacy protected, harmonized data. Particularly, aspects are directed to a computer implemented method that includes receiving an algorithm and input data requirements associated with the algorithm, identifying data assets as being available from a data host based on the input data requirements, curating the data assets within a data storage structure that is within infrastructure of the data host, and integrating the algorithm into a secure capsule computing framework. The secure capsule computing framework serves the algorithm to the data assets within the data storage structure in a secure manner that preserves privacy of the data assets and the algorithm. The computer implemented method further includes running the data assets through the algorithm to obtain an inference.
1 . A system comprising:
one or more data processors; and
a non-transitory computer readable storage medium containing instructions which, when executed on the one or more data processors, cause the one or more data processors to perform actions including:
receiving an algorithm and input data requirements associated with the algorithm, wherein the input data requirements include optimization and/or validation selection criteria for data assets to be run on the algorithm;
obtaining the data assets based on the optimization and/or validation selection criteria for the data assets;
preparing the data assets for processing by the algorithm, wherein the preparing comprises applying harmonizing transformers to the data assets to generate harmonized data assets; and
running the harmonized data assets through the algorithm, wherein the algorithm is in a secure capsule computing framework that serves the algorithm to the harmonized data assets in a secure manner that preserves privacy of the data assets and the algorithm.
2 . The system of claim 1 , wherein the preparing further comprises:
preparing a transformer prototype set of data to use as a guide for developing algorithms for data transformation, wherein the transformer prototype set of data captures key attributes of a harmonization process;
creating a first set of harmonizing transformers for transformation of the data assets based on a present format of data in the transformer prototype set of data;
applying the first set of harmonizing transformers to the data assets to generate transformed data assets;
preparing a harmonization prototype set of data to use as a guide for developing algorithms for data transformation, wherein the harmonization prototype set of data captures key attributes of the harmonization process;
creating a second set of harmonizing transformers for transformation of the transformed data assets based on a present format of data in the harmonization prototype set of data; and
applying the second set of harmonizing transformers to the transformed data assets to generate the harmonized data assets.
3 . The system of claim 2 , wherein the preparing further comprises de-identifying the transformer prototype set of data and making the de-identified transformer prototype set of data available to an algorithm developer for the purpose of creating the first set of harmonizing transformers for transformation of the data assets.
4 . The system of claim 2 , wherein the preparing further comprises annotating the transformed data assets according to a predefined annotation protocol to generate annotated data sets, wherein the second set of harmonizing transformers is applied to the annotated data sets to generate harmonized data assets.
5 . The system of claim 2 , wherein the applying the first set of harmonizing transformers to the data assets and the applying the second set of harmonizing transformers to the annotated data assets are performed within a data storage structure separate from the secure capsule computing framework.
6 . The system of claim 1 , wherein the harmonized data assets through the algorithm comprises executing a training workflow that includes: creating multiple instances of the algorithm, splitting the harmonized data assets into sets of training data and one or more sets of testing data, training the multiple instances of the algorithm on the sets of training data, integrating results from the training each of the multiple instances of the algorithm into a fully federated model, running the one or more sets of testing data through the fully federated model, and computing performance of the fully federated model based on the running of the one or more sets of testing data.
7 . The system of claim 1 , wherein the running the harmonized data assets through the algorithm comprises executing a validation workflow that includes: splitting the data assets in one or more sets of validation data, running the one or more sets of validation data through the algorithm, and computing performance of the algorithm based on the running of the one or more sets of validation data.
8 . The system of claim 1 , wherein the secure capsule computing framework is provisioned within a computing infrastructure configured to accept encrypted code required to run the algorithm.
9 . The system of claim 1 , wherein the actions further comprise reporting aggregate results of the training workflow and performance metrics for the fully federated model.
10 . A system comprising:
one or more data processors; and
a non-transitory computer readable storage medium containing instructions which, when executed on the one or more data processors, cause the one or more data processors to perform actions including:
receiving an algorithm and input data requirements associated with the algorithm, wherein the input data requirements include optimization and/or validation selection criteria for data assets to be run on the algorithm;
obtaining the data assets based on the optimization and/or validation selection criteria for the data assets;
preparing the data assets for processing by the algorithm; and
running the data assets through the algorithm, wherein the running comprises: passing the data assets from a data storage structure to the algorithm in a secure capsule computing framework, and optimizing, validating, or computing inference with the algorithm using the data assets, wherein the algorithm is in a secure capsule computing framework that serves the algorithm to the data assets in accordance with encrypted code stored inside the secure capsule computing framework.
11 . The system of claim 10 , wherein the algorithm is optimized using the data assets, and wherein the running the data assets through the algorithm comprises executing a training workflow that includes: creating multiple instances of the algorithm, splitting the data assets into sets of training data and one or more sets of testing data, training the multiple instances of the algorithm on the sets of training data, integrating results from the training each of the multiple instances of the algorithm into a fully federated model, running the one or more sets of testing data through the fully federated model, and computing performance of the fully federated model based on the running of the one or more sets of testing data.
12 . The system of claim 10 , wherein the running the data assets through the algorithm comprises executing a validation workflow that includes: splitting the data assets in one or more sets of validation data, running the one or more sets of validation data through the algorithm, and computing performance of the algorithm based on the running of the one or more sets of validation data.
13 . The system of claim 12 , wherein the actions further comprise reporting optimized hyperparameters and performance metrics of the algorithm.
14 . The system of claim 10 , wherein the optimization and/or validation selection criteria define characteristics, formats and requirements for the data assets to be run on the algorithm.
15 . The system of claim 10 , wherein the preparing the data assets comprises applying one or more transforms to the data assets, annotating the data assets, harmonizing the data assets, or a combination thereof.
16 . The system of claim 10 , wherein the obtaining comprises:
identifying the data assets as being available based on the optimization and/or validation selection criteria for the data assets;
selecting a data storage structure from multiple data storage structures based on a type of the algorithm, a type of data within the data assets, system requirements of a data processing system, or a combination thereof;
provisioning the data storage structure within an infrastructure; and
curating the data assets within the data storage structure.