Secure and fair competitive bidding
There are provided a system, a method and computer program product providing a data-to-model challenge platform for enabling bidders publicly build, test, evaluate, validate and optimize AI models on proprietary data while at the same time avoiding the need to grant them access to the data itself. Rather, the bidder develops analytics on the enterprise's data to solve a task desirable to the organization and submits the analytics for evaluation against other bidders. The offering organization evaluates all submissions from the bidders and rank submissions against each other using the same metrics, wherein the metrics for selecting the winning bidder can include response time, number and rate of attempts to solve the analytics, quality of results, code compactness, team size etc. Allowing the submissions of the bidders to be visible only to the offering organization can occur subject to an agreement that the offering organization and the bidder sign.
1 . A method for a secure bidding process for bidding teams, the method comprising:
providing, via a communications portal to a provider's secure computing system, a challenge task specification requiring a bidding team to create a data science model solution (model) for a provider based on a provider's data set, said provider's dataset being stored in a secure storage system residing behind a firewall and prevented from direct access by said bidding team, and the challenge task specification comprising metadata indicating a type of data or data modality, labels or signal types that are to be detected or predicted, an amount or size of the data the bidding team's model is required to analyze and sample data representative of the data type, the metadata communicated to each bidding team via a template provided in a software container structure;
receiving, via the communications portal, from a plurality of respective bidding teams, a respective acceptance for building and training, by the respective bidding team, a respective model for solving said challenge task;
receiving, via said communications portal, a respective model built based on the metadata of said challenge task specification and submitted from a respective bidding team, said model submission being trained without remotely accessing said provider's data set;
evaluating, by a programmed processor of said provider's secure computing system, each bidding team's model submission against a common set of metrics;
selecting, by the programmed processor at the provider's secure computing system, a bidding team based on said submitted model evaluation, said bidding team to provide said model for use by said provider to run with a provider's data set,
wherein said model and additional objects for specifying a data flow through said model for training said model are wrapped inside the software container structure for communication over said communications portal, and said receiving said respective model submission comprising:
unpacking, at said computer system, said wrapped container structure to enable a training and evaluation of said model by said programmed processor at said secure computing system.
2 . The method as claimed in claim 1 , wherein a bidding team builds the model in a provider's secure network system.
3 . The method as claimed in claim 1 , wherein the common set of metrics used for evaluating a bidder's submitted model comprises one or more selected from: a response time, a number and rate of attempts to solve the analytics, a model accuracy, an amount of energy used to obtain a result, a quality of code, a quality of a result, a compactness of developed code, and a team size.
4 . The method as claimed in claim 3 , further comprising:
configuring said programmed processor as a leaderboard, said leaderboard automatically performing a ranking of said bidder team against each other team using said common set of metrics; and
automatically updating said leaderboard by each submission and presenting said bidder team ranking to the provider.
5 . The method as claimed in claim 1 , wherein a bidding team's model submission comprises:
a specification defining and initializing a machine-learned model including any input data pre-processing and hyperparameter values for training said model; and
a specification further comprising a flow of said input data through said defined and initialized machine-learned model including any post-processing of a model output data.
6 . A system for a secure bidding process for bidding teams, the system comprising:
a memory device;
a processor connected to the memory device,
wherein the processor is configured to:
provide, via a communications portal to a provider's secure computing system, a challenge task specification requiring a bidding team to create a data science model solution (model) for a provider based on a provider's data set, said provider's dataset being stored in a secure storage system residing behind a firewall and prevented from direct access by said bidding team, and the challenge task specification comprising metadata indicating a type of data or data modality, labels or signal types that are to be detected or predicted, an amount or size of the data the bidding team's model is required to analyze and sample data representative of the data type, the metadata communicated to each bidding team via a template provided in a software container structure;
receive, via the communications portal, from a plurality of respective bidding teams, a respective acceptance for building and training, by the respective bidding team, a respective model for solving said challenge task;
receive, via said communications portal, a respective model built based on the metadata of said challenge task specification and submitted from a respective bidding team, said model submission being trained without remotely accessing said provider's data set;
evaluate at said provider's secure computing system, each bidding team's model submission against a common set of metrics;
select at the provider's secure computing system, a bidding team based on said submitted model evaluation, said bidding team to provide said model for use by said provider to run with a provider's data set,
wherein said model and additional objects for specifying a data flow through said model for training said model are wrapped inside the software container structure for communication over said communications portal, and said receiving said respective model submission comprising:
unpacking, at said computer system, said wrapped container structure to enable a training and evaluation of said model by said programmed processor at said secure computing system.
7 . The system as claimed in claim 6 , wherein a bidding team builds the model in a provider's secure network system.
8 . The system as claimed in claim 6 , wherein the common set of metrics used for evaluating a bidder's submitted model comprises one or more selected from: a response time, a number and rate of attempts to solve the analytics, a model accuracy, an amount of energy used to obtain a result, a quality of code, a quality of a result, a compactness of developed code, and a team size.
9 . The system as claimed in claim 8 , wherein said processor is configured as a leaderboard for leaderboard automatically performing a ranking of said bidder team against each other team using said common set of metrics; and automatically updating said leaderboard by each submission and presenting said bidder team ranking to the provider.
10 . The system as claimed in claim 6 , wherein a bidding team's model submission comprises:
a specification defining and initializing a machine-learned model including any input data pre-processing and hyperparameter values for training said model; and
a specification further comprising a flow of said input data through said defined and initialized machine-learned model including any post-processing of a model output data.
11 . A computer program product for a secure bidding process for bidding teams, the computer program product comprising a computer readable storage medium, the computer readable storage medium excluding a propagating signal, the computer readable storage medium readable by a processing circuit and storing instructions run by the processing circuit for performing a method, said method steps comprising:
providing, via a communications portal to a provider's secure computing system, a challenge task specification requiring a bidding team to create a data science model solution (model) for a provider based on a provider's data set, said provider's dataset being stored in a secure storage system residing behind a firewall and prevented from direct access by said bidding team, and the challenge task specification comprising metadata indicating a type of data or data modality, labels or signal types that are to be detected or predicted, an amount or size of the data the bidding team's model is required to analyze and sample data representative of the data type, the metadata communicated to each bidding team via a template provided in a software container structure;
receiving, via the communications portal, from a plurality of respective bidding teams, a respective acceptance for building and training, by the respective bidding team, a respective model for solving said challenge task;
receiving, via said communications portal, a respective model built based on the metadata of said challenge task specification and submitted from a respective bidding team, said model submission being trained without remotely accessing said provider's data set;
evaluating, by a programmed processor of said provider's secure computing system, each bidding team's model submission against a common set of metrics; and
selecting, by the programmed processor at the provider's secure computing system, a bidding team based on said submitted model evaluation, said bidding team to provide said model for use by said provider to run with a provider's data set,
wherein said model and additional objects for specifying a data flow through said model for training said model are wrapped inside the software container structure for communication over said communications portal, and said receiving said respective model submission comprising:
unpacking, at said computer system, said wrapped container structure to enable a training and evaluation of said model by said programmed processor at said secure computing system.
12 . The computer program product as claimed in claim 11 , wherein a bidding team building the model in a provider's secure network system.
13 . The computer program product as claimed in claim 11 , wherein the common set of metrics used for evaluating a bidder's submitted model comprises one or more selected from: a response time, a number and rate of attempts to solve the analytics, a model accuracy, an amount of energy used to obtain a result, a quality of code, a quality of a result, a compactness of developed code, and a team size.
14 . The computer program product as claimed in claim 13 , wherein said method steps further comprise:
configuring said programmed processor as a leaderboard, said leaderboard automatically performing a ranking of said bidder team against each other team using said common set of metrics; and
automatically updating said leaderboard by each submission and presenting said bidder team ranking to the provider.
15 . The computer program product as claimed in claim 11 , wherein a bidding team's model submission comprises:
a specification defining and initializing a machine-learned model including any input data pre-processing and hyperparameter values for training said model; and
a specification further comprising a flow of said input data through said defined and initialized machine-learned model including any post-processing of a model output data.