Techniques for adaptive and context-aware automated service composition for machine learning (ML)
The present disclosure relates to systems and methods for using existing data ontologies for generating machine learning solutions for a high-precision search of relevant services to compose pipelines with minimal human intervention. Data ontologies can be used to create a combination of non-logic based and logic-based sematic services that can significantly outperform both kinds of selection in terms of precision. Quality of Service (QoS) and product Key Performance Indicator (KPI) constraints can be used as part of architecture selection in developing, training, validating, and improving machine learning models. For data sets without existing ontologies, one or more ontologies be generated and stored for future use.
1. A method for automatically creating a machine learning application bespoke to a hardware platform for use in a production environment, the method comprising:
receiving a first input, wherein the first input identifies a location of data;
receiving a second input, wherein the second input describes a prediction for the machine learning application;
receiving a third input, wherein the third input comprises one or more constraints for the machine learning application;
accessing a memory containing one or more ontologies of the data, wherein each ontology describes a hierarchy for one or more data sets of one or more data objects, wherein each data object exhibits one or more attributes that allows the data object to be classified into the one or more sets of data objects according to rules;
extracting one of more attributes of the one or more data sets to find a first ontology that correlates to the prediction for the machine learning application according to the one or more constraints;
generating a product graph based on the first ontology, the one or more constraints, and one or more previous product graphs stored in the memory, wherein the product graph relates the one or more data objects to a collection of nodes and edges, wherein the edges represent links between the nodes; wherein nodes comprises a basic unit of a data structure; and
storing the product graph in the memory.
2. The method of claim 1 , further comprising:
generating an ontology based at least in part on analyzing the attributes of the one or more data objects, wherein the one or more attributes comprise:
one or more functional semantics of each service, the functional semantics include at least one of a feature selection, an outlier detection, and an explainability metric;
one or more preconditions and one or more effects, wherein the preconditions are conditions that exist before executing a service and the one or more effects are conditions that exist after executing the service;
one or more parameters, wherein the one or more parameters concerned with a provenance of the service;
one or more metrics, wherein the one or more metrics comprise quality of service metrics; and
one or more indicators, wherein the one or more indicators comprise key performance indicators; and
storing the ontology in the memory.
3. The method of claim 2 , wherein the quality of service metrics comprises at least one of latency metrics, response time metrics, throughput metrics, reliability metrics, availability metrics, success rate metrics, and privacy metrics.
4. The method of claim 1 , further comprising:
generating metadata from the one or more ontologies, wherein the metadata describes one or more concepts of the one or more ontologies from a machine learning perspective based at least in part on run-time expectations; and
saving the metadata to a memory.
5. The method of claim 1 , further comprising:
searching one or more services based at least in part on one or more descriptors from the one or more previous product graphs;
correlating the one or more descriptors with the prediction of the machine learning application;
generating a list of available services based at least in part on said correlating; and
saving the list of available services in a memory.
6. The method of claim 5 , wherein the searching uses a metaheuristics approach for determining a solution.
7. The method of claim 1 , further comprising:
presenting the product graph on a display, the product graph including one or more metrics of a performance of the product graph;
determining a set of compromises, wherein the set of compromises comprises a selection of the one or more constraints in view of the prediction of the machine learning application; and
presenting the set of compromises on a display.
8. A computer-program product tangibly embodied in a non-transitory machine-readable storage medium, including instructions configured to cause a data processing apparatus to perform operations for automatically creating a machine learning application for use in a production environment, the operations comprising:
receiving a first input, wherein the first input identifies a location of data;
receiving a second input, wherein the second input describes a prediction for the machine learning application;
receiving a third input, wherein the third input comprises one or more constraints for the machine learning application;
accessing a memory containing one or more ontologies of the data, wherein each ontology describes one or more data sets of one or more data objects, wherein each data object exhibits one or more attributes that allows the data object to be classified into the one or more sets of data objects according to rules;
extracting one of more attributes of the one or more data sets to find a first ontology that correlates to the prediction for the machine learning application according to the one or more constraints;
generating a product graph based on the first ontology, the one or more constraints, and one or more previous product graphs stored in the memory, wherein the product graph relates the one or more data objects to a collection of nodes and edges, wherein the edges represent links between the nodes; wherein nodes comprises a basic unit of a data structure; and
storing the product graph in the memory.
9. The computer-program product of claim 8 , including instructions configured to cause the data processing apparatus to perform further operations comprising:
generating an ontology based at least in part on analyzing the attributes of the one or more data objects, wherein the one or more attributes comprise:
one or more functional semantics of each service, the functional semantics include at least one of a feature selection, an outlier detection, and an explainability metric;
one or more preconditions and one or more effects, wherein the preconditions are conditions that exist before executing a service and the one or more effects are conditions that exist after executing the service;
one or more parameters, wherein the one or more parameters concerned with a provenance of the service;
one or more metrics, wherein the one or more metrics comprise quality of service metrics; and
one or more indicators, wherein the one or more indicators comprise key performance indicators; and
storing the ontology in the memory.
10. The computer-program product of claim 9 , wherein the quality of service metrics comprises at least one of latency metrics, response time metrics, throughput metrics, reliability metrics, availability metrics, success rate metrics, and privacy metrics.
11. The computer-program product of claim 8 , including instructions configured to cause the data processing apparatus to perform further operations comprising:
generating metadata from the one or more ontologies, wherein the metadata describes one or more concepts of the one or more ontologies from a machine learning perspective based at least in part on run-time expectations; and
saving the metadata to a memory.
12. The computer-program product of claim 8 , including instructions configured to cause the data processing apparatus to perform further operations comprising:
searching one or more services based at least in part on one or more descriptors from the one or more previous product graphs;
correlating the one or more descriptors with the prediction of the machine learning application;
generating a list of available services based at least in part on said correlating; and
saving the list of available services in a memory.
13. The computer-program product of claim 12 , wherein the searching uses a metaheuristics approach for determining a solution.
14. The computer-program product of claim 8 , including instructions configured to cause the data processing apparatus to perform further operations comprising:
presenting the product graph on a display, the product graph including one or more metrics of a performance of the product graph;
determining a set of compromises, wherein the set of compromises comprises a selection of the one or more constraints in view of the prediction of the machine learning application; and
presenting the set of compromises on a display.
15. A system for automatically creating a machine learning application for use in a production environment, comprising:
one or more data processors; and
a non-transitory computer-readable storage medium containing instructions which, when executed on the one or more data processors, cause the one or more data processors to perform operations comprising:
receiving a first input, wherein the first input identifies a location of data;
receiving a second input, wherein the second input describes a prediction for the machine learning application;
receiving a third input, wherein the third input comprises one or more constraints for the machine learning application;
accessing a memory containing one or more ontologies of the data, wherein each ontology describes one or more data sets of one or more data objects, wherein each data object exhibits one or more attributes that allows the data object to be classified into the one or more sets of data objects according to rules;
extracting one of more attributes of the one or more data sets to find a first ontology that correlates to the prediction for the machine learning application according to the one or more constraints;
generating a product graph based on the first ontology, the one or more constraints, and one or more previous product graphs stored in the memory, wherein the product graph relates the one or more data objects to a collection of nodes and edges, wherein the edges represent links between the nodes; wherein nodes comprises a basic unit of a data structure; and
storing the product graph in the memory.
16. The system of claim 15 , wherein the instructions are configured to cause the one or more data processors to perform further operations comprising:
generating an ontology based at least in part on analyzing the attributes of the one or more data objects, wherein the one or more attributes comprise:
one or more functional semantics of each service, the functional semantics include at least one of a feature selection, an outlier detection, and an explainability metric;
one or more preconditions and one or more effects, wherein the preconditions are conditions that exist before executing a service and the one or more effects are conditions that exist after executing the service;
one or more parameters, wherein the one or more parameters concerned with a provenance of the service;
one or more metrics, wherein the one or more metrics comprise quality of service metrics; and
one or more indicators, wherein the one or more indicators comprise key performance indicators; and
storing the ontology in the memory.
17. The system of claim 16 , wherein the quality of service metrics comprises at least one of latency metrics, response time metrics, throughput metrics, reliability metrics, availability metrics, success rate metrics, and privacy metrics.
18. The system of claim 16 , wherein the instructions are configured to cause the one or more data processors to perform further operations comprising:
generating metadata from the one or more ontologies, wherein the metadata describes one or more concepts of the one or more ontologies from a machine learning perspective based at least in part on run-time expectations; and
saving the metadata to a memory.
19. The system of claim 17 , wherein the non-transitory computer-readable storage medium includes further instructions which, when executed on the one or more data processors, cause the one or more data processors to perform further operations comprising:
searching one or more services based at least in part on one or more descriptors from the one or more previous product graphs;
correlating the one or more descriptors with the prediction of the machine learning application;
generating a list of available services based at least in part on said correlating; and
saving the list of available services in a memory.
20. The system of claim 19 , wherein the searching uses a metaheuristics approach for determining a solution.