Digital assistant extension automatic ranking and selection

1. A method for selecting and ranking service extensions, comprising:

receiving a query from a user or an inference related to the user;

identifying any bots that activate in response to the query or the inference;

ranking the activated bots by performing acts for each activated bot comprising:

identifying a set of features corresponding to the activated bot, at least a portion of the set of features being pre-calculated based on the activated bot processing a set of curated queries;

presenting at least a portion of the set of features to a ranking and response selection model trained via a machine learning process using the set of curated queries;

identifying a rank for the activated bot using the ranking and response selection model and the portion of the set of features; and

placing the activated bot in rank order compared to other activated bots to create a ranked list of bots;

selecting at least one bot from the ranked list of bots; and

presenting an output of the at least one bot to the user.

2. The method of claim 1 , further comprising converting the query from one format to another format for processing.

3. The method of claim 1 , wherein the set of features comprises at least one of:

features extracted from results of a language understanding model;

features extracted from how well a conversation progress under a hypothesis that a specific response is accepted and presented to the user;

features extracted from statistics collected for models used by tasks activities of the activated bot;

features extracted from a configuration of the activated bot; and

features extracted from an input source.

4. The method of claim 1 , wherein the set of features further comprises at least one of:

a correlation metric between a language understanding model of a bot and a built-in language understanding model;

a list of tasks that the bot handles; and

at least one metric describing at least one query that causes the bot to trigger.

5. The method of claim 1 , further comprising;

retrieving a set of curated queries;

executing the set of curated queries against a submitted bot;

gathering a set of features characterizing a response by the submitted bot to the set of curated queries; and storing the set of features.

6. The method of claim 5 , further comprising:

extracting from the submitted bot a set of features comprising tasks handled by the submitted bot.

7. The method of claim 5 , further comprising;

identifying a set of positive trigger examples or a set of negative trigger examples or both based on the response by the submitted bot to the set curated queries.

8. The method of claim 1 , further comprising:

identifying a set of representative bots;

identifying a training set of curated queries; and

using a machine learning process, creating the ranking and response selection model from the set of represented bots and the training set of curated queries.

9. A computing system comprising:

a processor and executable instructions accessible on a machine-readable medium that, when executed, cause the computing system to perform operations comprising:

receiving a query from a user or an inference related to the user;

identifying any bots that activate in response to the query or the inference;

ranking the activated bots by performing operations for each activated bot comprising:

identifying a set of features corresponding to the activated bot, at least a portion of the set of features being pre-calculated based on the activated bot processing a set of c mated queries;

presenting at least a portion of the set of features to a ranking and response selection model trained via a machine learning process using the set of curated queries;

identifying a rank for the activated bot using the ranking and response selection model and the portion of the set of features; and

placing the activated bot in rank order compared to other activated bots to create a ranked list of bots;

selecting at least one bot from the ranked list of bots; and

presenting an output of the at least one bot to the user.

10. The computing system of claim 9 , wherein the set of features comprises features corresponding to the activated bot in the context of other activated bots and features calculated from a set of curated queries.

11. The computing system of claim 9 , wherein the set of features comprises at least one of;

a confidence score of a language understanding model;

an indicator of entities detected by the language understanding model;

a confidence score distribution;

a frequency of triggering on top of an existing intent; and

an indicator of a type of queries that cause triggering.

12. The computing system of claim 9 , wherein the set of features further comprises at least one of:

a correlation metric between a language understanding model of a bot and a built-in language understanding model;

a list of tasks that the bot handles; and

at least one metric describing at least one query that causes the bot to trigger.

13. The computing system of claim 9 , further comprising:

retrieving a set of curated queries;

executing the set of curated queries against a submitted bot;

gathering a set of features characterizing a response by the submitted bot to the set of coated queries; and

storing the set of features.

14. The computing system of claim 13 , further comprising:

extracting from the submitted bot a set of features comprising tasks handled by the submitted bot.

15. The computing system of claim 13 , further comprising:

identifying a set of positive trigger examples or a set of negative trigger examples or both based on the submitted bot response to the set of curated queries.

16. The computing system of claim 9 , further comprising:

identifying a set of representative bots;

identifying a training set of curated queries; and

using a machine learning process, creating the ranking and response selection model from the set of represented bots and the training set of curated queries.

17. A non-transitory machine-readable medium having executable instructions encoded thereon, which, when executed by at least one processor of a machine, cause the machine to perform operations comprising:

training a ranking and response selection model by performing operations comprising:

identifying a set of representative bots;

identifying a training set of curated queries; and

using a machine learning process, creating the ranking and response selection

model from the set of represented bots and the training set of curated queries;

receiving a query from a user or an inference related to the user;

identifying any bots that activate in response to the query or the inference:

ranking the activated bots by performing operations for each activated bot comprising:

identifying a set of features corresponding to the activated bot in the context of other activated bots;

identifying a second set of features calculated from a second set of curated queries;

combining the set of features and the second set of features into a set of extracted features;

presenting a subset of the set of extracted features to the ranking and response selection model;

identifying a rank for the activated bot using the ranking and response selection model and the subset of extracted features; and

placing the activated bot in rank order compared to other activated bots to create a ranked list of bots;

selecting at least one bot from the ranked list of bots; and

presenting an output of the at least one bot to the user.

18. The non-transitory machine-readable medium of claim 17 , wherein the machine learning process comprises at least one of:

a regression based method;

an instance based method;

a regularization method;

a decision tree method;

a Bayesian method;

a clustering method;

an association rule learning method;

a neural network method;

a deep learning method;

a dimensionality reduction method; and

an ensemble method.

19. The non-transitory machine-readable medium of claim 17 , wherein the second set of features is extracted by operations comprising:

retrieving a set of curated queries;

executing the set of curated queries against a submitted bot;

gathering the second set of features characterizing a response by the submitted bot to the set of curated queries; and

storing the set of features.

20. The non-transitory machine-readable medium of claim 17 , wherein the set of features or second set of features comprises at least one of:

features extracted from results of a language understanding model;

features extracted from how well a conversation progress under a hypothesis that a specific response is accepted and presented to the user;

features extracted from statistics collected for models used by tasks activities of the activated bot;

features extracted from a configuration of the activated bot; and

features extracted from an input source.