Granted Patent B2

US 12,287,792 · App. 17/938,494 · Granted Apr 29, 2025

Inventors: Mariana Sá Correia Leite Almeida (Lisbon, PT); Bruno Emanuel Da Graça Martins (Amora, PT); Tiago Manuel Reis Mesquita (Lisbon, PT)

Assignee: Zendesk, Inc.

G06F16/24556

Abstract

Claims (132)

1. A method, comprising:

storing queries and templates;

selecting a subset of the stored queries based on one or more of:

a percentage of all stored queries; or

a time period in which each stored query was received;

iteratively performing the following steps until a performance metric for a model for matching a future query to one of the stored templates exceeds a performance threshold:

selecting a random subset of the stored templates;

generating a distribution based on the subset of stored queries and the random subset of templates by identifying each query of the subset of stored queries associated with each template of the random subset of stored templates;

selecting a subset of queries from the distribution;

generating a batch of training data based on the selected subset of queries from the distribution, wherein the batch of training data comprises one or more positively labeled training instances and one or more negatively labeled training instances, and

executing a multi-term loss function on the batch of training data to train the model for matching the future query to one of the stored templates,

wherein the multi-term loss function comprises:

a first weighted loss term corresponding to a similarity between a query of the subset of queries from the distribution and a template of the random subset of stored templates;

a second weighted loss term corresponding to a similarity between a first query of the subset of queries from the distribution and a second query of the subset of queries from the distribution;

a third weighted loss term corresponding to a similarity between a template of the random subset of stored templates and a second template of the random subset of stored templates; and

a fourth weighted loss term corresponding to a similarity between a template of the random subset of stored templates and a query of the subset of queries from the distribution, and

wherein the multi-term loss function is configured to reduce a number of training iterations until the performance metric for the model for matching the future query to one of the stored templates exceeds the performance threshold.

2. The method of claim 1 , wherein executing the multi-term loss function comprises:

for each respective query in the random subset of queries in the batch of training data:

executing the multi-term loss function for each combination of the respective query and each template of the random subset of stored templates to yield a query/template value; and

aggregating the query/template value for the respective query in the first weighted loss term.

3. The method of claim 2 , wherein executing the multi-term loss function further comprises:

for each respective query in the random subset of queries in the batch of training data:

executing the multi-term loss function for each combination of the respective query and another query in the random subset of queries to yield a query/query value; and

aggregating the query/query value for the respective query in the second weighted loss term.

4. The method of claim 2 , wherein executing the multi-term loss function further comprises:

for each respective template in the random subset of stored templates:

executing the multi-term loss function for each combination of the respective template and another template in the random subset of stored templates to yield a template/template value; and

aggregating the template/template value for the respective template in the third weighted loss term.

5. The method of claim 2 , wherein executing the multi-term loss function further comprises:

for each respective template in the random subset of stored templates:

executing the multi-term loss function for each combination of the respective template and a query in the random subset of queries to yield a template/query value; and

aggregating the template/query value for the respective template in the fourth weighted loss term.

6. The method of claim 2 , further comprising:

determining a first weight value for the first weighted loss term;

determining a second weight value for the second weighted loss term;

determining a third weight value for the third weighted loss term; and

determining a fourth weight value for the fourth weighted loss term.

7. The method of claim 1 , further comprising:

receiving a new query, different from the stored queries; and

determining a predicted template from the stored templates for the new query by applying the model to the new query.

8. The method of claim 7 , further comprising initiating a response to the new query using the predicted template.

9. A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform a method comprising:

storing queries and templates;

selecting a subset of the stored queries based on one or more of:

a percentage of all stored queries; or

a time period in which each stored query was received;

iteratively performing the following steps until a performance metric for a model for matching a future query to one of the stored templates exceeds a performance threshold;

selecting a random subset of the stored templates;

generating a distribution based on the subset of stored queries and the random subset of templates by identifying each query of the subset of stored queries associated with each template of the random subset of stored templates;

selecting a subset of queries from the distribution;

generating a batch of training data based on the selected subset of queries from the distribution, wherein the batch of training data comprises one or more positively labeled training instances and one or more negatively labeled training instances; and

executing a multi-term loss function on the batch of training data to train the model for matching the future query to one of the stored templates,

wherein the multi-term loss function comprises:

a first weighted loss term corresponding to a similarity between a query of the subset of queries from the distribution and a template of the random subset of stored templates;

a second weighted loss term corresponding to a similarity between a first query of the subset of queries from the distribution and a second query of the subset of queries from the distribution;

a third weighted loss term corresponding to a similarity between a template of the random subset of stored templates and a second template of the random subset of stored templates; and

a fourth weighted loss term corresponding to a similarity between a template of the random subset of stored templates and a query of the subset of queries from the distribution, and

wherein the multi-term loss function is configured to reduce a number of training iterations until the performance metric for the model for matching the future query to one of the stored templates exceeds the performance threshold.

10. The non-transitory computer-readable medium of claim 9 , wherein executing the multi-term loss function comprises:

for each respective query in the random subset of queries in the batch of training data:

executing the multi-term loss function for each combination of the respective query and each template of the random subset of stored templates to yield a query/template value; and

aggregating the query/template value for the respective query in the first weighted loss term.

11. The non-transitory computer-readable medium of claim 10 , wherein executing the multi-term loss function further comprises:

for each respective query in the random subset of queries in the batch of training data:

executing the multi-term loss function for each combination of the respective query and another query in the random subset of queries to yield a query/query value; and

aggregating the query/query value for the respective query in the second weighted loss term.

12. The non-transitory computer-readable medium of claim 10 , wherein executing the multi-term loss function further comprises:

for each respective template in the random subset of stored templates:

executing the multi-term loss function for each combination of the respective template and another template in the random subset of stored templates to yield a template/template value; and

aggregating the template/template value for the respective template in the third weighted loss term.

13. The non-transitory computer-readable medium of claim 10 , wherein executing the multi-term loss function further comprises:

for each respective template in the random subset of stored templates:

executing the multi-term loss function for each combination of the respective template and a query in the random subset of queries to yield a template/query value; and

aggregating the template/query value for the respective template in the fourth weighted loss term.

14. The non-transitory computer-readable medium of claim 10 , further comprising:

determining a first weight value for the first weighted loss term,

determining a second weight value for the second weighted loss term;

determining a third weight value for the third weighted loss term; and

determining a fourth weight value for the fourth weighted loss term.

15. The non-transitory computer-readable medium of claim 9 , further comprising:

receiving a new query, different from the stored queries; and

determining a predicted template from the stored templates for the new query by applying the model to the new query.

16. The non-transitory computer-readable medium of claim 15 , further comprising initiating a response to the new query using the predicted template.

17. A system, comprising:

a repository of stored queries;

a repository of stored templates;

one or more processors; and

memory storing instructions that, when executed by the one or more processors, cause the system to:

select a subset of the stored queries based on one or more of:

a percentage of all stored queries; or

a time period in which each stored query was received;

iteratively performing the following steps until a performance metric for a model for matching a future query to one of the stored templates exceeds a performance threshold:

select a random subset of the stored templates;

generate a distribution based on the subset of stored queries and the random subset of templates by identifying each query of the subset of stored queries associated with each template of the random subset of stored templates;

select a subset of queries from the distribution;

generate a batch of training data based on the selected subset of queries from the distribution, wherein the batch of training data comprises one or more positively labeled training instances and one or more negatively labeled training instances; and

execute a multi-term loss function on batch of training data to train a model for matching a future query to one of the stored templates,

wherein the multi-term loss function comprises:

a first weighted loss term corresponding to a similarity between a query of the subset of queries from the distribution and a template of the random subset of stored templates;

a second weighted loss term corresponding to a similarity between a first query of the subset of queries from the distribution and a second query of the subset of queries from the distribution;

a third weighted loss term corresponding to a similarity between a template of the random subset of stored templates and a second template of the random subset of stored templates; and

a fourth weighted loss term corresponding to a similarity between a template of the random subset of stored templates and a query of the subset of queries from the distribution, and

wherein the multi-term loss function is configured to reduce a number of training iterations until the performance metric for the model for matching the future query to one of the stored templates exceeds the performance threshold.

18. The system of claim 17 , wherein in order to execute the multi-term loss function, the memory further stores instructions that, when executed by the one or more processors, cause the system to:

for each respective query in the random subset of queries in the batch of training data:

execute the multi-term loss function for each combination of the respective query and each template of the random subset of stored templates to yield a query/template value; and

aggregate the query/template value for the respective query in the first weighted loss term.

19. The system of claim 18 , wherein in order to execute the multi-term loss function, the memory further stores instructions that, when executed by the one or more processors, cause the system to:

for each respective query in the random subset of queries in the batch of training data:

execute the multi-term loss function for each combination of the respective query and another query in the random subset of queries to yield a query/query value; and

aggregate the query/query value for the respective query in the second weighted loss term.

20. The system of claim 18 , wherein in order to execute the multi-term loss function, the memory further stores instructions that, when executed by the one or more processors, cause the system to:

for each respective template in the random subset of stored templates:

execute the multi-term loss function for each combination of the respective template and another template in the random subset of stored templates to yield a template/template value; and

aggregate the template/template value for the respective template in the third weighted loss term.

21. The system of claim 18 , wherein in order to execute the multi-term loss function, the one or more processors are further configured to cause the system to:

for each respective template in the random subset of stored templates:

execute the multi-term loss function for each combination of the respective template and a query in the random subset of queries to yield a template/query value; and

aggregate the template/query value for the respective template in the fourth weighted loss term.

22. The system of claim 18 , wherein the memory further stores instructions that, when executed by the one or more processors, cause the system to:

determine a first weight value for the first weighted loss term;

determine a second weight value for the second weighted loss term;

determine a third weight value for the third weighted loss term; and

determine a fourth weight value for the fourth weighted loss term.

23. The system of claim 17 , wherein the memory further stores instructions that, when executed by the one or more processors, cause the system to:

receive a new query, different from the stored queries; and

determine a predicted template from the stored templates for the new query by applying the model to the new query.

24. The system of claim 23 , wherein the memory further stores instructions that, when executed by the one or more processors, cause the system to initiate a response to the new query using the predicted template.

25. The system of claim 17 , wherein the memory further stores instructions that, when executed by the one or more processors, cause the system to:

test a performance of the model; and

when the performance of the model exceeds a threshold, replace a production model with the model.

Assignments (1)

Continuity (2)

References Cited (9)