Granted Patent B2

US 12,298,982 · App. 17/093,885 · Granted May 13, 2025

Inventors: Wenzhuo Yang (Singapore, SG); Jia Li (Mountain View, CA); Chenxi Li (Belmont, CA); Latrice Barnett (Oakland, CA); Markus Anderle (Moraga, CA); Simo Arajarvi (Dublin, IE); Harshavardhan Utharavalli (Hyderabad, IN); Caiming Xiong (Menlo Park, CA); Richard Socher (Menlo Park, CA); Chu Hong Hoi (Singapore, SG)

Assignee: Salesforce, Inc.

G06F16/2457G06N20/20

Abstract

Claims (89)

1. A system comprising:

a non-transitory memory; and

one or more processors coupled to the non-transitory memory and configured to read instructions from the non-transitory memory to cause the system to perform operations comprising:

receiving a user query for generating a recommendation for one or more items and one or more explanations associated with the one or more items;

for each item in a plurality of items:

obtaining input features comprising at least one user feature and at least one item feature;

determining, using one or more relevancy models in one or more first machine learning networks and the input features, a predicted score of the item;

determining, using a first diversity model in second machine learning networks, first diversity scores and a first explainability score for the item from at least a first portion of the input features;

determining, using a second diversity model in the second machine learning networks, second diversity scores and a second explainability score for the item from at least a second portion of the input features, the second portion of the input features different from the first portion of the input features, wherein the second machine learning networks are associated with a weight parameter indicative of the degree of diversity in recommendations;

determining an explanation narrative for the item using the first and second explainability scores; and

combining the predicted score, the first diversity scores of the first diversity model and the second diversity scores of the second diversity model into a combined score for the item;

ranking the plurality of items according to combined scores;

providing the recommendation that includes the one or more items from the plurality of items, wherein the one or more items correspond to top combined scores;

receiving a user input indicative of an updated value for the weight parameter;

generating an updated recommendation for the one or more items and the one or more explanations associated with the one or more items based on the updated value for the weight parameter; and

providing the updated recommendation that includes the one or more items from the plurality of items.

2. The system of claim 1 , wherein the operations further comprise:

determining a diversity loss associated with the first and second diversity scores;

determining an accuracy loss using the predicted score; and

determining a total loss using the diversity loss and the accuracy loss.

3. The system of claim 2 , wherein the operations further comprise:

training the one or more first machine learning networks and the second machine learning networks to minimize the total loss.

4. The system of claim 1 , wherein the operations further comprise determining the explanation narrative concurrently with the first and second explainability scores.

5. The system of claim 1 , wherein to determine the predicted score, the operations further comprise:

determining a user vector from the at least one user feature and an item feature in the at least one item feature, wherein the item feature is associated with the user feature;

determining an item vector from the at least one item feature; and

determining the predictive score from the user vector and the item vector.

6. The system of claim 1 , wherein the operations further comprise:

determining a popularity score in the first or second explainability score using an item identifier feature in the at least one item feature.

7. The system of claim 1 , wherein the operations further comprise:

determining a category score in the first or second explainability score using an item identifier feature in the at least one item feature and at least one category feature from the at least one user feature.

8. The system of claim 1 , wherein the operations further comprise:

determining a continuous score in the first or second explainability score using an item identifier feature in the at least one item feature and at least one continuous feature from the at least one user feature.

9. A method comprising:

receiving a user query for generating a recommendation for one or more items and one or more explanations associated with the one or more items;

obtaining a first feature comprising at least one user feature and a second feature comprising at least one item feature from a plurality of items;

determining, using one or more first machine learning networks, the first feature, and the second feature, predicted scores for the plurality of items, wherein the one or more first machine learning networks are trained for a recommendation task;

determining, using a first diversity model in second machine learning networks, a first portion of the first feature and a first portion of the second feature, first diversity scores for the plurality of items and explanation narratives;

determining, using a second diversity model in the second machine learning networks, a second portion of the first feature and a second portion of the second feature, second diversity scores for the plurality of items and the explanation narratives, wherein the second machine learning networks are associated with a weight parameter, and wherein varying the weight parameter during the determining the first diversity score and the second diversity scores changes the one or more items included in the recommendation; and

determining combined scores for the plurality of items from the predicted scores, the first diversity scores from the first diversity model and the second diversity scores from the second diversity model;

identifying the one or more items from the plurality of items as the items with highest scores from the combined scores; and

generating the recommendation that includes the one or more items and corresponding one or more explanation narratives.

10. The method of claim 9 , further comprising:

determining a diversity loss associated with the first and second diversity scores;

determining an accuracy loss using the predicted scores; and

determining a total loss using the diversity loss and the accuracy loss.

11. The method of claim 10 , further comprising:

training the one or more first machine learning networks and the second machine learning networks to minimize the total loss.

12. The method of claim 9 , wherein determining the predicted scores further comprises:

determining a user vector from the at least one user feature and an item feature in the at least one item feature, wherein the item feature is associated with the user feature;

determining an item vector from the at least one item feature; and

determining the predicted scores from the user vector and the item vector.

13. The method of claim 9 , further comprising:

determining, using the second machine learning networks, explainability scores for the plurality of items;

determining a popularity score in the explainability scores using an item identifier feature in the at least one item feature; and

determining the one or more explanation narratives based on the popularity score.

14. The method of claim 9 , further comprising:

determining, using the second machine learning networks, explainability scores for the plurality of items;

determining a category score in the explainability scores using an item identifier feature in the at least one item feature and at least one category feature from the at least one user feature; and

determining the one or more explanation narratives based on the category score.

15. The method of claim 11 , further comprising:

determining, using the second machine learning networks, explainability scores for the plurality of items;

determining a continuous score in the explainability scores using an item identifier feature in the at least one item feature and at least one continuous feature from the at least one user feature; and

determining the one or more explanation narratives based on the continuous score.

16. A non-transitory machine-readable medium having stored thereon machine-readable instructions executable to cause a system to perform operations comprising:

receiving a user query for generating a recommendation for one or more items and one or more explanations associated with the one or more items;

for each item in a plurality of items:

obtaining input features comprising at least one user feature and at least one item feature;

determining, using one or more first machine learning networks and the input features, a predicted score of the item;

determining, using a first diversity model in second machine learning networks, first diversity scores and a first explainability score for the item from at least a first portion of the input features;

determining, using a second diversity model in the second machine learning networks, second diversity scores and a second explainability score for the item from at least a second portion of the input features, wherein the second machine learning networks are associated with a weight parameter indicative of the degree of diversity in recommendations;

combining the predicted score, the first diversity scores of the first diversity model and second diversity scores of the second diversity model into a combined score for the item; and

ranking the plurality of items according to the combined scores;

determining explanation narratives for the ranked plurality of items using the first and second explainability scores;

providing the recommendation that includes the one or more items in the plurality of items that correspond to highest scores and one or more explanation narratives corresponding to the one or more items;

receiving a user input indicative of an updated value for the weight parameter;

generating an updated recommendation for the one or more items and the one or more explanations associated with the one or more items based on the updated value for the weight parameter; and

providing the updated recommendation that includes the one or more items from the plurality of items.

17. The non-transitory machine-readable medium of claim 16 , wherein the operations further comprise:

determining an accuracy loss of the one or more first machine learning networks from the predictive score of the item;

determining a diversity loss of the second machine learning networks from the first and second diversity scores; and

determining a total loss from the accuracy loss and the diversity loss.

18. The non-transitory machine-readable medium of claim 16 , wherein the first or second explainability score include a category score, a popular score, and a continuous score.

19. The non-transitory machine-readable medium of claim 16 , wherein to determine the predicted score, the operations further comprise:

determining a user vector from the at least one user feature and an item feature in the at least one item feature, wherein the item feature is associated with the user feature;

determining an item vector from the at least one item feature; and

determining the predictive score from the user vector and the item vector.

20. The non-transitory machine-readable medium of claim 17 , wherein the operations further comprise:

training the one or more first machine learning networks and the second machine learning networks to minimize the total loss.

Assignments (2)

Priority Claims (1)

Continuity (1)

References Cited (32)