IP Library Granted Patent US 10,963,639
Granted Patent B2
US 10,963,639 · App. 16/297,137 · Granted Mar 30, 2021

Systems and methods for identifying sentiment in text strings

Inventors: Gregor Stewart (San Mateo, CA); Tzu-Ting Kuo (Mountain View, CA); Andrew Yeager (Mountain View, CA)
Assignee: Medallia, Inc.
G06F40/253G06F16/904G06F16/906G06F40/216G06N20/00
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,963,639
App. No.
16/297,137
Granted
Mar 30, 2021
Kind
B2
Abstract

Embodiments discussed herein refer to systems and methods for identifying relevantly similar sentiment in text strings.

Claims (49)

1. A computer-implemented method comprising:

receiving a text input;

evaluating the text input with a first model to determine an initial sentiment and confidence thereof;

if the confidence exceeds, or is equal to, a threshold, using the initial sentiment;

if the confidence is below the threshold, accessing a list including at least one secondary sentiment and evaluating the text input, in combination with each secondary sentiment, with a relevantly similar analysis model to generate a relevantly similar confidence (RSC) score corresponding to each secondary sentiment included in the list, wherein an evaluation of each generated RSC score determines whether to use the initial sentiment or a secondary sentiment as a resolved sentiment; and

displaying the resolved sentiment associated with the text string.

2. The method of claim 1 , wherein the list of secondary sentiments is a client specific posting list comprising a plurality of correction inputs, wherein each RSC score is based on a determination of how relevant and similar the text input is to each of the plurality of correction inputs.

3. The method of claim 2 , wherein each of the plurality of correction inputs has been previously provided by a user.

4. The method of claim 1 , wherein when the RSC score is null or less than a threshold, the initial sentiment is set as the resolved sentiment.

5. The method of claim 1 , wherein when only one RSC score is equal to or greater than a threshold, the secondary sentiment corresponding to that particular RSC score is selected as the resolved sentiment.

6. The method of claim 1 , wherein when multiple RSC scores are equal to or greater than a threshold, the method further comprises using the secondary sentiments corresponding to the RSC scores that are equal to or greater than to threshold as input factors to select a tertiary sentiment as the resolved sentiment.

7. The method of claim 6 , further comprising:

receiving a user input as the tertiary sentiment.

8. The method of claim 1 , wherein the list is associated with a particular client, and wherein the relevantly similar analysis model only accesses the list associated with that particular client when generating at least one RSC score for the text input for that particular client.

9. The method of claim 1 , wherein the initial sentiment and RSC scores are generated in real-time.

10. The method of claim 1 , wherein the list is populated by client administered entries.

11. The method of claim 1 , wherein the input text is a limited text input.

12. The method of claim 1 , wherein the relevantly similar analysis model is trained with a dataset used by the first model, and wherein the relevantly similar analysis model uses the dataset and the list to evaluate the input text to generate the RSC score.

13. A computer-implemented method, comprising:

displaying a sentiment results page comprising a subset of a plurality of user responses, wherein each displayed user response displays a text string that formed a basis for a sentiment associated with the user response;

displaying a sentiment correction overlay in response to receiving a user selection of one of the text strings, wherein the sentiment correction overlay enables a user to manually select a corrected sentiment for the user selected text string;

receiving a user selection to select a corrected sentiment for the user selected text string; and

temporarily incorporating the corrected sentiment for the user selected text string into a client specified corrected sentiment corrections database that is used by a relevantly similar analysis model to determine whether a text input is relevantly similar to any text strings contained in the database.

14. The method of claim 13 , wherein the user selection to manually select a corrected sentiment for the user selected text string is a first user corrected sentiment, the method further comprising:

displaying, in the sentiment correction overlay, a second text string requesting user selection of a corrected sentiment; and

receiving a second user corrected sentiment for the second text string.

15. The method of claim 14 , further comprising temporarily incorporating the second user corrected sentiment into the client specified corrected sentiment corrections database.

16. The method of claim 14 , further comprising using the first user corrected sentiment and the second user corrected sentiment to train the relevantly similar analysis model.

17. The method of claim 16 , further comprising displaying sentiment training results overlay, wherein the sentiment training results overlay shows before and after sentiment classifications assigned to one or more text strings, wherein the before sentiment classification is based on a result generated by the relevantly similar analysis model prior to being trained with at least the first user corrected sentiment and the second user corrected sentiment, and wherein the after sentiment classification is based on a result generated by the relevantly similar analysis model after being trained with at least the first user corrected sentiment and the second user corrected sentiment.

18. The method of claim 17 , further comprising:

providing an option for the user to permanently incorporate the at least the first user corrected sentiment and the second user corrected sentiment into the client specified corrected sentiment corrections database;

providing an option for the user to remove the at least the first user corrected sentiment and the second user corrected sentiment from the client specified corrected sentiment corrections database; and

providing an option to view additional examples of before and after sentiment classifications.

19. The method of claim 18 , further comprising:

receiving selection of the option for the user to permanently incorporate the at least the first user corrected sentiment and the second user corrected sentiment into the client specified corrected sentiment corrections database; and

permanently incorporating the at least the first user corrected sentiment and the second user corrected sentiment into the client specified corrected sentiment corrections database.

20. The method of claim 18 , further comprising:

receiving selection of the option for the user to remove the at least the first user corrected sentiment and the second user corrected sentiment from the client specified corrected sentiment corrections database; and

removing the at least the first user corrected sentiment and the second user corrected sentiment from the client specified corrected sentiment corrections database.

21. The method of claim 18 , further comprising:

receiving selection an option to view additional examples of before and after sentiment classifications; and

displaying additional examples of before and after sentiment classifications.

22. A computer-implemented method, comprising:

training a relevantly similar analysis model that is operative to analyze a text input to determine whether the text input is relevantly similar to other text inputs, the training comprising:

receiving a text seed;

retrieving a plurality of text strings determined to be similar to the text seed from a database;

assessing each of the plurality of text strings to identify which of the plurality of text strings are relevantly similar to the text seed; and

using the text strings identified to be relevantly similar as training inputs for the relevantly similar analysis model; and

using the relevantly similar analysis model to produce second order sentiment results for text inputs when first order sentiment results for the text inputs do not meet confidence criteria.

Assignments (6)
RELEASE OF SECURITY INTEREST Recorded Apr 13, 2022
From: WELLS FARGO BANK NA
To: MEDALLION, INC
Reel/Frame 059581/0865 →
CORRECTIVE ASSIGNMENT TO CORRECT THE THE LIST OF PATENT PROPERTY NUMBER TO INCLUDE TWO PATENTS THAT WERE MISSING FROM THE ORIGINAL FILING PREVIOUSLY RECORDED AT REEL: 057968 FRAME: 0430. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT. Recorded Nov 1, 2021
From: WELLS FARGO BANK, NATIONAL ASSOCIATION, AS ADMINISTRATIVE AGENT
To: MEDALLIA, INC.
Reel/Frame 057982/0092 →
SECURITY INTEREST Recorded Oct 29, 2021
From: MEDALLIA, INC.
To: WILMINGTON TRUST, NATIONAL ASSOCIATION, AS COLLATERAL AGENT
Reel/Frame 057964/0016 →
RELEASE OF SECURITY INTEREST Recorded Oct 29, 2021
From: WELLS FARGO BANK, NATIONAL ASSOCIATION, AS ADMINISTRATIVE AGENT
To: MEDALLIA, INC.
Reel/Frame 057968/0430 →
SECURITY INTEREST Recorded Jul 28, 2021
From: MEDALLIA, INC.
To: WELLS FARGO BANK, NATIONAL ASSOCIATION
Reel/Frame 057011/0012 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 8, 2019
From: STEWART, GREGOR; KUO, TZU-TING; YEAGER, ANDREW
To: MEDALLIA, INC.
Reel/Frame 048550/0628 →
Continuity (1)
Related Publication 20200285696A1 · Sep 10, 2020