IP Library Granted Patent US 11,657,231
Granted Patent B2
US 11,657,231 · App. 16/915,758 · Granted May 23, 2023

Capturing rich response relationships with small-data neural networks

Inventor: John Hewitt (Philadelphia, PA)
Assignee: Qualtrics, LLC
G06F40/30G06F40/205G06F40/216G06F40/253G06F40/268G06F40/284G06N3/0445G06N3/08
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,657,231
App. No.
16/915,758
Granted
May 23, 2023
Kind
B2
Abstract

The present disclosure relates to a response analysis system that employs a small-data training dataset to train a neural network that accurately performs domain-agnostic opinion mining. For example, in one or more embodiments, the response analysis system trains a response classification neural network using part of speech information (e.g., syntactic information) to learn and apply response classification labels for opinion text responses. In particular, the response analysis system employs part of speech information patterns without regard to word patterns to determine whether words in a text response correspond to an opinion, the target of the opinion, or neither. In addition, the trained response classification neural network has a significantly reduced learned parameter space, which decreases processing, memory requirements, and overall complexity.

Claims (43)

1. A system comprising:

at least one processor; and

at least one non-transitory computer-readable storage medium storing instructions that, when executed by the at least one processor, cause the system to:

analyze a plurality of input sentences to determine a classification label for each word within each input sentence of the plurality of input sentences utilizing a word-agnostic classifier;

identify input sentences from the plurality of input sentences associated with a first classification label; and

provide, for display within a graphical user interface, the input sentences based on the input sentences being associated with the first classification label.

2. The system of claim 1 , wherein the plurality of input sentences comprises free-form text survey responses.

3. The system of claim 1 , wherein the classification label comprises one of a target classification label, an opinion classification label, or a neither classification label.

4. The system of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the system to utilize a word-agnostic classification neural network as the word-agnostic classifier to determine the classification label for each word of the plurality of input sentences.

5. The system of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the system to identify the input sentences from the plurality of input sentences by identifying input sentences that comprise at least one word corresponding to the first classification label.

6. The system of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the system to provide, for display within the graphical user interface, a set of words determined to correspond with the first classification label from an input sentence of the input sentences.

7. The system of claim 6 , further comprising instructions that, when executed by the at least one processor, cause the system to:

identify, from the input sentence of the input sentences, a word that is determined to correspond with a second classification label and associated with the set of words determined to correspond with the first classification label; and

provide, for display within the graphical user interface, the word determined to correspond with the second classification label in relation to the set of words determined to correspond with the first classification label.

8. The system of claim 7 , wherein the first classification label comprises an opinion classification label and the second classification label comprises a target classification label and further comprising instructions that, when executed by the at least one processor, cause the system to:

provide, for display within the graphical user interface, a target portion, wherein the target portion comprises the word determined to correspond with the target classification label; and

provide, for display within the graphical user interface, an opinion portion, wherein the opinion portion comprises the set of words determined to correspond with the opinion classification label.

9. The system of claim 1 , wherein the first classification label comprises an opinion classification label and further comprising instructions that, when executed by the at least one processor, cause the system to:

analyze the input sentences associated with the opinion classification label to determine opinion statistics for the input sentences, wherein the opinion statistics comprise a positive opinion, negative opinion, or neutral opinion; and

provide, for display within the graphical user interface, the input sentences sorted based on the opinion statistics.

10. A non-transitory computer-readable medium storing instructions that, when executed by at least one processor, cause a computing device to:

analyze a plurality of input sentences to determine a classification label for each word within each input sentence of the plurality of input sentences utilizing a word-agnostic classifier;

identify input sentences from the plurality of input sentences associated with a first classification label; and

provide, for display within a graphical user interface, the input sentences based on the subset of input sentences being associated with the first classification label.

11. The non-transitory computer-readable medium of claim 10 , wherein the plurality of input sentences comprises free-form text survey responses.

12. The non-transitory computer-readable medium of claim 10 , wherein the classification label comprises one of a target classification label, an opinion classification label, or a neither classification label.

13. The non-transitory computer-readable medium of claim 10 , further comprising instructions that, when executed by the at least one processor, cause the computing device to provide, for display within the graphical user interface, a set of words determined to correspond with the first classification label from an input sentence of the input sentences.

14. The non-transitory computer-readable medium of claim 13 , further comprising instructions that, when executed by the at least one processor, cause the computing device to:

identify, from the input sentence of the input sentences, a word that is determined to correspond with a second classification label and associated with the set of words determined to correspond with the first classification label; and

provide, for display within the graphical user interface, the word determined to correspond with the second classification label in relation to the set of words determined to correspond with the first classification label.

15. The non-transitory computer-readable medium of claim 14 , wherein the first classification label comprises an opinion classification label and the second classification label comprises a target classification label.

16. A computer-implemented method comprising:

analyzing a plurality of input sentences to determine a classification label for each word within each input sentence of the plurality of input sentences utilizing a word-agnostic classifier;

identifying input sentences from the plurality of input sentences associated with a first classification label; and

providing, for display within a graphical user interface, the input sentences based on the subset of input sentences being associated with the first classification label.

17. The computer-implemented method of claim 16 , wherein the plurality of input sentences comprises free-form text survey responses.

18. The computer-implemented method of claim 16 , further comprising providing, for display within the graphical user interface, a set of words determined to correspond with the first classification label from an input sentence of the input sentences.

19. The computer-implemented method of claim 18 , further comprising:

identifying, from the input sentence of the input sentences, a word that is determined to correspond with a second classification label and associated with the set of words determined to correspond with the first classification label; and

providing, for display within the graphical user interface, the word determined to correspond with the second classification label in relation to the set of words determined to correspond with the first classification label.

20. The computer-implemented method of claim 19 , wherein the first classification label comprises an opinion classification label and the second classification label comprises a target classification label and further comprising:

providing, for display within the graphical user interface, a target portion, wherein the target portion comprises the word determined to correspond with the target classification label; and

providing, for display within the graphical user interface, an opinion portion, wherein the opinion portion comprises the set of words determined to correspond with the opinion classification label.

Assignments (3)
SECURITY INTEREST Recorded May 18, 2026
From: QUALTRICS, LLC; PRESS GANEY ASSOCIATES LLC; CLARABRIDGE, INC.; DELIGHTED, LLC; RIOSOFT HOLDINGS, INC.; INMOMENT, INC.; LEXALYTICS, INC.; INMOMENT RESEARCH, LLC; ALLEGIANCE SOFTWARE, INC.
To: JPMORGAN CHASE BANK, N.A., AS COLLATERAL AGENT
Reel/Frame 075583/0001 →
SECURITY INTEREST Recorded Jun 29, 2023
From: QUALTRICS, LLC; CLARABRIDGE, INC.; NEW DEBDEN MERGER SUB II LLC
To: JPMORGAN CHASE BANK, N.A., AS COLLATERAL AGENT
Reel/Frame 064162/0976 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 20, 2020
From: HEWITT, JOHN
To: QUALTRICS, LLC
Reel/Frame 053251/0726 →