IP Library Granted Patent US 12,380,460
Granted Patent B2
US 12,380,460 · App. 18/629,399 · Granted Aug 5, 2025

Digital survey creation by providing optimized suggested content

Inventors: John Thimsen (Seattle, WA); Jeffrey Wade Starr (Orem, UT)
Assignee: Qualtrics, LLC
G06Q30/0203G06F16/9024
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,380,460
App. No.
18/629,399
Granted
Aug 5, 2025
Kind
B2
Abstract

The present disclosure relates to a question recommendation system that intelligently optimizes a survey being created by a user by providing customized suggestions. For example, in one or more embodiments, the question recommendation system provides a suggested question based on questions previous added by a user while creating a survey. In particular, the question recommendation system provides various recommendations to the user to further optimize a survey being created. For instance, the question recommendation system provides recommendations with respect to improving question ordering, question phrasing, and question type as well as recommends removing potentially inefficient questions.

Claims (86)

1. A computer-implemented method comprising:

generating a survey graph comprising:

a first node corresponding to a first encoded survey prompt representing a first survey question encoded into a vector space, the first encoded survey prompt comprising a first semantic label; and

a plurality of additional nodes corresponding to additional encoded survey prompts representing additional survey questions encoded into the vector space;

receiving, from an administrator client device, a user-generated survey prompt for a new digital survey;

generating, from the user-generated survey prompt, a second encoded survey prompt comprising a second semantic label;

determining that the second semantic label matches the first semantic label;

based on determining that the second semantic label matches the first semantic label, determining to traverse the survey graph to identify a suggested survey prompt corresponding to an additional node of the plurality of additional nodes within the survey graph;

providing, to the administrator client device, an option to use the suggested survey prompt;

based on receiving an indication of an interaction with the option to use the suggested survey prompt, generating the new digital survey comprising the user-generated survey prompt and the suggested survey prompt; and

providing, to a respondent client device, the new digital survey comprising the user-generated survey prompt and the suggested survey prompt.

2. The computer-implemented method of claim 1 , wherein identifying the suggested survey prompt further corresponding to the additional node of the plurality of additional nodes further comprises:

identifying survey prompt response metrics associated with nodes of the plurality of additional nodes; and

selecting the suggested survey prompt based on a survey prompt response metric associated with the additional node.

3. The computer-implemented method of claim 1 , further comprising:

determining that adding the suggested survey prompt to the new digital survey will result in a survey pattern that matches completed survey patterns; and

providing the option to use the suggested survey prompt based on determining that adding the suggested survey prompt to the new digital survey will result in a survey pattern that matches completed survey patterns.

4. The computer-implemented method of claim 1 , further comprising:

identifying an additional survey graph linked to the survey graph through an adjoining node positioned in a shared semantic space with the first node, wherein the adjoining node is encoded with an additional semantic label that matches the first semantic label and the second semantic label; and

based on identifying the additional survey graph linked to the survey graph, determining to traverse the additional survey graph to identify an additional suggested survey prompt corresponding to the adjoining node.

5. The computer-implemented method of claim 4 , further comprising:

providing, to the administrator client device, an option to use the additional suggested survey prompt; and

based on receiving an indication of a user interaction with the option to use the additional suggested survey prompt, generating a digital survey comprising the user-generated survey prompt, the suggested survey prompt, and the additional suggested survey prompt.

6. The computer-implemented method of claim 1 , further comprising:

based on the first node satisfying a survey prompt response metric, extracting survey prompt wording from the first node; and

providing the survey prompt wording from the first node to the administrator client device as the suggested survey prompt with the option to use the suggested survey prompt.

7. The computer-implemented method of claim 1 , further comprising:

based on receiving an indication of an interaction with the option to use the suggested survey prompt, determining to traverse the survey graph to identify an additional suggested survey prompt corresponding to a further node of the plurality of additional nodes within the survey graph; and

providing, to the administrator client device, an additional option to use the additional suggested survey prompt.

8. A non-transitory computer-readable medium storing instructions that, when executed by at least one processor, cause a computer system to:

generate a survey graph comprising:

a first node corresponding to a first encoded survey prompt representing a first survey question encoded into a vector space, the first encoded survey prompt comprising a first semantic label; and

a plurality of additional nodes corresponding to additional encoded survey prompts representing additional survey questions encoded into the vector space;

receive, from an administrator client device, a user-generated survey prompt for a new digital survey;

generate, from the user-generated survey prompt, a second encoded survey prompt comprising a second semantic label;

determine that the second semantic label matches the first semantic label;

based on determining that the second semantic label matches the first semantic label, determine to traverse the survey graph to identify a suggested survey prompt corresponding to an additional node of the plurality of additional nodes within the survey graph;

provide, to the administrator client device, an option to use the suggested survey prompt;

based on receiving an indication of an interaction with the option to use the suggested survey prompt, generate the new digital survey comprising the user-generated survey prompt and the suggested survey prompt; and

provide, to a respondent client device, the new digital survey comprising the user-generated survey prompt and the suggested survey prompt.

9. The non-transitory computer-readable medium of claim 8 , further comprising instructions that, when executed by at least one processor, cause a computer system to identify the suggested survey prompt further corresponding to the additional node of the plurality of additional nodes by:

identifying survey prompt response metrics associated with nodes of the plurality of additional nodes; and

selecting the suggested survey prompt based on a survey prompt response metric associated with the additional node.

10. The non-transitory computer-readable medium of claim 8 , further comprising instructions that, when executed by at least one processor, cause a computer system to:

determine that adding the suggested survey prompt to the new digital survey will result in a survey pattern that matches completed survey patterns; and

provide the option to use the suggested survey prompt based on determining that adding the suggested survey prompt to the new digital survey will result in a survey pattern that matches completed survey patterns.

11. The non-transitory computer-readable medium of claim 8 , further comprising instructions that, when executed by at least one processor, cause a computer system to:

identify an additional survey graph linked to the survey graph through an adjoining node positioned in a shared semantic space as the first node, wherein the adjoining node is encoded with an additional semantic label that matches the first semantic label and the second semantic label; and

based on identifying the additional survey graph linked to the survey graph, determine to traverse the additional survey graph to identify an additional suggested survey prompt corresponding to the adjoining node.

12. The non-transitory computer-readable medium of claim 11 , further comprising instructions that, when executed by at least one processor, cause a computer system to:

provide, to the administrator client device, an option to use the additional suggested survey prompt; and

based on receiving an indication of a user interaction with the option to use the additional suggested survey prompt, generate a digital survey comprising the user-generated survey prompt, the suggested survey prompt, and the additional suggested survey prompt.

13. The non-transitory computer-readable medium of claim 8 , further comprising instructions that, when executed by at least one processor, cause a computer system to:

based on the first node satisfying a survey prompt response metric, extract survey prompt wording from the first node; and

provide the survey prompt wording from the first node to the administrator client device as the suggested survey prompt with the option to use the suggested survey prompt.

14. The non-transitory computer-readable medium of claim 8 , further comprising instructions that, when executed by at least one processor, cause a computer system to:

based on receiving an indication of an interaction with the option to use the suggested survey prompt, determine to traverse the survey graph to identify an additional suggested survey prompt corresponding to a further node of the plurality of additional nodes within the survey graph; and

provide, to the administrator client device, an additional option to use the additional suggested survey prompt.

15. A system comprising:

at least one processor; and

at least one non-transitory computer-readable storage medium storing instructions that, when executed by the at least one processor, cause the system to:

generate a survey graph comprising:

a first node corresponding to a first encoded survey prompt representing a first survey question encoded into a vector space, the first survey question comprising a first semantic label; and

a plurality of additional nodes corresponding to additional encoded survey prompts representing additional survey questions encoded into the vector space;

receiving, from an administrator client device, a user-generated survey prompt for a new digital survey;

generating, from the user-generated survey prompt, a second encoded survey prompt comprising a second semantic label;

determining that the second semantic label matches the first semantic label;

based on determining that the second semantic label matches the first semantic label, determining to traverse the survey graph to identify a suggested survey prompt corresponding to an additional node of the plurality of additional nodes within the survey graph;

providing, to the administrator client device, an option to use the suggested survey prompt;

based on receiving an indication of an interaction with the option to use the suggested survey prompt, generate the new digital survey comprising the user-generated survey prompt and the suggested survey prompt; and

provide, to a respondent client device, the new digital survey comprising the user-generated survey prompt and the suggested survey prompt.

16. The system of claim 15 , further comprising instructions that, when executed by the at least one processor, cause the system to identify the suggested survey prompt further corresponding to the additional node of the plurality of additional nodes by:

identifying survey prompt response metrics associated with nodes of the plurality of additional nodes; and

selecting the suggested survey prompt based on a survey prompt response metric associated with the additional node.

17. The system of claim 15 , further comprising instructions that, when executed by the at least one processor, cause the system to:

determine that adding the suggested survey prompt to the new digital survey will result in a survey pattern that matches completed survey patterns; and

provide the option to use the suggested survey prompt based on determining that adding the suggested survey prompt to the new digital survey will result in a survey pattern that matches completed survey patterns.

18. The system of claim 15 , further comprising instructions that, when executed by the at least one processor, cause the system to:

identify an additional survey graph linked to the survey graph through an adjoining node positioned in a shared semantic space as the first node, wherein the adjoining node is encoded with an additional semantic label that matches the first semantic label and the second semantic label; and

based on identifying the additional survey graph linked to the survey graph, determine to traverse the additional survey graph to identify an additional suggested survey prompt corresponding to the adjoining node.

19. The system of claim 18 , further comprising instructions that, when executed by the at least one processor, cause the system to:

provide, to the administrator client device, an option to use the additional suggested survey prompt; and

based on receiving an indication of a user interaction with the option to use the additional suggested survey prompt, generate a digital survey comprising the user-generated survey prompt, the suggested survey prompt, and the additional suggested survey prompt.

20. The system of claim 15 , further comprising instructions that, when executed by the at least one processor, cause the system to:

based on the first node satisfying a survey prompt response metric, extract survey prompt wording from the first node; and

provide the survey prompt wording from the first node to the administrator client device as the suggested survey prompt with the option to use the suggested survey prompt.

Assignments (2)
SECURITY INTEREST Recorded May 18, 2026
From: QUALTRICS, LLC; PRESS GANEY ASSOCIATES LLC; CLARABRIDGE, INC.; DELIGHTED, LLC; RIOSOFT HOLDINGS, INC.; INMOMENT, INC.; LEXALYTICS, INC.; INMOMENT RESEARCH, LLC; ALLEGIANCE SOFTWARE, INC.
To: JPMORGAN CHASE BANK, N.A., AS COLLATERAL AGENT
Reel/Frame 075583/0001 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 8, 2024
From: THIMSEN, JOHN; STARR, JEFFREY WADE
To: QUALTRICS, LLC
Reel/Frame 068837/0958 →
Continuity (3)
Continuation 17819854 · Aug 15, 2022
Continuation 16256778 · Jan 24, 2019
Related Publication 20240403905A1 · Dec 5, 2024
References Cited (32)
US 7475339B2 · Holloway · 2009 [cited by examiner]
US 8429179B1 · Mirhaji · 2013 [cited by examiner]
US 8433715B1 · Mirhaji · 2013 [cited by examiner]
US 8676937B2 · Rapaport · 2014 [cited by examiner]
US 20030033193A1 · Holloway · 2003 [cited by examiner]
US 20040049416A1 · Alison, III · 2004 [cited by examiner]
US 20100004975A1 · White · 2010 [cited by examiner]
US 20120158633A1 · Eder · 2012 [cited by examiner]
US 20120290950A1 · Rapaport · 2012 [cited by examiner]
US 20130111323A1 · Taghaddos · 2013 [cited by examiner]
US 20130185657A1 · Gunawardena · 2013 [cited by examiner]
US 20130246049A1 · Mirhaji · 2013 [cited by examiner]
US 20130275448A1 · Mirhaji · 2013 [cited by examiner]
US 20140059089A1 · Bryden · 2014 [cited by examiner]
US 20150026260A1 · Worthley · 2015 [cited by examiner]
US 20170046329A1 · Mirhaji · 2017 [cited by examiner]
US 20170123643A1 · Berns · 2017 [cited by examiner]
US 20170124174A1 · Starr · 2017 [cited by examiner]
US 20180300794A1 · Viederman · 2018 [cited by examiner]
US 20190065017A1 · Keller · 2019 [cited by examiner]
US 20200042530A1 · Ross · 2020 [cited by examiner]
Minas, Mark. “Creating semantic representations of diagrams.” International Workshop on Applications of Graph Transformations with Industrial Relevance. Springer, Berlin, Heidelberg, 1999. (Year: 1999) (Year: 1999). [cited by examiner]
Leskovec, Jure, Marko Grobelnik, and Natasa Milic-Frayling. “Learning sub-structures of document semantic graphs for document summarization.” LinkKDD Workshop. vol. 133. 2004. (Year: 2004) (Year: 2004). [cited by examiner]
Shen, Dan, and Mirella Lapata. “Using semantic roles to improve question answering.” Proceedings of the 2007 joint conference on empirical methods in natural language processing and computational natural language learni… [cited by examiner]
Yih, Scott Wen-tau, et al. “Semantic parsing via staged query graph generation: Question answering with knowledge base.” (2015). (Year: 2015). [cited by examiner]
A. Singh, A. Taneja and G. Mangalaraj, “Creating online surveys: some wisdom from the trenches tutorial,” in IEEE Transactions on Professional Communication, vol. 52, No. 2, pp. 197-212, Jun. 2009, doi: 10.1109/TPC.2009… [cited by examiner]
U.S. Appl. No. 16/256,778, Sep. 25, 2020, Office Action. [cited by applicant]
U.S. Appl. No. 16/256,778, May 17, 2021, Office Action. [cited by applicant]
U.S. Appl. No. 16/256,778, Nov. 24, 2021, Office Action. [cited by applicant]
U.S. Appl. No. 16/256,778, Apr. 12, 2022, Notice of Allowance. [cited by applicant]
U.S. Appl. No. 17/819,854, May 15, 2023, Office Action. [cited by applicant]
U.S. Appl. No. 17/819,854, Nov. 13, 2023, Notice of Allowance. [cited by applicant]