IP Library Granted Patent US 11,809,983
Granted Patent B2
US 11,809,983 · App. 16/117,918 · Granted Nov 7, 2023

Machine-learning-based digital survey creation and management

Inventors: Ryan Long (Lehi, UT); Johnny Russ (Provo, UT)
Assignee: Qualtrics, LLC
G06N3/08G06F40/247G06F40/279G06F40/30G06N3/044G06F3/0482
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,809,983
App. No.
16/117,918
Granted
Nov 7, 2023
Kind
B2
Abstract

This disclosure relates to methods, non-transitory computer readable media, and systems that use machine-learning techniques to facilitate the creation, timing of distribution, or follow-up actions for digital surveys. In one such implementation, the disclosed methods, non-transitory computer readable media, and systems use a survey-creation-machine learner to generate suggested survey questions for an administrator designing a digital survey. Additionally, in some implementations, the disclosed methods, non-transitory computer readable media, and systems use specialized machine learners to suggest timeframes in which to send digital surveys or to suggest action items to follow up on responses to the survey questions.

Claims (90)

1. A method comprising:

receiving, from an administrator device and via a survey creation graphical user interface, user input to create an initial survey question;

inputting the initial survey question into a survey-creation-machine learner to:

apply natural-language processing to identify textual features of the initial survey question comprising an intent of the initial survey question;

select a representative survey question for the initial survey question based on the identified textual features comprising the intent of the initial survey question; and

determine, from a correlation database and based on the representative survey question, a first suggested survey question corresponding to a first survey category and a second suggested survey question corresponding to a second survey category;

receiving, from the survey-creation-machine learner, the first suggested survey question and the second suggested survey question; and

providing, for display within the survey creation graphical user interface of the administrator device, a first selectable option for the first suggested survey question and a second selectable option for the second suggested survey question.

2. The method of claim 1 , wherein utilizing the survey-creation-machine learner to identify the textual features of the initial survey question comprises:

extracting terms from the initial survey question; or

determining an intent for the initial survey question.

3. The method of claim 2 , wherein utilizing the survey-creation-machine learner to select the representative survey question for the initial survey question comprises:

identifying the extracted terms within the representative survey question from among candidate-survey questions;

identifying synonymous terms within the representative survey question corresponding to the extracted terms from the initial survey question; or

determining that a reciprocal intent of the representative survey question corresponds to the intent of the initial survey question.

4. The method of claim 1 , further comprising:

inputting training survey responses corresponding to a demographic group to a suggested-action-machine learner;

utilizing the suggested-action-machine learner to identify suggested-training-action items for the demographic group;

training the suggested-action-machine learner to select suggested-action items for the demographic group by comparing the suggested-training-action items to ground-truth-action items corresponding to the demographic group; and

utilizing the suggested-action-machine learner to determine a suggested-action item based on a survey response.

5. The method of claim 1 , further comprising:

receiving an indication from the administrator device of a user selection of the first selectable option for the first suggested survey question;

identifying supplementary suggested survey questions corresponding to the first survey category; and

providing the supplementary suggested survey questions for display within an updated survey creation graphical user interface of the administrator device.

6. The method of claim 1 , further comprising, before receiving the user input to create the initial survey question:

inputting a training survey question into the survey-creation-machine learner;

utilizing the survey-creation-machine learner to:

identify training textual features of the training survey question; and

select a candidate-representative-survey question for the training survey question based on the identified training textual features; and

training the survey-creation-machine learner to select representative survey questions for initial survey questions by comparing the candidate-representative-survey question to a ground-truth-representative-survey question.

7. The method of claim 6 , wherein training the survey-creation-machine learner to select the representative survey questions for the initial survey questions comprises:

determining a loss from a loss function based on comparing the candidate-representative-survey question to the ground-truth-representative-survey question; and

adjusting machine-learning parameters of the survey-creation-machine learner based on an objective to decrease the loss in a subsequent iteration.

8. The method of claim 1 , wherein the survey-creation-machine learner comprises a recurrent neural network or a Naïve Bayes Support Vector Machine.

9. A system comprising:

at least one processor; and

at least one non-transitory computer readable storage medium storing instructions that, when executed by the at least one processor, cause the system to:

receive, from an administrator device and via a survey creation graphical user interface, user input to create an initial survey question;

utilize a survey-creation-machine learner to:

apply natural-language processing to identify textual features of the initial survey question comprising an intent of the initial survey question;

select a representative survey question for the initial survey question based on the identified textual features comprising the intent of the initial survey question; and

determine, from a correlation database and based on the representative survey question, a first suggested survey question corresponding to a first survey category and a second suggested survey question corresponding to a second survey category;

receive, from the survey-creation-machine learner, the first suggested survey question and the second suggested survey question; and

provide, for display within the survey creation graphical user interface of the administrator device, a first selectable option for the first suggested survey question and a second selectable option for the second suggested survey question.

10. The system of claim 9 , further comprising instructions that, when executed by the at least one processor, cause the system to:

receive, from the administrator device, demographic indicators for target survey recipients, a recipient location for the target survey recipients, and a time range in which to send the target survey recipients the initial survey question and the first suggested survey question;

utilize a survey-timeframe-machine learner to determine from within the time range a suggested timeframe in which to send the initial survey question and the first suggested survey question to the target survey recipients, the suggested timeframe corresponding to a highest predicted response rate for the target survey recipients; and

provide the suggested timeframe for display within a graphical user interface of the administrator device.

11. The system of claim 10 , further comprising instructions that, when executed by the at least one processor, cause the system to utilize the survey-timeframe-machine learner to determine the suggested timeframe by:

determining a first response rate for a first survey cluster comprising responses from prior survey recipients who received digital surveys within a first timeframe and who correspond to the demographic indicators and the recipient location for the target survey recipients;

determining a second response rate for a second survey cluster comprising responses from prior survey recipients who received digital surveys within a second timeframe and who correspond to the demographic indicators and the recipient location for the target survey recipients;

applying a first machine-learning parameter to the first response rate to determine a first weighted response rate; and

applying a second machine-learning parameter to the second response rate to determine a second weighted response rate.

12. The system of claim 11 , further comprising instructions that, when executed by the at least one processor, cause the system to utilize the survey-timeframe-machine learner to determine the suggested timeframe by:

determining that the first weighted response rate exceeds the second weighted response rate; and

selecting a current corollary of the first timeframe as the suggested timeframe based on the first weighted response rate exceeding the second weighted response rate.

13. The system of claim 10 , further comprising instructions that, when executed by the at least one processor, cause the system to, before receiving the demographic indicators, the recipient location, and the time range:

input, into the survey-timeframe-machine learner, training demographic indicators for prior survey recipients, a training recipient location for the prior survey recipients, and a training time range in which the prior survey recipients received survey questions;

utilize the survey-timeframe-machine learner to determine from within the training time range a training timeframe for sending the survey questions; and

train the survey-timeframe-machine learner to determine suggested timeframes for sending suggested survey questions to survey recipients of identified demographic indicators within identified recipient locations by comparing the training timeframe to a ground-truth timeframe for sending the survey questions, the ground-truth timeframe corresponding to a highest recorded response rate for the prior survey recipients.

14. The system of claim 13 , further comprising instructions that, when executed by the at least one processor, cause the system to train the survey-timeframe-machine learner to determine the suggested timeframes for sending the suggested survey questions to the survey recipients by:

determining a loss from a loss function based on comparing the training timeframe to the ground-truth timeframe; and

adjusting machine-learning parameters of the survey-timeframe-machine learner based on an objective to decrease the loss in a subsequent iteration.

15. A non-transitory computer readable medium storing instructions thereon that, when executed by at least one processor, cause a computing system to:

receive, from an administrator device and via a survey creation graphical user interface, user input to create an initial survey question;

utilize a survey-creation-machine learner to:

apply natural-language processing to identify textual features of the initial survey question comprising an intent of the initial survey question;

select a representative survey question for the initial survey question based on the identified textual features comprising the intent of the initial survey question; and

determine, from a correlation database and based on the representative survey question, a first suggested survey question corresponding to a first survey category and a second suggested survey question corresponding to a second survey category;

receive, from the survey-creation-machine learner, the first suggested survey question and the second suggested survey question; and

provide for display within the survey creation graphical user interface of the administrator device, a first selectable option for the first suggested survey question and a second selectable option for the second suggested survey question.

16. The non-transitory computer readable medium of claim 15 , further comprising instructions that, when executed by the at least one processor, cause the computing system to:

provide the first suggested survey question to recipient devices associated with survey recipients;

receive a response to the first suggested survey question from a recipient device of the recipient devices; and

utilize a suggested-action-machine learner to determine a suggested action item based on the response.

17. The non-transitory computer readable medium of claim 16 , further comprising instructions that, when executed by the at least one processor, cause the computing system to utilize the suggested-action-machine learner to determine the suggested action item by:

determining a suggested follow-up survey question;

determining a suggested follow-up contact;

determining a suggested meeting with a survey recipient associated with the recipient device; or

determining a suggested strategy, tactic, or initiative for a manager or team of the survey recipient associated with the recipient device.

18. The non-transitory computer readable medium of claim 16 , further comprising instructions that, when executed by the at least one processor, cause the computing system to utilize the suggested-action-machine learner to determine the suggested action item by:

applying natural-language processing to identify terms within the response; and

determining the suggested action item based on the response and the terms.

19. The non-transitory computer readable medium of claim 16 , further comprising instructions that, when executed by the at least one processor, cause the computing system to, before receipt of the user input to create the initial survey question:

input, into the suggested-action-machine learner, a training response to a survey question from a training survey respondent;

utilize the suggested-action-machine learner to determine a suggested-training-action item based on the training response; and

train the suggested-action-machine learner to determine suggested action items for responses by determining a loss from a loss function based on comparing the suggested-training-action item to a ground-truth-action item for the training response and adjusting machine-learning parameters of the suggested-action-machine learner based on an object to decrease the loss in a subsequent iteration.

20. The non-transitory computer readable medium of claim 16 , further comprising instructions that, when executed by the at least one processor, cause the computing system to:

receive a demographic indicator for a survey recipient who provided the response to the first suggested survey question and an action-item-completion indicator indicating that the survey recipient completed or failed to complete the suggested action item; and

utilize an additional suggested-action-machine learner to determine a suggested-follow-up-action item based on the demographic indicator and the action-item-completion indicator.

Assignments (3)
SECURITY INTEREST Recorded May 18, 2026
From: QUALTRICS, LLC; PRESS GANEY ASSOCIATES LLC; CLARABRIDGE, INC.; DELIGHTED, LLC; RIOSOFT HOLDINGS, INC.; INMOMENT, INC.; LEXALYTICS, INC.; INMOMENT RESEARCH, LLC; ALLEGIANCE SOFTWARE, INC.
To: JPMORGAN CHASE BANK, N.A., AS COLLATERAL AGENT
Reel/Frame 075583/0001 →
SECURITY INTEREST Recorded Jun 29, 2023
From: QUALTRICS, LLC; CLARABRIDGE, INC.; NEW DEBDEN MERGER SUB II LLC
To: JPMORGAN CHASE BANK, N.A., AS COLLATERAL AGENT
Reel/Frame 064162/0976 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 30, 2018
From: LONG, RYAN; RUSS, JOHNNY
To: QUALTRICS, LLC
Reel/Frame 046759/0370 →
Continuity (1)
Related Publication 20200074294A1 · Mar 5, 2020
Cited By (2)
US 12,393,890 US 12,393,891