Systems and methods for attribute characterization of usability testing participants
Systems and methods for attribute determination in a usability study are provided. The system includes the ability to collect screener questions and response pairs and determine the type of question. The question and response pairs may be processed for topic and entity extractions using machine learning (ML) models. From the collected topics and entities, a dictionary of attributes may be generated and eventually expanded/added to as new information regarding the participant becomes available. This attribute dictionary may take the form of a vector dictionary, in some particular embodiments. In some cases, the type of question being posed may dictate how the response is processed. These include a Boolean style question, a quantitative question, a single response question and a multi-response type question.
1 . A method for identifying attributes in a plurality of usability study participants comprising:
monitoring question and response pairs corresponding to participants in the study by inserting a virtual tracking code to a web site targeted for the study at a local machine to each participant, wherein the tracking code collects data including at least one of number of clicks, keystrokes, keywords, scrolls, and time on tasks, and providing the collected data to a server for additional analysis;
receiving question and response pairs for participants in the study;
identifying a topic of a question portion of a question and response pair using at least one topic machine learning (ML) model;
receiving a plurality of name entity recognition (NER) ML models;
selecting a preferred NER ML model from the plurality of NER ML models based upon accuracy of each NER ML model to the topic;
identifying an entity attribute of at least one of a response and the question portion of the question and response pair using the preferred response NER ML model;
decoding the entity attribute of the at least one of the response and the question portion of the question and response pair to generate a vector for a corresponding participant; and
storing the vector in a vector dictionary.
2 . The method of claim 1 , further including:
identifying a question type; and
processing the question and response pair based on the question type:
wherein the processing the question and responses pair includes at least one of a Boolean response processing, a quantitative response processing, a single response processing, and a multi-response processing.
3 . The method of claim 2 , wherein the Boolean response processing includes collecting a binary state for the question topic.
4 . The method of claim 2 , wherein the single response processing includes performing a topic extraction and an entity extraction on the responses.
5 . The method of claim 2 , wherein the multi-response processing includes performing at least two topic extractions and an entity extraction for each topic on the responses.
6 . The method of claim 1 , further comprising fielding the participants in a usability study.
7 . The method of claim 6 , wherein the fielding the participants includes:
detecting fraudulent participants based upon the vector for each participant;
screening the participants based upon the vector for each participant;
predicting a conversion rate for the participants based upon the vector for each participant;
selecting a provider based upon the conversion rate of the participants in the provider; and
onboarding participants from the provider to the usability study.
8 . The method of claim 7 , further comprising generating a question recommendation for the usability study based upon the vector for each participant.
9 . A system for identifying attributes in a plurality of usability study participants comprising:
a system server configured to,
monitor question and response pairs corresponding to participants in the study by inserting a virtual tracking code to a web site targeted for the study at a local machine to each participant, wherein the tracking code collects data including at least one of number of clicks, keystrokes, keywords, scrolls, and time on tasks, and providing the collected data to a server for additional analysis,
receive question and response pairs for participants in the study,
identify a topic of a question portion of a question and response pair using at least one topic machine learning (ML) model,
receive a plurality of name entity recognition (NER) ML models,
select a preferred NER ML model from the plurality of NER ML models based upon accuracy of each NER ML model to the topic,
identify an entity attribute of at least one of a response and the question portion of the question and response pair using the preferred NER ML model,
decode the entity attribute of the at least one of the response and the question portion of the question and response pair to generate a vector for a corresponding participant, and
a database configured to store the at least one vector as a vector dictionary.
10 . The system of claim 9 , further including:
identifying question type; and
processing the question and response pair based on question type:
wherein the processing the question and responses pair includes at least one of a Boolean response processing, a quantitative response processing, a single response processing, and a multi-response processing.
11 . The method of claim 10 , wherein the Boolean response processing includes collecting a binary state for the question topic.
12 . The system of claim 10 , wherein the single response processing includes performing a topic extraction and an entity extraction on the responses.
13 . The system of claim 10 , wherein the multi-response processing includes performing at least two topic extractions and an entity extraction for each topic on the responses.
14 . The system of claim 9 , further comprising fielding the participants in a usability study.
15 . The system of claim 14 , wherein the fielding the participants includes:
detecting fraudulent participants based upon the vector for each participant;
screening the participants based upon the vector for each participant;
predicting a conversion rate for the participants based upon the vector for each participant;
selecting a provider based upon the conversion rate of the participants in the provider; and
onboarding participants from the provider to the usability study.
16 . The system of claim 15 , further comprising generating a question recommendation for the usability study based upon the vector for each participant.
17 . A method of predicting fulfillment criteria for a usability study comprising:
monitoring question and response pairs corresponding to participants in the study by inserting a virtual tracking code to a web site targeted for the study at a local machine to each participant, wherein the tracking code collects data including at least one of number of clicks. keystrokes, keywords, scrolls, and time on tasks, and providing the collected data to a server for additional analysis; and
performing topic and entity extractions on a question/response pair using a machine learning (ML) model by:
identifying a topic of a question portion of the question and response pair using at least one topic ML model;
receiving a plurality of name entity recognition (NER) ML models;
selecting a preferred NER ML model from the plurality of NER ML models based upon accuracy of each NER ML model to the topic;
performing an entity extraction on the response using the preferred NER ML model;
decoding the extracted topic and entity to generate an attribute for a plurality of study participants;
estimating a conversion rate of a subset of the study participants based upon the rarity of an attribute and the number of the plurality of study participants that are known to have said attribute;
estimate a time to field based upon the estimated conversion rate and a number of extended study offers;
querying a historical study database to compare the usability study to previous usability studies to estimate duration of the study; and
estimate a time to completion for the study based upon the estimated time to field and the estimated duration.