Translator-based scoring and benchmarking for user experience testing and design optimizations
Techniques are described herein for providing adaptable testing and benchmarking of user experiences with respect to one or more products. In some embodiments, the techniques include systems and methods for predicting performance of facets of a user experience under new testing and benchmarking methodologies. The systems and methods may generate a prediction for the results of a UX test even though the methodology and mechanics to quantify the user experience may vary significantly from previous methodologies. The techniques allow for methodologies to evolve over time without losing historical context or the ability to meaningfully compare historical test results with tests run using updated testing and benchmark models. Further, the techniques allow for benchmarks to be computed in real-time or near real-time as methodologies change without requiring tests to be run according to the new methodologies.
1. A method comprising:
receiving a first set of values that quantify a user experience associated with a product according to at least one source methodology;
generating, by a model based on the first set of values that quantify the user experience associated with the product according to said at least one source methodology, at least one prediction that quantifies the user experience associated with the product according to a target methodology, wherein the target methodology is different than said at least one source methodology, wherein the target methodology defines an updated benchmark model that modifies a benchmark model used by said at least one source methodology;
presenting, based on said at least one prediction for the user experience associated with the product according to the target methodology, an indication of a predicted performance of the product with respect to at least one facet of the user experience;
tracking changes to the updated benchmark model; and
responsive to detecting changes to the updated benchmark model, generating, by the model, new predictions for the user experience under the updated benchmark model without running a test to directly measure benchmarks under the updated benchmark model.
2. The method of claim 1 , further comprising:
receiving, by the model, feedback associated with said at least one prediction;
responsive to receiving the feedback, performing at least one update to at least one parameter of the model; and
generating, by the model based at least in part on said at least one update to said at least one parameter of the model, at least one additional prediction for the user experience associated with the product.
3. The method of claim 1 , further comprising:
detecting a change in the target methodology; and
responsive to detecting the change in the target methodology, updating at least one parameter of the model.
4. The method of claim 1 , wherein one or more of the first set of values are generated by running a user experience test according to the first methodology.
5. The method of claim 1 , further comprising:
determining that an input value to the model is missing;
responsive to determining that the input value to the model is missing, inferring the input value based on one or more other input values to the model;
wherein the first set of values includes the input value inferred from the one or more other input values to the model.
6. The method of claim 1 ,
wherein the model is a neural network comprising a plurality of cells and connection weights between different cells of the plurality of cells;
wherein the connection weights between different cells of the plurality of cells are determined based at least in part on relationship strengths between the facet tests in the target methodology and facet tests in said at least one source methodology.
7. The method of claim 6 , further comprising:
performing unsupervised adjustments of the plurality of connection weights based at least in part on an estimation error of the model;
wherein the estimation error is determined based at least in part on a difference in said at least one prediction and a result of running a user experience test according to the target methodology.
8. The method of claim 1 , wherein the indication identifies at least one facet that is predicted to underperform relative to a benchmark.
9. The method of claim 1 , further comprising:
executing an action based at least in part on said at least one facet that is predicted to underperform relative to the benchmark; wherein the action includes at least one of recommending or deploying an update to the product that is predicted to improve the user experience with respect to said at least one facet.
10. One or more non-transitory computer-readable media storing instructions which, when executed by one or more hardware processors cause:
receiving a first set of values that quantify a user experience associated with a product according to at least one source methodology;
generating, by a model based on the first set of values that quantify the user experience associated with the product according to said at least one source methodology, at least one prediction that quantifies the user experience associated with the product according to a target methodology, wherein the target methodology is different than said at least one source methodology, wherein the target methodology defines an updated benchmark model that modifies a benchmark model used by said at least one source methodology;
presenting, based on said at least one prediction for the user experience associated with the product according to the target methodology, an indication of a predicted performance of the product with respect to at least one facet of the user experience;
tracking changes to the updated benchmark model; and
responsive to detecting changes to the updated benchmark model, generating, by the model, new predictions for the user experience under the updated benchmark model without running a test to directly measure benchmarks under the updated benchmark model.
11. The media of claim 10 , wherein the instructions further cause:
receiving, by the model, feedback associated with said at least one prediction;
responsive to receiving the feedback, performing at least one update to at least one parameter of the model; and
generating, by the model based at least in part on said at least one update to said at least one parameter of the model, at least one additional prediction for the user experience associated with the product.
12. The media of claim 10 , wherein the instructions further cause:
detecting a change in the target methodology; and
responsive to detecting the change in the target methodology, updating at least one parameter of the model.
13. The media of claim 10 , wherein one or more of the first set of values are generated by running a user experience test according to the first methodology.
14. The media of claim 10 , wherein the instructions further cause:
determining that an input value to the model is missing;
responsive to determining that the input value to the model is missing, inferring the input value based on one or more other input values to the model;
wherein the first set of values includes the input value inferred from the one or more other input values to the model.
15. The media of claim 10 ,
wherein the model is a neural network comprising a plurality of cells and connection weights between different cells of the plurality of cells;
wherein the connection weights between different cells of the plurality of cells are determined based at least in part on relationship strengths between the facet tests in the target methodology and facet tests in said at least one source methodology.
16. The media of claim 10 , wherein the indication identifies at least one facet that is predicted to underperform relative to a benchmark.
17. The media of claim 16 , wherein the instructions further cause:
executing an action based at least in part on said at least one facet that is predicted to underperform relative to the benchmark; wherein the action includes at least one of recommending or deploying an update to the product that is predicted to improve the user experience with respect to said at least one facet.
18. A system comprising:
one or more hardware processors;
one or more non-transitory computer-readable media storing instructions which, when executed by the one or more hardware processors cause:
receiving a first set of values that quantify a user experience associated with a product according to at least one source methodology;
generating, by a model based on the first set of values that quantify the user experience associated with the product according to said at least one source methodology, at least one prediction that quantifies the user experience associated with the product according to a target methodology, wherein the target methodology is different than said at least one source methodology, wherein the target methodology defines an updated benchmark model that modifies a benchmark model used by said at least one source methodology;
presenting, based on said at least one prediction for the user experience associated with the product according to the target methodology, an indication of a predicted performance of the product with respect to at least one facet of the user experiences;
tracking changes to the updated benchmark model; and
responsive to detecting changes to the updated benchmark model, generating, by the model, new predictions for the user experience under the updated benchmark model without running a test to directly measure benchmarks under the updated benchmark model.
19. The system of claim 18 , wherein the instructions further cause:
receiving, by the model, feedback associated with said at least one prediction;
responsive to receiving the feedback, performing at least one update to at least one parameter of the model; and
generating, by the model based at least in part on said at least one update to said at least one parameter of the model, at least one additional prediction for the user experience associated with the product.
20. The system of claim 18 , wherein the instructions further cause:
detecting a change in the target methodology; and
responsive to detecting the change in the target methodology, updating at least one parameter of the model.