Entity scoring calibration
Entity scoring calibration includes receiving values for a metric by which to calibrate a reputation scoring model. The reputation scoring model is usable to determine a reputation score. The reputation scoring model to be calibrated is based at least in part on a set of model parameters associated with reputation scoring components. It further includes receiving a plurality of feedback items pertaining to one or more entities. It further includes calibrating the reputation scoring model at least in part by adjusting at least some of the model parameters associated with the reputation scoring components such that reputation scores generated by the calibrated reputation scoring model track the values for the metric.
1. A system, comprising:
one or more processors configured to:
receive values for a target metric by which to calibrate a reputation scoring model, wherein the reputation scoring model is usable to determine a reputation score, and wherein the reputation scoring model to be calibrated is based at least in part on a plurality of model parameters associated with reputation scoring components;
receive a plurality of feedback items pertaining to one or more entities;
calibrate the reputation scoring model at least in part by adjusting at least some of the plurality of model parameters associated with the reputation scoring components such that reputation scores generated by the calibrated reputation scoring model track the values for the target metric, wherein a type of calibration that is performed is based on a type of the target metric for which the reputation scoring model is to be calibrated, wherein the reputation scoring model comprises a machine learning model, wherein performing the calibration comprises updating weights of the machine learning model at least in part by solving an optimization problem, and wherein a type of optimization that is performed is based at least in part on the type of the target metric;
use an adjusted model parameter associated with a reputation scoring component to determine an impact of the reputation scoring component on the target metric that the reputation scoring model is calibrated to track;
based at least in part on the impact of the reputation scoring component on the tracked target metric determined using the model parameter adjusted based on the calibrating, provide as output a recommended action associated with the reputation scoring component to improve the target metric that the reputation scoring model is calibrated to track; and
update the calibration of the reputation scoring model based at least in part on new values of the target metric, including re-running the calibration and updating the weights of the machine learning model at least in part by using the new values of the target metric and performing the type of optimization that is based at least in part on the type of the target metric; and
a memory coupled to the one or more processors and configured to provide the one or more processors with instructions.
2. The system recited in claim 1 , wherein the one or more processors are further configured to, based at least in part on the calibration, predict a change in value of the metric in response to a change in the reputation score.
3. The system recited in claim 1 , wherein calibrating the reputation scoring model comprises solving an objective function.
4. The system recited in claim 1 , wherein the metric by which the reputation scoring model is calibrated comprises one of sales, return on investment, search ranking, and a customer retention metric.
5. The system recited in claim 1 , wherein the reputation scoring components comprise at least one of review volume, sentiment, responsiveness, and listings accuracy.
6. The system recited in claim 2 , wherein the plurality of feedback items comprises at least one of reviews, surveys, listings, social comments, and search results.
7. The system recited in claim 1 , wherein for a first type of metric determined relative to at least one other entity, the calibration comprises updating the weights of the machine learning model at least in part by performing a pairwise optimization.
8. A method, comprising:
receiving values for a target metric by which to calibrate a reputation scoring model, wherein the reputation scoring model is usable to determine a reputation score, and wherein the reputation scoring model to be calibrated is based at least in part on a plurality of model parameters associated with reputation scoring components;
receiving a plurality of feedback items pertaining to one or more entities;
calibrating the reputation scoring model at least in part by adjusting at least some of the plurality of model parameters associated with the reputation scoring components such that reputation scores generated by the calibrated reputation scoring model track the values for the target metric, wherein a type of calibration that is performed is based on a type of the target metric for which the reputation scoring model is to be calibrated, wherein the reputation scoring model comprises a machine learning model, wherein performing the calibration comprises updating weights of the machine learning model at least in part by solving an optimization problem, and wherein a type of optimization that is performed is based at least in part on the type of the target metric;
using an adjusted model parameter associated with a reputation scoring component to determine an impact of the reputation scoring component on the target metric that the reputation scoring model is calibrated to track;
based at least in part on the impact of the reputation scoring component on the tracked target metric determined using the model parameter adjusted based on the calibrating, providing as output a recommended action associated with the reputation scoring component to improve the target metric that the reputation scoring model is calibrated to track; and
updating the calibration of the reputation scoring model based at least in part on new values of the target metric, including re-running the calibration and updating the weights of the machine learning model at least in part by using the new values of the target metric and performing the type of optimization that is based at least in part on the type of the target metric.
9. The method of claim 8 , further comprising, based at least in part on the calibration, predicting a change in value of the metric in response to a change in the reputation score.
10. The method of claim 8 , wherein calibrating the reputation scoring model comprises solving an objective function.
11. The method of claim 8 , wherein the metric by which the reputation scoring model is calibrated comprises one of sales, return on investment, search ranking, and a customer retention metric.
12. The method of claim 8 , wherein the plurality of feedback items comprises at least one of reviews, surveys, listings, social comments, and search results.
13. The method of claim 8 , wherein for a first type of metric determined relative to at least one other entity, the calibration comprises updating the weights of the machine learning model at least in part by performing a pairwise optimization.
14. A computer program product embodied in a non-transitory computer readable storage medium and comprising computer instructions for:
receiving values for a target metric by which to calibrate a reputation scoring model, wherein the reputation scoring model is usable to determine a reputation score, and wherein the reputation scoring model to be calibrated is based at least in part on a plurality of model parameters associated with reputation scoring components;
receiving a plurality of feedback items pertaining to one or more entities;
calibrating the reputation scoring model at least in part by adjusting at least some of the plurality of model parameters associated with the reputation scoring components such that reputation scores generated by the calibrated reputation scoring model track to the values for the target metric, wherein a type of calibration that is performed is based on a type of the target metric for which the reputation scoring model is to be calibrated, wherein the reputation scoring model comprises a machine learning model, wherein performing the calibration comprises updating weights of the machine learning model at least in part by solving an optimization problem, and wherein a type of optimization that is performed is based at least in part on the type of the target metric;
using an adjusted model parameter associated with a reputation scoring component to determine an impact of the reputation scoring component on the metric that the reputation scoring model is calibrated to track;
based at least in part on the impact of the reputation scoring component on the tracked target metric determined using the model parameter adjusted based on the calibrating, providing as output a recommended action associated with the reputation scoring component to improve the target metric that the reputation scoring model is calibrated to track; and
updating the calibration of the reputation scoring model based at least in part on new values of the target metric, including re-running the calibration and updating the weights of the machine learning model at least in part by using the new values of the target metric and performing the type of optimization that is based at least in part on the type of the target metric.
15. The computer program product of claim 14 , further comprising computer instructions for:
based at least in part on the calibration, predicting a change in value of the metric in response to a change in the reputation score.
16. The computer program product of claim 14 , wherein calibrating the reputation scoring model comprises solving an objective function.