Method and apparatus for testing automated valuation models
View Patent ↗A method and apparatus for real time testing of automated valuation models using various indicators of accuracy. These indicators are then weighted according to their value as indicators of accuracy using individualized weighting factors or an equation. A ranking is then computed based upon the factors and their weights. This method is preformed continuously, so as to achieve real-time or periodically updated automated valuation model accuracy rankings.
1. A computer-based method of measuring the accuracy of a plurality of automated valuation models for real estate properties, the method comprising:
receiving new valuation data for a subject property;
receiving an automated valuation of said subject property from each of a plurality of automated valuation models, each of said automated valuations performed without reference to said new valuation data;
comparing said new valuation data for said subject property with said automated valuation of said subject property received from each of said automated valuation models; and
computing a ranking of each of said plurality of automated valuation models according to accuracy criteria, wherein an accuracy criterion for an automated valuation model is based at least in part on said comparison of said new valuation data and said automated valuation of said subject property received from said automated valuation model,
wherein the method is performed in its entirety using computer hardware.
2. The method of claim 1 , wherein said accuracy criterion for an automated valuation model includes a plurality of indicators of accuracy based at least in part upon said new valuation data and said automated valuation.
3. The method of claim 2 , wherein said computing further includes weighting said plurality of indicators of accuracy.
4. The method of claim 1 , further comprising updating a database with said ranking of each of said plurality of automated valuation models.
5. The method of claim 1 , further comprising updating a database with said new valuation data.
6. The method of claim 1 , further comprising updating an automated valuation model with said new valuation data.
7. The method of claim 1 , further comprising requesting the automated valuation of said subject property from each of said plurality of automated valuation models, wherein each of said automated valuations are requested and compared prior to each of said plurality of automated valuation models being updated with said new valuation data.
8. The method of claim 1 , wherein said ranking of the automated valuation models is performed in real time.
9. The method of claim 1 , wherein said ranking of the automated valuation models is performed periodically, at time intervals, or upon receipt of a command.
10. The method of claim 1 , further comprising updating a database with said automated valuation of said subject property.
11. The method of claim 1 , wherein said new valuation data comprises property sale data.
12. The method of claim 1 , wherein said new valuation data comprises an appraisal for said subject property.
13. The method of claim 1 , wherein said new valuation data comprises an automated valuation for said subject property.
14. The method of claim 1 wherein said comparing is performed using a percentage of properties for which said automated valuation is returned.
15. The method of claim 1 wherein said comparing is performed using a percentage of properties for which said new valuation data is within a percentage of said automated valuation.
16. The method of claim 1 wherein said comparing is performed using a percentage of properties for which said new valuation data is outside of a percentage of said automated valuation.
17. The method of claim 1 wherein said comparing is performed using at least one of: a median error percentage of said automated valuation with reference to said new valuation data, a mean squared error percentage of said automated valuation with reference to said new valuation data, an absolute mean error percentage of said automated valuation with reference to said new valuation data.
18. The method of claim 1 wherein said accuracy criterion for an automated valuation model is determined using:
Score= f *(( h+c )−(| m|+a+s+o ))
Where
f =multiplier factor
h =hit rate percentage
c =automated valuation model accuracy
m =median error to the reference value
a =absolute mean error
s =mean squared error
o =percentage incidence of outlier values.
19. The method of claim 1 further comprising categorizing said subject property according to at least one of: geographic area, economic tier, and property type.
20. The method of claim 1 further comprising: the steps of
updating a database with said ranking;
updating a database with said new valuation data; and
updating a database with said automated valuations.
21. The method of claim 1 , wherein said computing a ranking further comprises ranking said plurality of automated valuation models based at least in part on a range of prices for said real estate properties.
22. The method of claim 1 , wherein said computing a ranking further comprises ranking said plurality of automated valuation models based at least in part on a cost to request an automated valuation from each of said plurality of automated valuation models.
23. The method of claim 22 , further comprising determining, based at least in part on the accuracy criteria and the costs, a primary automated valuation model that provides highest accuracy for the cost.
24. The method of claim 1 , wherein an automated valuation model requires a format for a valuation request, the method further comprises formatting the valuation request according to the required format.
25. A computer-based method of measuring the accuracy of automated valuation models for real estate properties, the method comprising:
accessing valuation data for a plurality of real estate properties;
computing a first accuracy score for each of a first plurality of automated valuation models, the first accuracy score based at least in part on the valuation data and automated valuations provided by the first plurality of automated valuation models for at least some of the real estate properties;
determining from the first plurality of automated valuation models a primary automated valuation model based at least in part on the first accuracy scores;
computing a second accuracy score for each of a second plurality of automated valuation models, the second plurality of automated valuation models including at least some of the first plurality of automated valuation models but not including the primary automated valuation model, the second accuracy score based at least in part on the valuation data and automated valuations provided by the second plurality of automated valuation models for real estate properties for which the primary automated valuation model was not able to return an automated valuation; and
determining from the second plurality of automated valuation models a secondary automated valuation model based at least in part on the second accuracy scores,
wherein the method is performed in its entirety by a computer system comprising computer hardware.
26. The method of claim 25 , wherein the second plurality of automated valuation models is the first plurality of automated valuation models except for the primary automated valuation model.
27. The method of claim 25 , wherein the primary automated valuation model and the secondary automated valuation model are determined for at least one of: a geographic area, an economic tier, a property type, and a range of prices for the real estate properties.
28. The method of claim 25 , wherein the valuation data comprises prior valuation data and new valuation data, the new valuation data obtained more recently than the prior valuation data and not included in the prior valuation data, and wherein (1) automated valuations provided by the first plurality of automated valuation models are performed without reference to the new valuation data, (2) automated valuations provided by the second plurality of automated valuation models are performed without reference to the new valuation data, or both (1) and (2).
29. The method of claim 25 , further comprising updating a database with information relating to the primary automated valuation model and the secondary automated valuation model.