Method and apparatus for testing automated valuation models
View Patent ↗A method and apparatus for real time testing of automated valuation using various indicators of accuracy. These indicators are then weighted according to their value as indicators of accuracy using individualized weighting factors or an equation. A ranking is then computed based upon the factors and their weights. This method is preformed continuously, so as to achieve real-time or periodically updated automated valuation model accuracy rankings.
1. A system for determining the accuracy of automated valuation models (AVMs) that provide automated valuations for real estate properties, the system comprising:
physical data storage configured to store (1) a reference valuation for a subject property and (2) a plurality of automated valuations of the subject property, each automated valuation of the subject property generated by an automated valuation model (AVM) without use of the reference valuation for the subject property, wherein different automated valuations correspond to different AVMs; and
a computer system in communication with the physical data storage, the computer system comprising computer hardware, the computer system programmed to:
calculate one or more indicators of accuracy for each AVM, the indicators of accuracy for each AVM based at least in part on the AVM's automated valuation of the subject property and the reference valuation of the subject property, wherein the AVM generates the automated valuation of the subject property before being updated with the reference valuation for the subject property, the reference valuation available to but not used by the AVM for the automated valuation; and
determine a ranking of the plurality of AVMs based at least in part on the one or more indicators of accuracy.
2. The system of claim 1 , wherein the reference valuation for the subject property comprises at least one of: a sale price for the subject property, an appraisal for the subject property, an AVM valuation of the subject property, and a foreclosure value for the subject property.
3. The system of claim 1 , wherein at least one of the indicators of accuracy comprises an error value calculated based at least in part on the automated valuation of the subject property by an AVM and the reference valuation of the subject property.
4. The system of claim 3 , wherein the error value comprises at least one of: an absolute mean error, a median error, and a mean squared error.
5. The system of claim 1 , wherein at least one of the indicators of accuracy is an AVM accuracy indicator based at least in part on the number of automated valuations of the subject property that are within a percentage of the reference valuation for the subject property.
6. The system of claim 1 , wherein at least one of the indicators of accuracy is an outlier indicator based at least in part on the number of automated valuations of the subject property that are outside a percentage of the reference valuation for the subject property.
7. The system of claim 1 , wherein at least one of the indicators of accuracy is a hit rate indicator based at least in part on the number of real estate properties for which the AVM is able to return an automated valuation.
8. The system of claim 1 , wherein the computer system is programmed to:
determine an accuracy score for each of the plurality of AVMs based at least in part on the one or more indicators of accuracy for each AVM; and
determine the ranking of the plurality of AVMs based at least in part on the accuracy scores.
9. The system of claim 8 , wherein the computer system is programmed to determine the accuracy score by weighting at least some of the indicators of accuracy.
10. The system of claim 1 , wherein the computer system is programmed to determine the ranking of the plurality of AVMs based at least in part on costs to request the automated valuations from the plurality of AVMs.
11. The system of claim 1 , wherein the computer system is programmed to determine the ranking of the plurality of AVMs in one or more categories for the real estate properties.
12. The system of claim 11 , wherein the one or more categories comprise at least one of: an economic tier, a property type, and a geographic area.
13. A system for determining the accuracy of automated valuation models that provide automated valuations for real estate properties, the system comprising:
physical data storage configured to store reference valuations for real estate properties and automated valuations of the real estate properties; and
a computer system in communication with the physical data storage, the computer system comprising computer hardware, the computer system programmed to:
determine first accuracy rankings for a first plurality of automated valuation models (AVMs), the first accuracy rankings based at least in part on the reference valuations for the real estate properties and automated valuations provided by the first plurality of automated valuation models for at least some of the real estate properties;
determine a primary automated valuation model (primary AVM) based at least in part on the first accuracy rankings;
determine second accuracy rankings for a second plurality of AVMs, the second plurality of AVMs including at least some of the first plurality of AVMs but not including the primary AVM, the second accuracy rankings based at least in part on the reference valuations of the real estate properties and automated valuations provided by the second plurality of AVMs for real estate properties for which the primary AVM was not able to return an automated valuation; and
determine a secondary automated valuation model (secondary AVM) based at least in part on the second accuracy rankings.
14. The system of claim 13 , wherein at least one of the automated valuations of a real estate property provided by the first plurality of AVMs or by the second plurality of AVMs is generated without use of the reference valuation of the real estate property.
15. The system of claim 13 , wherein the second plurality of AVMs includes the first plurality of AVMs except for the primary AVM.
16. The system of claim 13 , wherein the primary AVM and the secondary AVM are determined for at least one of: a geographic area, an economic tier, and a property type for the real estate properties.
17. A system for determining the accuracy of automated valuation models (AVMs) that provide automated valuations for real estate properties, the system comprising:
physical data storage configured to store at least one indicator of accuracy for each of a plurality of automated valuation models (AVMs), the at least one indicator of accuracy for an AVM based at least in part on (1) a reference valuation for a subject property and (2) a valuation generated by the AVM for the subject property, wherein the valuation is generated without use of the reference valuation for the subject property; and
a computer system in communication with the physical data storage, the computer system comprising computer hardware, the computer system programmed to:
determine a ranking of the plurality of AVMs based at least in part on the indicators of accuracy for the AVMs;
determine a primary AVM based on the ranking of the plurality of AVMs; and
determine a secondary AVM based at least in part on indicators of accuracy for real estate properties for which the primary AVM was not able to return an automated valuation.
18. The system of claim 17 , wherein at least one of the indicators of accuracy comprises an error value calculated based at least in part on the automated valuation of the subject property by an AVM and the reference valuation of the subject property.
19. The system of claim 18 , wherein the error value comprises at least one of: an absolute mean error, a median error, and a mean squared error.
20. The system of claim 17 , wherein the computer system is programmed to:
determine an accuracy score for each of the plurality of AVMs based at least in part on the at least one indicator of accuracy for each AVM; and
determine the ranking of the plurality of AVMs based at least in part on the accuracy scores.
21. The system of claim 20 , wherein the computer system is programmed to determine the accuracy score by weighting at least some of the indicators of accuracy.