Systems and methods for clustering data to forecast risk and other metrics
Methods and systems are provided for clustering data for use in training models to predict a loss metric for insurance. The historical data is cleaned and aggregated to a quarterly time granularity and a zip code geographic granularity, and features on which to generate the clusters are selected. Clusters are generated using one or more clustering algorithms, evaluated, and the best set of clusters is selected. Clusters may be grouped together based on cluster characteristics. Each cluster or group of clusters is used to train a development model and a forecast model for the loss metric. The accuracy of the development and forecast models is evaluated, and the clustering process is repeated until the accuracy and stability of the development and forecast models is sufficient. At each iteration, historical data may be removed and/or hyperparameters of the clustering algorithms may be adjusted.
1. A computer-implemented method of clustering historical data, the method comprising:
receiving historical data comprising historical policyholder data, historical policy data, historical claims data, historical environmental data, and historical loss metric data, the historical data further comprising at least one input variable;
processing the historical data, comprising:
joining the historical data;
cleaning the historical data; and
aggregating the historical data to a specified time granularity and specified second granularity;
generating a set of features based on the processed historical data;
inputting the feature set into a plurality of clustering algorithms to generate a plurality of sets of clusters, each clustering algorithm generating a set of clusters based on the feature set, each clustering algorithm comprising one of the following: k-means, k-medians, mean shift, DBSCAN, Gaussian mixture model, hierarchical clustering, affinity propagation, and spectral clustering;
evaluating each of the plurality of sets of clusters for stability; and
selecting the most stable set of clusters.
2. The method of claim 1 , wherein the specified time granularity is quarterly.
3. The method of claim 1 , wherein the specified second granularity is a geographic granularity.
4. The method of claim 1 , wherein generating the set of features comprises:
adding input variables of known importance to the feature set; and
adding at least one newly created variable to the feature set.
5. The method of claim 4 , wherein the newly created variable comprises a smoothed loss metric value.
6. The method of claim 4 , wherein the newly created variable comprises a mean of the loss metric over time.
7. The method of claim 4 , wherein the newly created variable comprises a trend of the loss metric over time.
8. The method of claim 4 , wherein the newly created variable comprises a standard deviation of the loss metric over time.
9. The method of claim 1 , wherein the stability of a set of clusters is determined based on the trend of the loss metric of each cluster.
10. The method of claim 1 , wherein the stability of a set of clusters is determined based on the separation between the loss metrics of each cluster.
11. The method of claim 1 , wherein the stability of a set of clusters is determined based on the exposure coverage of the set of clusters.
12. The method of claim 1 , further comprising:
for each cluster in the set of clusters, training a loss metric model based the historical data comprising the cluster.
13. A method of clustering historical data, the method comprising:
receiving historical data comprising historical policyholder data, historical policy data, historical claims data, historical external data, and historical loss metric data, the historical data further comprising at least one input variable;
segmenting the historical data into a plurality of segments;
processing the historical data, comprising:
joining the historical data;
cleaning the historical data; and
for each segment, aggregating the historical data to a specified time granularity and specified second granularity;
for each segment, generating a set of features based on the processed historical data;
for each segment, inputting the feature set into a plurality of clustering algorithms to generate a plurality of sets of clusters, each clustering algorithm generating a set of clusters based on the feature set, each clustering algorithm comprising one of the following: k-means, k-medians, mean shift, DBSCAN, Gaussian mixture model, hierarchical clustering, affinity propagation, and spectral clustering;
evaluating each of the plurality of sets of clusters for stability; and
for each segment, selecting the most stable set of clusters.
14. The method of claim 13 , further comprising
grouping a cluster from one segment with a cluster from a different segment based on an attribute of the loss metric of the clusters.
15. The method of claim 14 , wherein the attribute relates to the value of the loss metric.
16. The method of claim 13 , wherein the specified time granularity is quarterly.
17. The method of claim 13 , wherein the specified second granularity is a geographic granularity.
18. The method of claim 13 , wherein generating the set of features comprises:
adding input variables of known importance to the feature set; and
adding at least one newly created variable to the feature set.
19. The method of claim 18 , wherein the newly created variable comprises one of the following: a smoothed loss metric value, the mean of the loss metric over time, the trend of the loss metric over time, and the standard deviation of the loss metric over time.
20. The method of claim 13 , wherein the stability of a set of clusters is determined based on one or more of the following: the trend of the loss metric of each cluster, the separation between the loss metrics of each cluster, and the exposure coverage of the set of clusters.