Modeling Response Variable That will change over time
This is a question about insurance data. The response variable is whether or not a given claim will be >100k (i.e. large). The data is evaluated as of a certain date, and it includes claims that are only a few days old and some that are years old. Any of these claims that are not yet >100k could be if we evaluate them in a month or even in a year. The new claims (less than six months) are a lot less likely to be large, even if they will end up being that way - usually the amount that will be paid isn't known that early.
What I would like to know is if there is any value to a logistic model that predicts whether a claim will be large (given various characteristics) even though the data used to build it isn't really reflective of the true proportion of claims that will be large when they are all closed and paid.
Also, can this model be used on the claims that were used to build it? E.G. A claim that is not large, but has a high predicted probability of being large.
Thanks in advance.
Posted Sep 1, 2017, 4:50 PM UTC
Do you have data on claims that were already paid? Without that, for the older claims, you only have information about how large claims will be given that they have yet to be paid at a certain time. The younger claims could be unpaid at some later date, or they could be paid earlier, so you have a missing data problem that will mess up the kind of model you're describing.
You might also want to look into survival analysis. A doctor might be interested in predicting whether, starting at time t, a patient with explanatory variables X will die by time t + k. They might use a Cox model to predict this, which is a generalized linear model similar to a logistic model. If you replace "patient" with "claims" and "die" with "reach 100K", you could use the same kind of model. However, this also requires that you have older paid claims, since those below 100k will technically have "survived" since they were filed.
A more general model would be a log-linear model that predicts the log of the actual claim amount based on the explanatory variables AND the age of the claim. Again, this would only predict the size of a claim given that it has not been paid by time t.
If you do have data on claims that were paid already, AND you have information about the amount for the same claim over time, you can use these to predict the amount a claim will be at time t + k given the explanatory variables and the amount at time t. Importantly, claims of amount Y paid at time t would also be at amount Y at time t + k. This could either be a log-linear model or a logistic model (Y = whether >100k).