Selectively regularizing some features harder than others?
Does anyone know of any precedent for selectively assigning a higher regularization penalty to some features than to others?
The context of my asking this is that I am currently thinking of adding some polynomial transformations of features to my model (the proper term for this escapes me at the moment, sorry... lifting?). The linear model already over-fits a little bit, but it also works alright, so I am thinking lasso regression might zero out some actually useful coefficients on the linear terms. Thus I wonder, what would happen if I applied an l1 or l2 penalty to just the higher order features I add, and what if I scale the penalty to be higher on the higher degree terms?
Does anyone know if there is a name for this approach and any research on it? As much as I'd love for this idea to be original, I'd love even more to crack open a paper that goes point by point through this line of thinking and provides some experimental results!
Posted Aug 16, 2017, 3:49 AM UTC
The closest thing I can think of is 'group lasso' in which multiple variates shrink together. So for example you'd tie all your linear and polynomial terms into two groups instead of each feature.
In this case I don't think it'd be greatly useful, esp. if you're already over fitting.
Interesting to read the group lasso section of the lasso wikipedia page, I had no idea that this kind of thing is fairly recent (paper published in '06)! It seems like I might have to see about trying some selective regularization out on my own.