Source
r/MachineLearning•
eric_he

[Discussion] Does SMOTE or other class balancing techniques actually improve Area Under Precision-Recall?

Discussion

Hi all,

I've been working on a prediction problem with a fairly strong class imbalance using tree ensembles. I have no problems with too much or too little data, but I do read everywhere that addressing class imbalance improves predictive power.

However, most of the papers (including the [paper](https://jair.org/index.php/jair/article/view/10302) for SMOTE) evaluate their performance in terms of benefits to the Area under the ROC curve (AuROC). This is not the best or most informative metric to use as the AuROC metric is sensitive to class imbalance. See [this paper](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4349800/) which explains why the Area under Precision-Recall (AuPR) metric is better in such situations. Essentially, AuROC is boosted when the classifier correctly predicts either a True Positive or a True Negative, which in cases of class imbalance can lead to drastic inflation of the score due to the True Negatives making up the majority of the dataset. The AuPR score only increases when the model correctly identifies a True Positive.

To address that, the SMOTE paper evaluates the AuROC on a separate balanced dataset each time it rebalances. However, the improvements to their AuROC score still derive from tradeoffs between precision and recall, making the benefits of the technique difficult to assess. (Moreover the AuROC benefits were marginal and susceptible to the multiple comparisons problem).

In my experience, using simple undersampling or oversampling techniques never changed my AuPR score, and only served to make me recalibrate my probability predictions. SMOTE, however, seems to be a fairly sophisticated technique and I have not seen it being questioned. Despite this, I can find no literature involving the AuPR metric.

Am I understanding something incorrectly? Do any of you have any experience with SMOTE and evaluating SMOTE, or other class balancing techniques? Did SMOTE or other class balancing techniques improve your predictive power?

Thank you so much!

Posted Jun 21, 2018, 3:01 PM UTC

514
uhm_throwaway2•

A better AuROC means a better AuPR. To see this, compare points on two ROC curves a and b (with a being strictly better than b) at a fixed FPR.

At that fixed FPR, TPR_a > TPR_b, and you can show that Pr_a > Pr_b and Re_a > Re_b.

So a curve that is higher in ROC space will also be higher in PR space.

In practice, I personally find that balancing classes really does lead to "better" performance, not just a shift in the probability calibration. See this stack answer for example:

https://stats.stackexchange.com/questions/157714/r-package-for-weighted-random-forest-classwt-option

4
eric_heOP•

Thanks, I forgot that higher AuROC strictly implies higher AuPR. And thank you for the reference!

2
lalaland11256•

There is an important caveat here. An ROC curve which is higher at any point than another curve implies a higher PR curve. However, if two ROC curves overlap, then the relationship between ROC and PR curves is a tad less straightforward. It is totally possible to have a higher AUROC and a lower AUPRC due to this overlap issue.

2
eric_heOP•

Yes, you're right, thank you!

1
acesh•

Perhaps worth looking at this NIPS 2015 paper. They proposed a curve called precision-recall-gain curve, the area under this curve is more consistent with normal ROC curves.

https://papers.nips.cc/paper/5867-precision-recall-gain-curves-pr-analysis-done-right

3
eric_heOP•

Thank you, will definitely read it!

1
xgrayskullx•

I am working on publishing a paper that evaluates under-sampling (via TomekLinks), over-sampling (via SMOTE), and the combination of the two in a variety of different algorithms on a fairly (4/96) imbalanced dataset. It's actually an interesting problem because we're trying to identify illicit behavior utilizing a model. The dataset has an unknown number of incorrectly labeled majority class members (should be minority class, but is labeled as majority. Think of trying to identify criminals, when you know some of the people who haven't been arrested have definitely robbed a bank)

What I'm seeing is that oversampling generally improves the performance of a model. Most specifically I'm looking at negative predictive value and specificity because those are the most relevant metrics for the use-case. However, the improvement in NPV and specificity do not seem to be as large with non-linear models as compared to linear models. It's very likely that these observations are very specific to the dataset I'm working with.

On a related note, at least with the models and datasets I'm working with, undersampling the majority class doesn't appear to improve performance (not surprising), and neither does a combination of oversampling (via SMOTE) and undersampling (via Tomek) (but in some instances, this works better than either alone).

2
eric_heOP•

Hi, thanks for your answer. This is interesting as the others in this post have had varying responses about the effectiveness of undersampling/oversampling. I think it's possibly a question of the complexity of the data itself; some sampling techniques may work better when there's not a lot of overlap between the two classes in data space compared to when there is a lot of overlap. Looking forward to your paper!

1
[deleted]•

[deleted]

1
eric_heOP•

With respect to either the Area under Precision-Recall or Area under ROC metrics, does a different data distribution between the train and test data matter? Both metrics evaluate the ranking of the probabilities the model assigns to test instances, rather than the probabilities themselves.

Even if the predicted probabilities are miscalibrated (e.g. on the set of data for which the model predicts 99% probability of a positive outcome, the data is positive 90% of the time), as long as the probabilities of positive instances tend to be larger than the probabilities of negative ones, it should be fine?

I guess my underlying assumption here is that rebalancing data only serves to miscalibrate probability predictions.

2
_zaytsev_•

Yeah I think rebalancing test sets is disingenuous at best and a surprising number of papers get away with it. Anyone who has worked with imbalanced data (~1% positive class) would know that the PR curves look nothing like the stuff drawn in textbooks.

As for if SMOTE works, a couple of times where we did use it, the results were same as oversampling the minority or undersampling the majority class (only in the training sets obviously). So in practice for us, undersampling works the best as it is the fastest (plus you can build ensemble models from these multiple undersampled sets if you have time/compute power). I think this paper Learning from Imbalanced Data is a good one to check out if you haven't already. It discusses SMOTE and a host of other techniques.

3
maxToTheJ•

As for if SMOTE works, a couple of times where we did use it, the results were same as oversampling the minority or undersampling the majority class (only in the training sets obviously). So in practice for us, undersampling works the best as it is the fastest (plus you can build ensemble models from these multiple undersampled sets if you have time/compute power).

Same experience here

3
eric_heOP•

Hi, thanks for sharing your experience. I am wondering what you mean by different-looking PR curves? Problems I've worked on have AuPR ranging from .14 to .8, with the curves downward sloping as one would expect.

There is no use in building multiple models using multiple undersampled sets if the base model is already a tree ensemble, is that correct?

I am reading the paper now and it is great.

2
_zaytsev_•

I meant the AuPR is lower for a dataset with highly imbalanced datasets in practice than what one would expect (of course that would change from dataset to dataset etc), my apologies I should have been clearer.

If you are training with an undersampled train set, then your classifier (tree ensemble or not) will learn on that undersampled set. If you have multiple of these undersampled sets (which you do if you have an imbalanced set), then each of these classifiers will still get to see different data points from the undersampled class. So it could still be beneficial. Now this might work (or not) but the kind of classifier you are using shouldn't affect the overall idea if that makes sense.

2
bobchennan•

https://jair.org/index.php/jair/article/view/10302

Guess you are right. Your case is different.

2