[Q] What to do with large sample size causing tests to be rejected?

Question

I'm running a Chi-Square test of independence to test if two categorical variables are related. Because of our sample size (3k to even 500k), the tests are always rejected which makes sense because even if an event is random, if it happens enough time the test would still consider it "too many times".

Edit: I want to emphasis that this is an expected behavior and I'm interested in how to properly handle this, rather than why this is happening. I knew this will happen.

I'm wondering what's the appropriate next step then? Is it to find a sample size, n, such that Chi-Square test is still efficient and just randomly draw samples up to n?

Or is there a similar test that's less affected by sample size?

I know there's Fisher's exact test but it seems like it's also affected by large sample size.

Posted Mar 16, 2022, 9:03 PM UTC · edited

110
D-Juice•

This is about practical significance, not statistical significance. Your chi-square tests are picking up small differences that appear unlikely to be due to chance. If they're too small for you to be interested in them, you can just say so.

7
efrique•

Because of our sample size (3k to even 500k), the tests are always rejected which makes sense because even if an event is random, if it happens enough time the test would still consider it "too many times".

You're mistaken. If there is truly no relationship and the assumptions hold, the rejection rate should be no more than the significance level.

In smallish samples - because of the mismatch between the distribution of the test statistic (which is discrete) and the chi-squared approximation - the rejection rate may be somewhat higher than desired under H0.

We can demonstrate (via both algebra and simulation) that in large samples the tests maintain the desired significance level (if the null is actually true, asymptotically the type I error rate doesn't exceed alpha, as long as the assumptions hold and in sufficiently large samples*, doesn't exceed it by much)

You can (in just the right circumstances) get some significance level mismatch (and some test bias as well) even with fairly large samples (but not so much in the 2x2 case). However this really does disappear asymptotically, so large samples is better for that, not worse.

What happens is you become more able to pick up very small effects. It's supposed to do that. If you're not interested in doing that, stop using hypothesis tests with point nulls. (That's like picking up a wrench and using it as a hammer, meanwhile complaining that it's bending the nails too often. If you want to hammer nails, pick up a hammer and leave the wrench alone.)

What you really do need to worry about (a lot) with very large samples is the impact of even very small amounts of sampling bias, since as sample size grows, the effect of such sampling biases quickly dominate the ever-shrinking sampling variation. People often fail to randomly sample but ignore its impact on the behavior of their inference.

What to do with large sample size causing tests to be rejected?

Point nulls are almost never exactly true; naturally, then, almost all tests should reject in large samples. If that seems undesirable, you're using the wrong tool (a test that has a point null). Maybe you need a different null but often it turns out the reason it seems problematic was that any test at all wouldn't actually answer the original research question (e.g. quite often it turns out to be a 'how much' question)

Rejecting false nulls is quite literally entirely the point of hypothesis tests. Tests that don't do that (that don't have the property of consistency - of the false-null rejection probability going to 1 as sample sizes goo to infinity), most people refuse to use. It's seen as a good reason to regard a test as bad.

see https://en.wikipedia.org/wiki/Consistency_(statistics) (especially the first sentence)

and also https://en.wikipedia.org/wiki/Consistency_(statistics)#Tests

You're asking for a test to not have that.

We go to considerable trouble to give people the very best possible chance to reject false nulls, even in large samples (if you study inference, getting the most information out of the data and getting the best power in a test is what a large fraction of the effort is ultimately devoted to, its why we get all bothered about sufficient statistics, Cramer-Rao bounds, the Neyman-Pearson lemma, generalized likelihood ratio tests, Wald tests, score tests etc etc -- when we're testing hypotheses, underlying almost all of that theory is the aim of rejecting false nulls as often as possible.)

If you're worried that your power is high when your sample size is large you had no business using this kind of hypothesis test at all, since it's not answering whatever it was you actually needed to do.

how to properly handle this,

In what sense? If a test is supposed to reject false nulls, then you reject when the test statistic is in the rejection region. Everything is fine, as long as the assumptions are reasonable.

Is it to find a sample size, n, such that Chi-Square test is still efficient and just randomly draw samples up to n?

Your question makes no sense. Efficient compared to what? You can't mean efficient compared to a test done on the original sample size, because it won't be.

I know there's Fisher's exact test but it seems like it's also affected by large sample size.

So will literally every suitable test of the same hypothesis. They're designed to reject more with larger sample size. Because we estimate the amount of dependence more and more precisely the test becomes more able to tell small dependence from independence. Fisher, Pearson chi-squared, G-test ... etc etc (there are many other choices) ... they all measure dependence slightly differently but they're all consistent.


* how large is sufficient will depend on the circumstances. Sometimes it's surprisingly large

7
monkeyunitedOP•

This is insanely good. Thank you so much for spending the time to type it out. It really enhanced my understanding of hypothesis testing and pointed out what I misunderstood/where my mistake is.

2
efrique•

Glad it helped. If you have any follow up issues, let me know. I may find time to add some more detail or clarification.

1
FinancialYear•

Heroic response.

1
yonedaneda•

Or is there a similar test that's less affected by sample size?

The whole point of a test is to reject when the null is false. If the sample is large, then you can detect very small effects. If the null is true, then you will reject with probability equal to your significance threshold (say, 5%), regardless of how large your sample is. There is no problem here, it just sounds like you're not using the right tool to address your research question. If you're not interested in detecting effects, no matter how small, then you don't want to be performing a test, since a test will detect very small effects with large sample sizes (which is what is happening here).

5
monkeyunitedOP•

Thank you. This makes sense.

1
nmolanog•

State your research question. Why are you running a chi test? Significance alone is misleading. Try to use a measure effect like an OR or a raw difference proportion, build also confidence intervals for those effect measures and use that to interpret and answer your research question. As stated by other users: you must understand the difference between statistical significance and practical significance. In your settings most statistical significant result are not practically significant. Again this can be only addressed using an effect measure.

3
monkeyunitedOP•

We have a model that predicts a type of health condition and we're trying to assess if the model is fair from gender's perspective.

You (and the other person) are absolutely right about practical significance. When our model says the likelihood for the condition is extremely low, we're seeing 3x more males being found with the condition compare to female. However, that's 300 vs 100 out of 500k and while model rejects null, ratio-wise there really isn't much differences.

Which is why I initially thought the large sample size is throwing the test off, but now I see I'm mistaken because the test is working the way exactly as intended.

This also means I need to go back to the drawing board and rethink about how I want to determine if there is bias in the model. Perhaps I need to create some metrics that make sense given the context, and measure the model result against that.

1
Adamworks•

You can detect the differences between each individual ridges of your fingerprint with a strong enough microscope.

2