[Q] What to do with large sample size causing tests to be rejected?
I'm running a Chi-Square test of independence to test if two categorical variables are related. Because of our sample size (3k to even 500k), the tests are always rejected which makes sense because even if an event is random, if it happens enough time the test would still consider it "too many times".
Edit: I want to emphasis that this is an expected behavior and I'm interested in how to properly handle this, rather than why this is happening. I knew this will happen.
I'm wondering what's the appropriate next step then? Is it to find a sample size, n, such that Chi-Square test is still efficient and just randomly draw samples up to n?
Or is there a similar test that's less affected by sample size?
I know there's Fisher's exact test but it seems like it's also affected by large sample size.
Posted Mar 16, 2022, 9:03 PM UTC · edited
This is about practical significance, not statistical significance. Your chi-square tests are picking up small differences that appear unlikely to be due to chance. If they're too small for you to be interested in them, you can just say so.