[Statistics] Is my general understanding of null hypothesis testing concept correct, also a couple of questions?
So say you want to test if two programs are faster in a computer. You say program A is faster than B.
But your null hypothesis is "there is no difference, if people used both programs they would be equal in terms of time taken".
So you conduct your experiment and get your results. You get a p-value. The p-value is how likely you are to observe data as at least as extreme as your sample. So if p = 0.01, does this mean youll hardly see an "outlier"?
Anyways, so if p < alpha, we can reject the null hypothesis and say "there is a difference, program A is faster".
and if p > alpha, we can fail to reject the null hypothesis, which means we cant say for certain there is no difference.
and theres Type I and II errors, type I is when you falsely reject a true null hypothesis, and Type || where you falsely reject a false null hypothesis, what exactly are these? and how would you have known youve made either one of these errors?
Posted Oct 24, 2020, 10:58 AM UTC
This is pretty correct. Some small fixes:
Presumably your statistic is average time taken, so when you say "they are the same" for the null hypothesis, you mean the average runtime for both programs are the same.
Your definition of the p-value lacks a few crucial elements. The p-value is the probability, assuming the null hypothesis is true, of seeing a difference between the null hypothesis and the the observed data as extreme or more extreme than what was actually observed.
So a p-value of 0.01 is not about outliers. It's that if the two programs really have the same average run time, there's only about a 1% chance of having seen the difference you saw.
If p> alpha, we cannot say that there IS a difference in the average run times. The evidence is not strong enough to move us from our initial assumption that there's no difference.
You have Type I and Type II errors defined correctly. A Type I error in this situation would mean that the two programs really do have the same average runtime, but the data lead to a p-value less than alpha, so you rejected the null hypothesis. A Type II error would be that program A really is faster, but the data lead to a p-value > alpha, so you did not reject the null hypothesis and continue to think the programs have the same runtime.
You don't know if you made an error. That's why consideration must be made about the consequences if Type I and Type II errors and alpha set accordingly.
Is the p-value the probability of a type I error? I can’t remember exactly how they are connected but I thought it was something like that
Alpha is the probability of a Type I error.
Thanks a lot for the clear explanation.
and so if there is a Type I error, it's pretty much means you say there is a difference, when there isn't, and Type ll is when you say there isn't a difference, but there is?
Correct.
Hey sorry couple more things shouldn't my definition of Type 2 error be non-rejection of a false null hypothesis ?
instead of falsely rejecting a false null hypothesis, cause you aren't rejecting anything?
and when we fail to reject we cant say for certain there is no difference, but we can say there is a significant difference if we reject H0? In Null Hypothesis Significance testing, is Significance just another word for difference?