[Q] Terminology of standard error vs standard deviation in bootstrap

Question

I have used bootstrap to train multiple models and evaluate them on the observations which were not included in each bootstrap sample. I wanted to report the uncertainty of my evaluation measure P. The number of bootstrap runs were around 100, so instead of reporting percentiles I've opted for standard error (or deviation?).

Let's say I have B bootstrap runs. For each bootstrap sample b I evaluate the precision (evaluation measure) of my test data predictions, let's denote this P_b. This is what I'm estimating from the samples.

I then calculate the standard error sqrt(1/(B-1) * sum((P - mean(P))^2) ) (imagine this as vectorised R code with P now a vector containing B precision measures of each bootstrap sample b).

My question is: should I report this as the standard error of the evaluation measure, or as the standard deviation? I got feedback that disagreed with my usage of "standard error" in this context, suggesting standard error can only be the deviation of multiple means, and what I had was rather a standard deviation of the precision measure. On the other hand, the P_b can perhaps be considered a summary "statistic" or a "parameter" that I'm estimating and in that case standard error would not be incorrect.

What is the correct terminology here? Would I have to do multiple runs of B bootstraps and calculate the deviation of the mean(P_B) in these multiple runs to be correct in calling it a standard error?

Posted May 26, 2020, 9:53 AM UTC · edited

94
AllenDowney•

What you get from the bootstrapping process is a sequence of values from the sampling distribution of whatever you are trying to estimate.

The standard deviation of those values is* the standard error of the estimate.

In other words, you compute it by computing a standard deviation, but the result is* a standard error.

* It is actually an estimate of the standard error, but that would make the previous sentences confusing.

10
chiefMeteor•

Standard error of a statistic is the standard deviation of the sampling distribution of that statistic. In the case of bootstrap, you are generating a sampling distribution of your statistic (P) based on 100 samples. So I think that if you take the standard deviation of those 100 values for P, that would be considered a standard error.

2
efrique•

imagine this as vectorised R code

Then your code has an error.

1
mLalushOP•

Fair point. In actuality I used sd(). To illustrate clearly that I wasn't taking some sample standard deviation but rather sd() on something calculated on bootstrap samples I decided to (rather sloppily) write it out more explicitly in code.

This is what I meant: \sqrt{\frac{1}{B-1} \sum^B_{b=1} (P^*_b - \bar{P})^2}.

Or corrected code: sqrt(1/(B-1) * sum((P - mean(P))^2) )

Any insight on the terminology?

2