# Confidence Interval around prediction with bootstrapping

I want to generate a confidence interval around my prediction (vector) $$\hat{y}$$. I have implemented the following procedure. However, I am not sure whether this makes sense in a statistical way:

1. I have a data set. First I split it into a 80% training set (2000 measurements), 10% valdidation set and 10% testing set (250 measurements)
2. I resample B ($$\sim 100$$) training sets from the original training set with replacement.

• For each of the B training datasets I train the model $$b$$ and
validate it (everytime I use the same validation set).

• I use the testset from point 1. and make a prediction $$\hat{y}_i^b$$ (so everytime I use the same test set, since I need the predictions for the same input values)

3. I calculate the average of the $$B$$ predictions. $$\bar{\hat{y}}_i=\frac{1}{B}\sum_{b=1}^B\hat{y}_i^b$$.
4. I calculate the variance ($$i\in [1,250]$$) $$\sigma_{\hat{y}_i}^2=\frac{1}{B-1}\sum_{b=1}^B(\hat{y}_i^b -\bar{\hat{y}}_i)^2$$

5. I guess the $$95\%$$ confidence interval for the prediction $$\hat{y}_i$$ is $$\hat{y}_i\in \bar{\hat{y}}_i\pm z_{0.025}\frac{\sigma_{\hat{y}_i}}{\sqrt{B}}$$ with $$z_{0.025}=1.96$$

6. If I sort the $$\hat{y}_i$$ values and plot it together with the upper and lower bound, I will get the prediction curve with a CI.

My biggest uncertainty relates to step 5). I read in a book Supervised Classification:Quite a Brief Overview, Marco Loog:

When the population standard deviation $$\sigma$$ is known and the parent population is normally distributed or $$N>30$$ the $$100(1-\alpha)$$ CI for the population mean is given by the symmetrical distribution for the standardized normal distribution $$z$$ $$\mu\in \bar{x}\pm z_{a/2}\frac{\sigma}{\sqrt{N}}$$

Is it correct to say here $$N=B$$ (number of bootstrap models or number of resampled trainingsets or number of estimators $$\hat{y}_i^b$$ ). Does the procedure make sense?