I want to generate a confidence interval around my prediction (vector) $\hat{y}$. I have implemented the following procedure. However, I am not sure whether this makes sense in a statistical way:

  1. I have a data set. First I split it into a 80% training set (2000 measurements), 10% valdidation set and 10% testing set (250 measurements)
  2. I resample B ($\sim 100$) training sets from the original training set with replacement.

    • For each of the B training datasets I train the model $b$ and
      validate it (everytime I use the same validation set).

    • I use the testset from point 1. and make a prediction $\hat{y}_i^b$ (so everytime I use the same test set, since I need the predictions for the same input values)

  3. I calculate the average of the $B$ predictions. $$\bar{\hat{y}}_i=\frac{1}{B}\sum_{b=1}^B\hat{y}_i^b$$.
  4. I calculate the variance ($i\in [1,250]$) $$\sigma_{\hat{y}_i}^2=\frac{1}{B-1}\sum_{b=1}^B(\hat{y}_i^b -\bar{\hat{y}}_i)^2$$

  5. I guess the $95\%$ confidence interval for the prediction $\hat{y}_i$ is $$\hat{y}_i\in \bar{\hat{y}}_i\pm z_{0.025}\frac{\sigma_{\hat{y}_i}}{\sqrt{B}}$$ with $z_{0.025}=1.96$

  6. If I sort the $\hat{y}_i$ values and plot it together with the upper and lower bound, I will get the prediction curve with a CI.

My biggest uncertainty relates to step 5). I read in a book Supervised Classification:Quite a Brief Overview, Marco Loog:

When the population standard deviation $\sigma$ is known and the parent population is normally distributed or $N>30$ the $100(1-\alpha)$ CI for the population mean is given by the symmetrical distribution for the standardized normal distribution $z$ $$\mu\in \bar{x}\pm z_{a/2}\frac{\sigma}{\sqrt{N}}$$

Is it correct to say here $N=B$ (number of bootstrap models or number of resampled trainingsets or number of estimators $\hat{y}_i^b$ ). Does the procedure make sense?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.