1
$\begingroup$

I want to generate a confidence interval around my prediction (vector) $\hat{y}$. I have implemented the following procedure. However, I am not sure whether this makes sense in a statistical way:

  1. I have a data set. First I split it into a 80% training set (2000 measurements), 10% valdidation set and 10% testing set (250 measurements)
  2. I resample B ($\sim 100$) training sets from the original training set with replacement.

    • For each of the B training datasets I train the model $b$ and
      validate it (everytime I use the same validation set).

    • I use the testset from point 1. and make a prediction $\hat{y}_i^b$ (so everytime I use the same test set, since I need the predictions for the same input values)

  3. I calculate the average of the $B$ predictions. $$\bar{\hat{y}}_i=\frac{1}{B}\sum_{b=1}^B\hat{y}_i^b$$.
  4. I calculate the variance ($i\in [1,250]$) $$\sigma_{\hat{y}_i}^2=\frac{1}{B-1}\sum_{b=1}^B(\hat{y}_i^b -\bar{\hat{y}}_i)^2$$

  5. I guess the $95\%$ confidence interval for the prediction $\hat{y}_i$ is $$\hat{y}_i\in \bar{\hat{y}}_i\pm z_{0.025}\frac{\sigma_{\hat{y}_i}}{\sqrt{B}}$$ with $z_{0.025}=1.96$

  6. If I sort the $\hat{y}_i$ values and plot it together with the upper and lower bound, I will get the prediction curve with a CI.

My biggest uncertainty relates to step 5). I read in a book Supervised Classification:Quite a Brief Overview, Marco Loog:

When the population standard deviation $\sigma$ is known and the parent population is normally distributed or $N>30$ the $100(1-\alpha)$ CI for the population mean is given by the symmetrical distribution for the standardized normal distribution $z$ $$\mu\in \bar{x}\pm z_{a/2}\frac{\sigma}{\sqrt{N}}$$

Is it correct to say here $N=B$ (number of bootstrap models or number of resampled trainingsets or number of estimators $\hat{y}_i^b$ ). Does the procedure make sense?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.