I have created a simple AdaBoostRegressor (sklearn) model which is trained on a feature set $X$ to predict a variable $y$. The model can be used to create a prediction based on a feature set, however I have taken it a step further and created a confidence interval for a prediction.

I am not sure exactly how to do this, but I have made the following attempt, here is the algorithm:

  1. Given a training set of features $X$, and $y$ variables; train the Adaboost model and estimate $\hat{y}$ for the entire training sample.
  2. Calculate the scaled model error $\epsilon = (\hat{y} - y)/y$ for the entire training sample.
  3. Based on newly introduced feature $X_f$, create a forecast $y_f$ based on the model from step 1.
  4. Based on the new features, for each tree find the node the new features belong to. Save the observations that fall in these nodes, and name them comparable observations. Remove any duplicates in case they appear.
  5. Given a the set of comparable observations from 4, collect the errors from step 2.
  6. Now we have a forecast $y_f$ from step 3, and comparable errors from step 5. The next step is to model the distribution of the errors from step 5; to do this I fit a gaussian kernel to the distribution. Then I invert it to extract the quantiles, since I want a 95% CI, I extract the quantiles for 2.5% ($q_{0.025}$) and 97.5% ($q_{0.975}$).
  7. The 95% interval for forecast $y_f$ then becomes $\left \{y_f*(1+q_{0.025}), y_f*(1+q_{0.975}) \right \}$

Unfortunately, what I notice from this approach, is that the upper and lower bounds of the CI are very large. Any thoughts on this methodology? or alternative approaches?

  • $\begingroup$ Could you please put your main specific question in the title? $\endgroup$
    – nbro
    Feb 20 at 10:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.