3
$\begingroup$

From this page in Interpretable-ml book and this article on Analytics Vidhya, it means to know what has happened inside an ML model to arrive at the result/prediction/conclusion.

In linear regression, new data will be multiplied with weights and bias will be added to make a prediction.

And in boosted tree models, it is possible to plot all the decisions as trees that results in a prediction.

And in feed-forward neural networks, we will have weights and biases just like linear regression and we just multiply weights and add bias at each layer, limiting values to some extent using some kind of activation function at every layer, arriving finally at prediction.

In CNNs, it is possible to see what happens to the input after having passed through a CNN block and what features are extracted after pooling (ref: what does a CNN see?).

Like I stated above, one can easily know what happens inside an ML model to make a prediction or conclusion. And I am unclear as to what makes them un-interpretable!. So, what exactly makes an algorithm or it's results un-interpretable or why are these called black box models? Or am I missing something?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

In a simple linear model of the form $y = \beta_0 + \beta_1 x $ we can see that increasing $x$ by a unit will increase the prediction on $y$ by $\beta_1$. Here we can completely determine what the effect on the models prediction will be by increasing $x$. With more complex models such as neural networks it is much more difficult to tell due to all the calculations that a single data point is involved in. For instance, in a CNN as you mentioned, if I changed the value of a pixel in an image we were passing through the CNN you wouldn't really be able to tell me exactly the effect this would have on the prediction like you can with the linear model.

Improve this answer
$\endgroup$
3
  • $\begingroup$ so, does that make tree-based model non-interpretable as well? since it wouldn't be completely obvious how change in an input affects the final outputs after having gauged through all the trees in the model? $\endgroup$
    – Naveen Reddy Marthala
    Commented Oct 9, 2020 at 10:14
  • 2
    $\begingroup$ @NaveenKumar Decision trees are interpretable to the extent that you know that at every node there's a decision based on an "if/else" statement, so you can recover how a prediction is made by following the path through the tree. In that sense, decision trees are interpretable (i.e. you know that if it was classified as "X", it's because at node $y$ a certain decision was made). However, I would say that interpretability depends on what the human really wants to know. $\endgroup$
    – nbro
    Commented Oct 9, 2020 at 10:19
  • 2
    $\begingroup$ @NaveenKumar nbro pretty much answered your question for me, and I agree it depends on what the human really wants to know. For instance, my background is statistics and I know that medical statisticians would have no interest in using e.g. a Neural Network because you wouldn't be able to tell which features are having the effect on the model; in machine learning often the case is that you don't get how the model works so long as it does, which is why more complex models are popular. $\endgroup$
    – David
    Commented Oct 9, 2020 at 10:55

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .