0
$\begingroup$

I am pretty new to AI and have recently been paying attention to AI explainability and the fact that it remains a hurdle within the path of commercializing certain AI systems in health for instance. I tried to do some digging myself by starting with the gradient descent algorithm as an optimization technique used to model error of predictions. Ideally, I am aware it is most suitable for predicted errors that fit a Gaussian distribution. One question I have not been able to find answers to and would really appreciate help from this community

What exactly does the AI explainability problem refer to? Is it related to the convex optimisation (G.D.) process ? Or randomly generated coefficients? How exactly do we mean when we say AI is a blackbox?

p.s: If this has already been answered elsewhere on this platform do share a link.

Further clarification on my question:

Please, reference this tweet by Y. Lecun only yesterday, I would like some more responses on relationship between the AI explainability problem and the weight optimization technique. How or why is G.D applied to optimising parameter weights contribute to AI being a blackbox. Or if this is another separate problem from AI explainability also do clarify.

Improve this question
$\endgroup$
1
  • $\begingroup$ Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. $\endgroup$
    – Community Bot
    Commented Jun 6, 2022 at 13:51

1 Answer 1

Reset to default
1
$\begingroup$

It has much more to do with the inference of new data points than the training of deep neural networks(DNNs). Because DNNs learn complicated non-linear relations between your input variables and your targets, it is hard (or altogether impossible) to tell why a certain new data point has been classified as it was.

Take for example a model that takes in a lot of data about your health status and tells the insurance company what your insurance premium should be. The company would like to be able to tell you that because of some factor X, your insurance premium is Y. If the model is some simple linear regression, they can tell you that e.g. because you smoke, your premium increase by XY dollars, since you can interpret the coefficient of the dummy variable: Smoker: yes/no.

However, with a DNN model, all they can tell you is that "Our model, that's very complicated and precise, told us that your premium is Y?" In other words, because of a large number of parameters and non-linear activation functions you cannot tell what's the precise effect of smoking on your insurance premium. I hope this makes it clearer :)

Improve this answer
$\endgroup$
1
  • $\begingroup$ Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center. $\endgroup$
    – Community Bot
    Commented Jun 6, 2022 at 13:53

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .