2
$\begingroup$

There is some sort of art to using the right loss function. However, I was wondering if there is a way to derive the loss function if I gave you a neural network model (the weights) as well as the training data.

The point of this exercise is to see what family of loss functions we would get. And how that compares to the loss function that actually gave rise to the model.

Improve this question
$\endgroup$
0

1 Answer 1

Reset to default
1
$\begingroup$

I don't think there's a way of doing what you want, at least, I've never seen such a thing (and, currently, I am not seeing how this could be done in the general case).

The same neural network model but with different (or same) weights could have been trained with the same loss function or not. For example, although it may not be a good idea, you can train a neural network for classification with the mean squared error, as opposed to the typical cross-entropy. Moreover, even if you know the loss function that the neural network is trained with, the training data alone may not lead to the same set of weights because the actual weights depend on different (possibly stochastic) factors, such as if (or how) you shuffle the data or the batch size.

Improve this answer
$\endgroup$
5
  • $\begingroup$ I deleted my comment as soon as I read it / it was poorly expressed and you are too fast. I realize that I neglected the architecture of the network as well $\endgroup$
    – aquagremlin
    Commented May 30, 2020 at 13:35
  • $\begingroup$ The idea that led me to this question is the fact that any function can be represented as a polynomial approximation (Taylor series for example). Since the loss function is thus represented, then maybe the tools of linear algebra can be used to derive it from the weights of a model and the training data. Having such a loss function should then allow training the original network from the training data from a single densely connected layer. $\endgroup$
    – aquagremlin
    Commented May 30, 2020 at 13:39
  • $\begingroup$ @aquagremlin I see your point. I am not excluding the possibility to do something in that direction, but, right now, I am not seeing a solution. $\endgroup$
    – nbro
    Commented May 30, 2020 at 13:42
  • $\begingroup$ Why bother to make a loss function like this? Because I want to see if it converges faster to a smaller error. And although derived from a specific set of weights and training data, it might be applicable to similar desired models -for example instead of having a single complicated network to recognize faces, we might have several simpler single layer networks for different parts of the world. These models would smaller, faster and deployable and smaller cheaper hardware $\endgroup$
    – aquagremlin
    Commented May 30, 2020 at 13:45
  • $\begingroup$ @aquagremlin When I was writing this answer, I found some papers that introduce a way of learning a loss function. For example, see the paper Learning to Teach with Dynamic Loss Function. $\endgroup$
    – nbro
    Commented May 30, 2020 at 13:50

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .