1
$\begingroup$

For a single neuron with 2 weights, I can plot the loss landscape and it looks like this (OR data, sigmoid activation, MAE loss):

enter image description here

But, when the neuron accepts more inputs, which means more than 2 weights required, or when there are more neurons, more layers in the network; how should the 3D loss landscape be plotted?

Improve this question
$\endgroup$
4
  • 2
    $\begingroup$ I don't think you can, as the weights increase so does the number of dimensions. You could perhaps plot 3 weights using color in the graph as the 4th dimension, but I don't think you really could for more $\endgroup$
    – Recessive
    Oct 18, 2019 at 4:17
  • $\begingroup$ @Recessive oh, but i can see some very complex loss landscape this way: pyimagesearch.com/wp-content/uploads/2019/10/… $\endgroup$
    – Dan D.
    Oct 18, 2019 at 4:30
  • $\begingroup$ the image in the comment right above is from this article: pyimagesearch.com/2019/10/14/… $\endgroup$
    – Dan D.
    Oct 18, 2019 at 4:31
  • 2
    $\begingroup$ Nope, that's actually just a mislabelled graph by medium. The original is about halfway down this page: firsttimeprogrammer.blogspot.com/search/label/…. You'll see this is a graph for f(x,y) = x**2 + y**2 –2*x*y. You can see the (falsely) modified image halfway down this page: medium.com/@RosieCampbell/…. EDIT: The medium post actually specifies this graph is for only 2 weights. It's just the pyimagesearch that loses this in translation. $\endgroup$
    – Recessive
    Oct 18, 2019 at 6:44

1 Answer 1

Reset to default
0
$\begingroup$

It seems not possible to plot loss values (z) against all combinations of weights in all layers, especially when the network is big with thousands or millions of params; in that case, the number of points to plot is too too big.

And also, the 3D space can't be used to plot more than 3 dimensions.

However, with a deep network with lots of weights, these can be plotted:

  • Loss value against any pair of 2 weights
  • Turn the layer right before output layer (single neuron) into a layer of 2 neurons, and loss can be plotted against these 2 weights (but doesn't make much sense as the meaning of loss value depends on all other weights also)

Example plot when there are 2 neurons in the layer right before output layer (of 1 neuron):

enter image description here

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .