1
$\begingroup$

For a single neuron with 2 weights, I can plot the loss landscape and it looks like this (OR data, sigmoid activation, MAE loss):

enter image description here

But, when the neuron accepts more inputs, which means more than 2 weights required, or when there are more neurons, more layers in the network; how should the 3D loss landscape be plotted?

$\endgroup$
  • 2
    $\begingroup$ I don't think you can, as the weights increase so does the number of dimensions. You could perhaps plot 3 weights using color in the graph as the 4th dimension, but I don't think you really could for more $\endgroup$ – Recessive Oct 18 '19 at 4:17
  • $\begingroup$ @Recessive oh, but i can see some very complex loss landscape this way: pyimagesearch.com/wp-content/uploads/2019/10/… $\endgroup$ – datdinhquoc Oct 18 '19 at 4:30
  • $\begingroup$ the image in the comment right above is from this article: pyimagesearch.com/2019/10/14/… $\endgroup$ – datdinhquoc Oct 18 '19 at 4:31
  • 2
    $\begingroup$ Nope, that's actually just a mislabelled graph by medium. The original is about halfway down this page: firsttimeprogrammer.blogspot.com/search/label/…. You'll see this is a graph for f(x,y) = x**2 + y**2 –2*x*y. You can see the (falsely) modified image halfway down this page: medium.com/@RosieCampbell/…. EDIT: The medium post actually specifies this graph is for only 2 weights. It's just the pyimagesearch that loses this in translation. $\endgroup$ – Recessive Oct 18 '19 at 6:44
0
$\begingroup$

It seems not possible to plot loss values (z) against all combinations of weights in all layers, especially when the network is big with thousands or millions of params; in that case, the number of points to plot is too too big.

And also, the 3D space can't be used to plot more than 3 dimensions.

However, with a deep network with lots of weights, these can be plotted:

  • Loss value against any pair of 2 weights
  • Turn the layer right before output layer (single neuron) into a layer of 2 neurons, and loss can be plotted against these 2 weights (but doesn't make much sense as the meaning of loss value depends on all other weights also)

Example plot when there are 2 neurons in the layer right before output layer (of 1 neuron):

enter image description here

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.