# How to plot Loss Landscape with more than 2 weights in the network

For a single neuron with 2 weights, I can plot the loss landscape and it looks like this (OR data, sigmoid activation, MAE loss):

But, when the neuron accepts more inputs, which means more than 2 weights required, or when there are more neurons, more layers in the network; how should the 3D loss landscape be plotted?

• I don't think you can, as the weights increase so does the number of dimensions. You could perhaps plot 3 weights using color in the graph as the 4th dimension, but I don't think you really could for more – Recessive Oct 18 '19 at 4:17
• @Recessive oh, but i can see some very complex loss landscape this way: pyimagesearch.com/wp-content/uploads/2019/10/… – datdinhquoc Oct 18 '19 at 4:30
• the image in the comment right above is from this article: pyimagesearch.com/2019/10/14/… – datdinhquoc Oct 18 '19 at 4:31
• Nope, that's actually just a mislabelled graph by medium. The original is about halfway down this page: firsttimeprogrammer.blogspot.com/search/label/…. You'll see this is a graph for f(x,y) = x**2 + y**2 –2*x*y. You can see the (falsely) modified image halfway down this page: medium.com/@RosieCampbell/…. EDIT: The medium post actually specifies this graph is for only 2 weights. It's just the pyimagesearch that loses this in translation. – Recessive Oct 18 '19 at 6:44

It seems not possible to plot loss values (z) against all combinations of weights in all layers, especially when the network is big with thousands or millions of params; in that case, the number of points to plot is too too big.

And also, the 3D space can't be used to plot more than 3 dimensions.

However, with a deep network with lots of weights, these can be plotted:

• Loss value against any pair of 2 weights
• Turn the layer right before output layer (single neuron) into a layer of 2 neurons, and loss can be plotted against these 2 weights (but doesn't make much sense as the meaning of loss value depends on all other weights also)

Example plot when there are 2 neurons in the layer right before output layer (of 1 neuron):