It seems not possible to plot loss values (z) against all combinations of weights in all layers, especially when the network is big with thousands or millions of params; in that case, the number of points to plot is too too big.
And also, the 3D space can't be used to plot more than 3 dimensions.
However, with a deep network with lots of weights, these can be plotted:
- Loss value against any pair of 2 weights
- Turn the layer right before output layer (single neuron) into a layer of 2 neurons, and loss can be plotted against these 2 weights (but doesn't make much sense as the meaning of loss value depends on all other weights also)
Example plot when there are 2 neurons in the layer right before output layer (of 1 neuron):