When training a neural network for binary classification on a highly imbalanced set its training loss decreases, however validation loss increase even though accuracy is very high (due to highly imbalanced dataset) and its other metrics (auc, recall, precision) are bad.
I want to figure out whether there is something wrong with the model architecture, therefore building a new model would be the right choice or is the model learning something; however you just need to tweak some hyperparameters.
By plotting activation outputs and weights at each layer we can see if there is a vanishing/exploding gradient or saturation at a certain layer (activation output being on extremes when using saturating activation functions like tanh, sigmoid).
My question is, is that only thing we can interpret from weights and activation output values?
What can weights distribution of layers at each layer tell us?
I'm very confused since from the DataCamp - TensorBoard Tutorial it says: “Visualizing network weights is important, because if the weights are wildly jumping here and there during learning it indicates something is wrong with the weight initialization or the learning rate.”,
however from: Stack Overflow - Understanding TensorBoard (weight) histograms I’ve understood the answer as: "changes in weight imply learning" because the answerer looks at the first 3 layers (where their weights do not change at each iteration) and says that it is not learning, and looks at last layer (weight changes) and says that it is learning.