What are the common pitfalls that we could face when training neural networks?

Question

Apart from the vanishing or exploding gradient problems, what are other problems or pitfalls that we could face when training neural networks?

Answer 1

There are several pitfalls or issues that require your attention when or before training or using neural networks. I will list some of them below, along with some questions you need to ask yourself before or while using neural networks.

• Over-fitting and under-fitting problems, and the related generalization problem. Is your neural network generalizing to unseen data?

• Availability of training and test data

  • Do you have enough data to train your neural network so that it generalizes well (i.e. it neither over-fits or under-fits)?
  • Is your test dataset big enough to assess the generalization ability of your neural network?
  • Is your data representative of the problem you are trying to solve?
  • Do you need to augment or normalize your data?
  • Do you need to use cross-validation?
  • Is your data independent and identically distributed (i.i.d.)? If your data is correlated, training could be unstable. Shuffling your data may be a feasible solution when your data is initially correlated.

• Do you have enough computational resources (i.e. GPUs) for training and testing your neural network?

• Are you solving a regression or classification problem? The type of the outputs and the loss function will typically be different in both cases

• Do you need explainability and transparency? If yes, neural networks aren't probably the best model to use, as the connections between the neurons are quite obscure and don't really represent any meaningful interaction. That's why neural networks are called black-boxes.

• Do you need uncertainty estimation? If yes, you may want to try Bayesian neural networks. Typical neural networks are not very appropriate for uncertainty estimation!

• If you use a neural network for function approximation (e.g. in reinforcement learning), you will lose certain convergence guarantees.

Answer 2

I can't say that it is the biggest problem of deep neural network but it is one of the big problem with deep neural network.

Other issue which happens a lot is overfitting on the training data hence network behaves badly on test set which can be solved using regression. So you have to make sure that network is generalised enough.

If the network is trying to have a classification then if the outputs are mutually inclusive or exclusive, depending upon that you define your loss function which will be categorical cross entropy for mutually inclusive output and binary cross entropy for mutually exclusive.

Other thing which I can think is about all zero initialization. Initializing the network weights as zero can lead to same gradient computation and due to which network does not learn most of the times.