Below is the loss of the same training run at different scales illustrating the plateau phenomenon. Source p3.

It seems to me that each dip adds constraints to the neural network optimization, and that neural networks must be highly over parameterized to enable the optimizer to find paths through to the next dip.

I am trying to train timeseries features for a covariance matrix (i.e. einsum('bic,bjc->bij', model(s0), model(s1))). The first dip corresponds to the neural net maximizing the cosine similarities, while being linear in the magnitude i.e. $\gamma\lVert s \rVert =\lVert model(s)\rVert$. The next dip corresponds to the the neural network being nonlinear in magnitude (i.e. silence, and everything else, being mapped onto the unit sphere).

Here are a few responses to plateauing

  1. Regularize the architecture (an architecture that is non-linear in magnitude)
  2. Regularize the loss function (punish being linear in magnitude)
  3. Iterate on the training set to find architectures/hyperparameters that plateau less.
  4. Brute force it. Just leave the neural network training.

I'm sceptical about 1 and 2 as it would only get you past the first plateau. I don't hear much about 3, although I think it might be a good idea. Does anyone do this in practice?


1 Answer 1


I think the primary solution to plateauing is improving the dataset.

The iteration process should be

  1. Train the neural network
  2. Identify what properties are causing the neural network to plateau.
  3. Gather more data, improve the targets, argument the existing dataset, to target these undesired properties.
  4. Find the next plateau, and repeat until the desired results are achieved.

We are changing the loss landscape using the dataset.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .