1
$\begingroup$

I am trying to build a recurrent neural network from scratch. It's a very simple model. I am trying to train it to predict two words (dogs and gods). While training, the value of cost function starts to increase for some time, after that, the cost starts to decrease again, as can be seen in the figure.

enter image description here

I am using the gradient descend method for optimization. Decreasing the step size/learning rate does not change the behavior. I have checked the code and math, again and again, I don't think there is an error (I could be wrong).

Why is the cost function not decreasing monotonically? Could there be a reason other than an error in my code/math? If there is an error, do you think that it is just a coincidence that each time the system finally converges to a very small value of error?

I am a beginner in the field of machine learning, hence, many questions I have asked may seem foolish to you. I am saving the values after every 100 iterations so the figure is actually for 15000 iterations.

About training: I am using one-hot encoding. As the training data has only two samples ("gods" and "dogs"), where each alphabet is represented as d=[1,0,0,0],o=[0,1,0,0],g=[0,0,1,0],s=[0,0,0,1]. The recurrent neural network (RNN) goes back to a maximum of 3 time units, (e.g for dogs, the first input is 'd', then 'o', followed by 'g' and s). So, for the second input, the RNN goes back to 1 input, for the third input the RNN observes both previous inputs and so on. After calculating the gradients for the word "dogs", the values of the gradients are saved and the process is repeated for the word "gods". The gradients calculated for the second input "gods" are summed with the gradients calculated for "dogs" at the end of each epoch/iteration, and then the sum is used to updated all the weights. In each epoch, the inputs remain the same i.e "gods" and "dogs". In mini-batch training, in some epoch the RNN may encounter new inputs, hence, the loss may increase. However, I do not think that what I am doing qualifies as mini-batch training as there are only two inputs, both inputs are used in each epoch, and the sum of calculated gradients is used to update the weights.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
0
$\begingroup$

In general, there's nothing wrong with training loss to increase from time-to-time during training.

This is because GD with minibatch is a stochastic process and doesn't guarantee that the loss will decrease at each step.

Improve this answer
$\endgroup$
2
  • $\begingroup$ But I am not training with minibatches. $\endgroup$
    – ZZ1
    Jul 7, 2020 at 1:15
  • $\begingroup$ @ZainAli So, how exactly are you training? Maybe edit your post to clarify this. $\endgroup$
    – nbro
    Jul 7, 2020 at 12:44
0
$\begingroup$

Since neural networks rely on stochasticity (i.e. randomness) to initialize their parameters and gradient descent selects random batches of training data at each iteration, is perfectly normal if the value of loss function fluctuates instead of decreasing monotonically.

Improve this answer
$\endgroup$
1
  • $\begingroup$ I understand that non-monotonicity is related to minibatches, but not to parameter initialization. $\endgroup$
    – Jaume Oliver Lafont
    Mar 25 at 7:24

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .