I have trained a recurrent neural network based on 1 stack of LSTM cells. I use it to solve a classification problem.
The RNN cell has 48 hidden states. The output of the last unfolded LSTM cell is linearly projected into two dimensions corresponding to two mutually exclusive classes. I train with softmax cross-entropy loss.
I also know that both my train and test sets are mislabeled(!) to a certain extent. Possibly about 10% of items labelled as class 1 are actually class 0. And the same hold in other direction.
What puzzles me is this. Every time I train the network from scratch and plot a precision recall curve for the validation set in the end. And every time it is different! Especially in the very beginning in the range that corresponds to high precision levels. Why is this happening (this instability)?
I tried various numbers of epochs, training rates, number of hidden states in LSTM. Every time it is the same, but for some combinations of these parameters the variability is less (e.g. 8 out of 10 train/test runs i see more or less the same precision recall curve and 2 times it is severely different and worse).