Why does an LSTM cycle on initialisation?

Question

I initialised an LSTM with Xavier initialisation, although I've found this occurs for all initialisations I have tested. When initialised, if the LSTM is tested with a random input, it will get stuck in a cycle, either over a few characters or just one. Example output:

nhhbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb

f(b(bf(bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb

kk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,m

I've also noticed the LSTM is particularly bad in this way, that even when trained it has a tendency to get stuck in loops like this. It seems it has difficulty truly retaining context strong enough to over power the input, activation and output gates with only the forget gate. Is there an explanation for this?

Answer 1

This doesn't necessarily answer the question, but it does give some possible solutions to mitigate the problem.

Apparently, the emphasised looping behaviour above is a result of improper initialisation, in which I was initialising with only positive and 0 weights. The looping behavior is largely diminished by proper initialisation, however it is not totally removed. See below for examples:

djbgnuywjkfrrkrkrpueeeeeffrrdv
kmkkkkkkkkkkkrkkkkkkkkkkkkkkkk
clfwtpjbsqeeeeeeesssjjeeeeeeee
ldwgmbvcmmbvcmmmgmmmgmmmgmmmmg
ywzrfotntntnttntntunuwqzffotnt
lvvurrrrrrxafllvvurrrrrrxcfhla

(These are samples from 6 different uniquely initialised LSTM's on the exact same architecture).

I have also found that this common behaviour is typically combated by adding a certain degree of randomness in feeding in the LSTM's output. This is normally done by interpreting the output at time step $t$ of the LSTM as a probability distribution, used to pick the next value fed into the LSTM at $t+1$. This helps break non-confident looping behaviour which is where looping behaviour most commonly occurs through some quick experimentation, while still mostly retaining confident predictions.

I also tested this by randomising the input a little bit, using a probability distribution where 90% of the time the correct input is selected and 10% a random one is, and back-propagating as normal. This didn't seem to have much of an effect on the looping behaviour, although perhaps might work as a good form of regularisation. I am yet to test it.

You can also use the temperature method in LSTMs, which explained really well here: https://cs.stackexchange.com/q/79241/20691.