How to improve prediction performance of periodic data?

I have a 1 column dataset of $$50 000$$ points where 95% of the values equal $$-50$$. The data looks like the following: $$\begin{matrix} \text{time} & \text{value}\\ 1&-50 \\ 2&-50 \\ 3&-50 \\ 4& -50 \\ 5&3 \\ 6&-50\\ 7&-50\\ 8&5 \end{matrix}$$ As an addition, I know the exact time instance in which I will get value $$\neq -50$$ (as in the example above these are instances $$5$$ and $$8$$). The data is somewhat periodic, so the values which are different from $$-50$$ are chosen from a finite set $$\mathcal{S}$$.

To predict the values I use a 3 layer LSTM network with l2 regularizer where along with the values I input another column that looks like that: $$\begin{matrix} \text{time} & \text{value} & \text{expect a change}\\ 1&-50 & 0 \\ 2&-50 & 0\\ 3&-50 & 0\\ 4& -50 & 1\\ 5&3 & 0\\ 6&-50& 0\\ 7&-50& 1\\ 8&5 & 0 \end{matrix}$$ which identifies the change one time instant in advance, so the LSTM will know to expect a change. However, the prediction performance is quite poor, it always predicts the change in the value but is far from real one and usually takes values out of the set $$\mathcal{S}$$. Any idea of how this could be improved?