I've trained a neural network that can predict the $(n+1)^{th}$ element in a sequence, given the $n^{th}$ element.

It does a pretty good job doing this, with very little error.

The problem emerges when I start using the network recursively. When I feed in the $n^{th}$ element, I get the $(n+1)^{th}$ element with very little error, as I said earlier. I feed the $(n+1)^{th}$ element (the prediction) back into the network to get the $(n+2)^{th}$ element. I do this multiple times to get whatever element I want to get.

The issue is that each time I feed in a past prediction to get a new prediction, the tiny error in the previous prediction is compounded. The error increases exponentially as I predict successive elements in the sequence.

This was a problem I anticipated before I even began training the model. I figured that I'd be able to make the model so accurate that this error buildup wouldn't be noticeable until the model got far into the predicted sequence. I severely underestimated the exponential error buildup, and I don't think it's possible for me to make my model more accurate than it already is.

I figured that this is a problem that many others would've faced in the past, but I haven't found any good solutions to the problem. Are there useful tricks/methods by which I can fix the issue?

Any help would be much appreciated.

Thank you.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.