I keep reading about how LSTMs can't remember the "important parts" of a sequence which is why attention-based mechanisms are required. I was trying to use LSTMs to find people's name format.
For example, "Millie Bobby Brown" can be seen as first_name middle_name last_name format, which I'll denote as 0, but then there's "Brown, Millie Bobby" which is last_name, first_name middle_name, which I'll denote as 1.
The LSTM seems to be overfitting to one classification of format. I suspect it's because it's not paying special attention to the comma which is a key feature of what format it could be. I'm trying to understand why an LSTM won't work for a case like this. It makes sense to me because LSTMs are better at identifying sequence to sequence generation and things such as summarization and sentiment analysis usually require attention. I suspect another reason why the LSTM is not able to infer the format is that the comma can be placed in different indexes of the sequence, so it could be losing its importance in the hidden state the longer the sequence is (not sure if that makes sense). Anyone else has any theories? I'm trying to convince my fellow researchers that a pure LSTM won't be sufficient for this problem.