3
$\begingroup$

I keep reading about how LSTMs can't remember the "important parts" of a sequence which is why attention-based mechanisms are required. I was trying to use LSTMs to find people's name format.

For example, "Millie Bobby Brown" can be seen as first_name middle_name last_name format, which I'll denote as 0, but then there's "Brown, Millie Bobby" which is last_name, first_name middle_name, which I'll denote as 1.

The LSTM seems to be overfitting to one classification of format. I suspect it's because it's not paying special attention to the comma which is a key feature of what format it could be. I'm trying to understand why an LSTM won't work for a case like this. It makes sense to me because LSTMs are better at identifying sequence to sequence generation and things such as summarization and sentiment analysis usually require attention. I suspect another reason why the LSTM is not able to infer the format is that the comma can be placed in different indexes of the sequence, so it could be losing its importance in the hidden state the longer the sequence is (not sure if that makes sense). Anyone else has any theories? I'm trying to convince my fellow researchers that a pure LSTM won't be sufficient for this problem.

Improve this question
$\endgroup$
8
  • $\begingroup$ Can you provide examples of misclassifications. $\endgroup$
    – respectful
    Dec 26, 2019 at 5:56
  • $\begingroup$ Have you tried a highly naive algorithm that outputs 1 in there is a comma and 0 otherwise? $\endgroup$
    – respectful
    Dec 26, 2019 at 21:12
  • $\begingroup$ This could depend on a lot of things, so if you could provide the architecture you're using, that would help. For example, if you were to try and use regression for this (ie, 1 input node: a=0,b=1,c=2... until ,=26), most likely you will get very bad results. If you are doing this, make the input a 1 hot vector of size 27 for all letters and comma. You'll likely get better results. $\endgroup$
    – Recessive
    Dec 27, 2019 at 1:05
  • $\begingroup$ @Recessive the architecture is using an autoencoder (Because I'm also trying to generate infer what is the first, middle and last name using stochastic variational inference). The encoder is made using an LSTM because hopefully it'd keep track of the position of the comma and spaces. Then that hidden state is passed into a standard nn for classification. $\endgroup$
    – user8714896
    Dec 27, 2019 at 3:38
  • $\begingroup$ @Recessive I'm very convinced if I embed the inputs better that I could just use a nn for very accurate classification. Like "Millie Bobby Brown" Can be embedded as "10101", 1=word, 0 =space and 2=comma. Because for format we don't actually care about the names itself, just the spaces and comma. So for format inference there's a lot of useless features being stored in the hidden state that are just added noise. $\endgroup$
    – user8714896
    Dec 27, 2019 at 3:40

2 Answers 2

Reset to default
1
$\begingroup$

The problem is not that RNN flavours such as LSTMs are incapable of keeping track of the "important" parts of the input. They also do not have much trouble recognizing commas in different places.

To prove this point, I recommend reading Andrej Karpathy's excelllent write-up about the behaviour of individual RNN "neurons".

Addressing specifically this comment in your question:

I suspect another reason why the LSTM is not able to infer the format is that the comma can be placed in different indexes of the sequence, so it could be losing its importance in the hidden state the longer the sequence is (not sure if that makes sense).

If commas are relevant to solve the task at hand, LSTMs can learn to remember its position or related information. This information is not necessarily diluted by repeated application of reccurrence with long sequences: networks can learn to propagate and promote crucial information from one recurrent state to the next.

Input sequences have arbitrary length, which means that LSTMs need to compress information about seen sequence elements

Rather, the input sequence has an arbitrary length, while the LSTM state vectors have a fixed size. State vectors are the only way for an LSTM to "keep track of important parts". This means that those fixed-size vectors are a bottleneck and there is an information-theoretic upper bound on the amount of information about "important parts" that can be kept by an LSTM.

LSTMs potentially take multiple decisions. For each decision, something else in the input sequence is most important

For tasks such as summarization that you mention in the question, an LSTM makes a series of predictions (predicting the tokens of the summary one token at a time). For each prediction, different things in the input sequence might be important. Put another way, for each decision, another view of the input may be most helpful.

This is a key motivation for using attention networks. Each time an LSTM is making a prediction, an attention network can provide a dynamic, optimally helpful view of the input sequence.

Improve this answer
$\endgroup$
0
$\begingroup$

You can do custom POS Tagging and use it as a multi featured sequence2sequence.

Improve this answer
$\endgroup$
1
  • $\begingroup$ I think that might still run into the same issues because the comma and positions end up varying based on name so the position isn't as much of a relevant feature $\endgroup$
    – user8714896
    Dec 27, 2019 at 9:35

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .