I have a string of characters encoding a molecule. I want to regress some properties of those molecules. I tried using an LSTM that encodes all one hot encdoed characters, and then I take the last hidden state fed into a linear layer to regress the property. This works fine, but I wanted to see if transformers can do better, since they are so good in NLP.
However, I am not quiet sure about two things:
- Pytorch transformer encoder layer has two masking parameters: "src_mask" and "src_key_padding_mask". The model needs the whole string to do the regression, so I dont think I need "src_mask", but I do padding with 0 for parallel processing, is that what "src_key_padding_mask" is for?
- What output from the transformer do I feed into the linear regression layer? For the LSTM I took the last hidden output. For the transformer, since everything is processed in parallel, I feel like I should rather use the sum of all, but it doesn't work well. Instead using only the last state works better, which seems arbitrary to me. Any ideas on how to properly do this, how do sentiment analysis model do it?